WO2020092635A1 - Redondance dans des véhicules autonomes - Google Patents

Redondance dans des véhicules autonomes Download PDF

Info

Publication number
WO2020092635A1
WO2020092635A1 PCT/US2019/058949 US2019058949W WO2020092635A1 WO 2020092635 A1 WO2020092635 A1 WO 2020092635A1 US 2019058949 W US2019058949 W US 2019058949W WO 2020092635 A1 WO2020092635 A1 WO 2020092635A1
Authority
WO
WIPO (PCT)
Prior art keywords
solution
sensor
module
operations
autonomous vehicle
Prior art date
Application number
PCT/US2019/058949
Other languages
English (en)
Inventor
Emilio FRAZZOLI
Andrea CENSI
Hsun-Hsien Chang
Philipp ROBBEL
Maria Antoinette MEIJBURG
Eryk Brian Nice
Eric WOLFF
Omar Al Assad
Francesco Seccamonte
Dmytro S. YERSHOV
Jeong Hwan Jeon
Shih-Yuan Liu
Tichakorn WONGPIROMSARN
Oscar Olof BEIJBOM
Katarzyna Anna Marczuk
Kevin SPIESER
Marc Lars Ljungdahl ALBERT
William Francis Cote
Ryan Lee JACOBS
Original Assignee
Frazzoli Emilio
Censi Andrea
Chang Hsun Hsien
Robbel Philipp
Meijburg Maria Antoinette
Eryk Brian Nice
Wolff Eric
Omar Al Assad
Francesco Seccamonte
Yershov Dmytro S
Jeong Hwan Jeon
Liu Shih Yuan
Wongpiromsarn Tichakorn
Beijbom Oscar Olof
Katarzyna Anna Marczuk
Spieser Kevin
Albert Marc Lars Ljungdahl
William Francis Cote
Jacobs Ryan Lee
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Frazzoli Emilio, Censi Andrea, Chang Hsun Hsien, Robbel Philipp, Meijburg Maria Antoinette, Eryk Brian Nice, Wolff Eric, Omar Al Assad, Francesco Seccamonte, Yershov Dmytro S, Jeong Hwan Jeon, Liu Shih Yuan, Wongpiromsarn Tichakorn, Beijbom Oscar Olof, Katarzyna Anna Marczuk, Spieser Kevin, Albert Marc Lars Ljungdahl, William Francis Cote, Jacobs Ryan Lee filed Critical Frazzoli Emilio
Priority to GB2017386.0A priority Critical patent/GB2587275B/en
Priority to KR1020237005763A priority patent/KR20230030029A/ko
Priority to KR1020207034459A priority patent/KR20210006926A/ko
Priority to CN201980072734.1A priority patent/CN112969622A/zh
Priority to US17/058,242 priority patent/US20210163021A1/en
Priority to DE112019005425.2T priority patent/DE112019005425T5/de
Priority to DKPA202070218A priority patent/DK202070218A1/en
Publication of WO2020092635A1 publication Critical patent/WO2020092635A1/fr

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W60/00—Drive control systems specially adapted for autonomous road vehicles
    • B60W60/001—Planning or execution of driving tasks
    • B60W60/0011—Planning or execution of driving tasks involving control alternatives for a single driving scenario, e.g. planning several paths to avoid obstacles
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/02—Ensuring safety in case of control system failures, e.g. by diagnosing, circumventing or fixing failures
    • B60W50/029—Adapting to failures or work around with other constraints, e.g. circumvention by avoiding use of failed parts
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/0055—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot with safety arrangements
    • G05D1/0077—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot with safety arrangements using redundant signals or controls
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units, or advanced driver assistance systems for ensuring comfort, stability and safety or drive control systems for propelling or retarding the vehicle
    • B60W30/08—Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
    • B60W30/09—Taking automatic action to avoid collision, e.g. braking and steering
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units, or advanced driver assistance systems for ensuring comfort, stability and safety or drive control systems for propelling or retarding the vehicle
    • B60W30/08—Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
    • B60W30/095—Predicting travel path or likelihood of collision
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/02—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to ambient conditions
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/02—Ensuring safety in case of control system failures, e.g. by diagnosing, circumventing or fixing failures
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/02—Ensuring safety in case of control system failures, e.g. by diagnosing, circumventing or fixing failures
    • B60W50/0205—Diagnosing or detecting failures; Failure detection models
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/02—Ensuring safety in case of control system failures, e.g. by diagnosing, circumventing or fixing failures
    • B60W50/023—Avoiding failures by using redundant parts
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W60/00—Drive control systems specially adapted for autonomous road vehicles
    • B60W60/001—Planning or execution of driving tasks
    • B60W60/0015—Planning or execution of driving tasks specially adapted for safety
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01D—MEASURING NOT SPECIALLY ADAPTED FOR A SPECIFIC VARIABLE; ARRANGEMENTS FOR MEASURING TWO OR MORE VARIABLES NOT COVERED IN A SINGLE OTHER SUBCLASS; TARIFF METERING APPARATUS; MEASURING OR TESTING NOT OTHERWISE PROVIDED FOR
    • G01D3/00—Indicating or recording apparatus with provision for the special purposes referred to in the subgroups
    • G01D3/10—Indicating or recording apparatus with provision for the special purposes referred to in the subgroups with provision for switching-in of additional or auxiliary indicators or recorders
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B9/00—Safety arrangements
    • G05B9/02—Safety arrangements electric
    • G05B9/03—Safety arrangements electric with multiple-channel loop, i.e. redundant control systems
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
    • G07C5/00—Registering or indicating the working of vehicles
    • G07C5/08—Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
    • G07C5/0808—Diagnosing performance data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/30—Services specially adapted for particular environments, situations or purposes
    • H04W4/38—Services specially adapted for particular environments, situations or purposes for collecting sensor information
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/30—Services specially adapted for particular environments, situations or purposes
    • H04W4/40—Services specially adapted for particular environments, situations or purposes for vehicles, e.g. vehicle-to-pedestrians [V2P]
    • H04W4/48—Services specially adapted for particular environments, situations or purposes for vehicles, e.g. vehicle-to-pedestrians [V2P] for in-vehicle communication
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W2050/0001—Details of the control system
    • B60W2050/0002—Automatic control, details of type of controller or control system architecture
    • B60W2050/0004—In digital systems, e.g. discrete-time systems involving sampling
    • B60W2050/0005—Processor details or data handling, e.g. memory registers or chip architecture
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W2050/0001—Details of the control system
    • B60W2050/0043—Signal treatments, identification of variables or parameters, parameter estimation or state estimation
    • B60W2050/005—Sampling
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/02—Ensuring safety in case of control system failures, e.g. by diagnosing, circumventing or fixing failures
    • B60W50/0205—Diagnosing or detecting failures; Failure detection models
    • B60W2050/0215—Sensor drifts or sensor failures
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/02—Ensuring safety in case of control system failures, e.g. by diagnosing, circumventing or fixing failures
    • B60W50/029—Adapting to failures or work around with other constraints, e.g. circumvention by avoiding use of failed parts
    • B60W2050/0292—Fail-safe or redundant systems, e.g. limp-home or backup systems
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2420/00—Indexing codes relating to the type of sensors based on the principle of their operation
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2420/00—Indexing codes relating to the type of sensors based on the principle of their operation
    • B60W2420/40—Photo or light sensitive means, e.g. infrared sensors
    • B60W2420/403—Image sensing, e.g. optical camera
    • B60W2420/408—
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2420/00—Indexing codes relating to the type of sensors based on the principle of their operation
    • B60W2420/54—Audio sensitive means, e.g. ultrasound
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2520/00—Input parameters relating to overall vehicle dynamics
    • B60W2520/10—Longitudinal speed
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2555/00—Input parameters relating to exterior conditions, not covered by groups B60W2552/00, B60W2554/00
    • B60W2555/20—Ambient conditions, e.g. wind or rain
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2556/00—Input parameters relating to data
    • B60W2556/35—Data fusion
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2556/00—Input parameters relating to data
    • B60W2556/45—External transmission of data to or from the vehicle
    • B60W2556/55—External transmission of data to or from the vehicle using telemetry
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60Y—INDEXING SCHEME RELATING TO ASPECTS CROSS-CUTTING VEHICLE TECHNOLOGY
    • B60Y2400/00—Special features of vehicle units
    • B60Y2400/30—Sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/30—Services specially adapted for particular environments, situations or purposes
    • H04W4/40—Services specially adapted for particular environments, situations or purposes for vehicles, e.g. vehicle-to-pedestrians [V2P]

Definitions

  • This description relates to redundancy in autonomous vehicles.
  • Autonomous vehicles can be used to transport people and/or cargo from one location to another.
  • An autonomous vehicle typically includes one or more systems, each of which performs one or more functions of the autonomous vehicle. For example, one system may perform a control function, while another system may perform a motion planning function.
  • a system includes two or more different autonomous vehicle operations subsystems, each of the two or more different autonomous vehicle operations subsystems being redundant with another of the two or more different autonomous vehicle operations subsystems.
  • Each operations subsystem of the two or more different autonomous vehicle operations subsystems includes a solution proposer configured to propose solutions for autonomous vehicle operation based on current input data, and a solution scorer configured to evaluate the proposed solutions for autonomous vehicle operation based on one or more cost assessments.
  • the solution scorer of at least one of the two or more different autonomous vehicle operations subsystems is configured to evaluate both the proposed solutions from the solution proposer of the at least one of the two or more different autonomous vehicle operations subsystems and at least one of the proposed solutions from the solution proposer of at least one other of the two or more different autonomous vehicle operations subsystems.
  • the system also includes an output mediator coupled with the two or more different autonomous vehicle operations subsystems and configured to manage autonomous vehicle operation outputs from the two or more different autonomous vehicle operations subsystems.
  • the disclosed technologies can be implemented as a method for operating, within an autonomous vehicle (AV) system of an AV, two or more redundant pipelines coupled with an output mediator, a first pipeline of the two or more redundant pipelines comprising a first perception module, a first localization module, a first planning module, and a first control module, and a second pipeline of the two or more redundant pipelines including a second perception module, a second localization module, a second planning module, and a second control module, where each of the first and second controller modules are connected with an output mediator.
  • AV autonomous vehicle
  • the method includes receiving, by the first perception module, first sensor signals from a first set of sensors of an AV, and generating, by the first perception module, a first world view proposal based on the first sensor signals; receiving, by the second perception module, second sensor signals from a second set of the sensors of the AV, and generating, by the second perception module, a second world view proposal based on the second sensor signals; selecting, by the first perception module, one between the first world view proposal and the second world view proposal based on a first perception-cost function, and providing, by the first perception module, the selected one as a first world view to the first localization module; selecting, by the second perception module, one between the first world view proposal and the second world view proposal based on a second perception-cost function, and providing, by the second perception module, the selected one as a second world view to the second localization module; generating, by the first localization module, a first AV position proposal based on the first world view; generating, by the second localization module, a second AV position proposal
  • a system includes two or more different autonomous vehicle operations subsystems, each of the two or more different autonomous vehicle operations subsystems being redundant with another of the two or more different autonomous vehicle operations subsystems; and an output mediator coupled with the two or more different autonomous vehicle operations subsystems and configured to manage autonomous vehicle operation outputs from the two or more different autonomous vehicle operations subsystems.
  • the output mediator is configured to selectively promote different ones of the two or more different autonomous vehicle operations subsystems to a prioritized status based on current input data compared with historical performance data for the two or more different autonomous vehicle operations subsystems.
  • the disclosed technologies can be implemented as a method performed by an output mediator for controlling output of two or more different autonomous vehicle operations subsystems of an autonomous vehicle, one of which having prioritized status.
  • the method includes receiving, under a current operational context, outputs from the two or more different autonomous vehicle operations subsystems; in response to determining that at least one of the received outputs is different from the other ones, promoting one of the autonomous vehicle operations subsystems which corresponds to the current operational context to prioritized status; and controlling issuance of the output of the autonomous vehicle operations subsystem having the prioritized status for operating the autonomous vehicle.
  • an autonomous vehicle includes a first control system.
  • the first control system is configured to provide output, in accordance with at least one input, that affects a control operation of the autonomous vehicle while the autonomous vehicle is in an autonomous driving mode and while the first control system is selected.
  • the autonomous vehicle also includes a second control system.
  • the second control system is configured to provide output, in accordance with at least one input, that affects a control operation of the autonomous vehicle while the autonomous vehicle is in an autonomous driving mode and while the second control system is selected.
  • the autonomous vehicle further includes at least one processor.
  • the at least one processor is configured to select at least one of the first control system and the second control system to affect the control operation of the autonomous vehicle.
  • This technique provides redundancy in control operations in case one control system suffers failure or degraded performance.
  • the redundancy in controls also allows an AV to choose which control system to use based on measured performance of the control systems.
  • a technique for detecting and handling of sensor failures in autonomous vehicle includes producing, via a first sensor, a first sensor data stream from one or more environmental inputs external to the autonomous vehicle while the autonomous vehicle is in an operational driving state and producing, via a second sensor, a second sensor data stream from the one or more environmental inputs external to the autonomous vehicle while the autonomous vehicle is in the operational driving state.
  • the first sensor and the second sensor can be configured to detect a same type of information.
  • the technique further includes detecting an abnormal condition based on a difference between the first sensor data stream and the second sensor data stream; and switching among the first sensor, the second sensor, or both as an input to control the autonomous vehicle in response to the detected abnormal condition.
  • an autonomous vehicle includes a first sensor configured to produce a first sensor data stream from one or more
  • the vehicle includes a processor coupled with the first sensor and the second sensor, the processor being configured to detect an abnormal condition based on a difference between the first sensor data stream and the second sensor data stream.
  • the processor is configured to switch among the first sensor, the second sensor, or both as an input to control the autonomous vehicle in response to a detection of the abnormal condition.
  • Detecting and handling sensor failures are important in maintaining the safe and proper operation of an autonomous vehicle.
  • a described technology can enable an autonomous vehicle to efficiency switch among sensors inputs in response to a detection of an abnormal condition.
  • Generating a replacement sensor data stream by transforming a functioning sensor data stream can enable an autonomous vehicle to continue to operate safely.
  • an autonomous vehicle includes a control system configured to affect a control operation of the autonomous vehicle, a control processor in communication with the control system, the control processor configured to determine instructions for execution by the control system, a telecommunications system in communication with the control system, the telecommunications system configured to receive instructions from an external source, wherein the control processor is configured to determine instructions that are executable by the control system from the instructions received from the external source and is configured to enable the external source in communication with the telecommunications system to control the control system when one or more specified conditions are detected.
  • an autonomous vehicle includes a control system configured to affect a first control operation of the autonomous vehicle, a control processor in communication with the control system, the control processor configured to determine instructions for execution by the control system, a telecommunications system in communication with the control system, the telecommunications system configured to receive instructions from an external source, and a processor configured to determine instructions that are executable by the control system from the instructions received from the external source and to enable the control processor or the external source in communication with the telecommunications system to operate the control system.
  • an autonomous vehicle includes a first control system configured to affect a first control operation of the autonomous vehicle, a second control system configured to affect the first control operation of the autonomous vehicle, and a telecommunications system in communication with the first control system, the telecommunications system configured to receive instructions from an external source, a control processor configured to determine instructions to affect the first control operation from the instructions received from the external source and is configured to determine an ability of the telecommunications system to communicate with the external source and in accordance with the determination select the first control system or the second control system.
  • a first autonomous vehicle has one or more sensors.
  • the first autonomous vehicle determines an aspect of an operation of the first autonomous vehicle based on data received from the one or more sensors.
  • the first autonomous vehicle also receives data originating at one or more other autonomous vehicles.
  • the first autonomous vehicle uses the determination and the received data to carry out the operation.
  • the exchange of information between autonomous vehicles can improve the redundancy of a fleet of autonomous vehicles as a whole, thereby improving the efficiency, safety, and effectiveness of their operation.
  • the first autonomous vehicle can transmit information regarding these conditions to other autonomous vehicles, such that they also have access to this information, even if they have not yet traversed that same route.
  • the other autonomous vehicles can preemptively adjust their operation to account for the conditions of the route and/or better anticipate the conditions of the route.
  • a method includes performing, by an autonomous vehicle (AV), an autonomous driving function of the AV in an environment, receiving, by an internal wireless communication device of the AV, an external message from an external wireless communication device that is located in the environment, comparing, by one or more processors of the AV, an output of the function with content of the external message or with data generated based on the content, and in accordance with results of the comparing, causing the AV to perform a maneuver.
  • AV autonomous vehicle
  • a method includes discovering, by an operating system (OS) of an autonomous vehicle (AV), a new component coupled to a data network of the AV, determining, by the AV OS, if the new component is a redundant component, in accordance with the new component being a redundant component, performing a redundancy configuration of the new component, and in accordance with the new component not being a redundant component, performing a basic configuration of the new component, wherein the method is performed by one or more special-purpose computing devices.
  • OS operating system
  • AV autonomous vehicle
  • Components can be added to an autonomous vehicle in a manner that accounts for whether or not the new module provides additional redundancy and/or will be the only component carrying out one or more functions of the autonomous vehicle.
  • redundant planning for an autonomous vehicle generally includes detecting that the autonomous vehicle is operating within its defined operational domain. If the autonomous vehicle is operating within its defined operational domain, at least two independent planning modules (that share a common definition of the operational domain) generate trajectories for the autonomous vehicle. Each planning module evaluates the trajectory generated by the other planning module for at least one collision with at least one object in a scene description. If one or both trajectories are determined to be unsafe (e.g., due to at least one collision being detected), the autonomous vehicle performs a safe stop maneuver or applies emergency braking using, for example, an autonomous emergency braking (AEB) system.
  • AEB autonomous emergency braking
  • Particular aspects of the foregoing techniques can provide one or more of the following advantages.
  • the disclosed redundant planning includes independent redundant planning modules with independent diagnostic coverage to ensure the safe and proper operation of an autonomous vehicle.
  • a method performed by an autonomous vehicle comprises: performing, by a first simulator, a first simulation of a first AV process/system using data output by a second AV process/system; performing, by a second simulator, a second simulation of the second AV process/system using data output by the first AV process/system; comparing, by one or more processors, the data output by the first and second process/system with data output by the first and second simulators; and in accordance with a result of the comparing, causing the AV to perform a safe mode maneuver or other action.
  • a system includes a component infrastructure including a set of interacting components implementing a system for an autonomous vehicle (AV), the infrastructure including a first component performing a function for operation of the AV, a second component performing the first function for operation of the AV concurrently with the first software component, a perception circuit confirmed for creating a model of an operating environment of the AV by combining or comparing a first output from the first component with a second output from the second component, and initiating an operation mode to perform the function on the AV based on the model of the operating environment.
  • AV autonomous vehicle
  • FIG. 1 shows an example of an autonomous vehicle having autonomous capability.
  • FIG. 2 shows an example“cloud” computing environment.
  • FIG. 3 shows an example of a computer system.
  • FIG. 4 shows an example architecture for an autonomous vehicle.
  • FIG. 5 shows an example of inputs and outputs that may be used by a perception module.
  • FIG. 6 shows an example of a LiDAR system.
  • FIG. 7 shows the LiDAR system in operation.
  • FIG. 8 shows the operation of the LiDAR system in additional detail.
  • FIG. 9 shows a block diagram of the relationships between inputs and outputs of a planning module.
  • FIG. 10 shows a directed graph used in path planning.
  • FIG. 11 shows a block diagram of the inputs and outputs of a control module.
  • FIG. 12 shows a block diagram of the inputs, outputs, and components of a controller.
  • FIG. 13 shows a block diagram of an example of an autonomous vehicle (AV) system that includes two or more synergistically redundant operations subsystems.
  • AV autonomous vehicle
  • FIG. 14 shows an example of an architecture for an AV which includes synergistically redundant perception modules.
  • FIG. 15 shows an example of an architecture for an AV which includes synergistically redundant planning modules.
  • FIG. 16 shows a block diagram of an example of an AV system that includes two or more synergistically redundant operations pipelines.
  • FIG. 17 shows an example of an architecture for an AV which includes synergistically redundant two-stage pipelines, each of which includes a perception module and a planning module.
  • FIG. 18 shows an example of an architecture for an AV which includes synergistically redundant two-stage pipelines, each of which includes a planning module and a control module.
  • FIG. 19 shows an example of an architecture for an AV which includes synergistically redundant two-stage pipelines, each of which includes a localization module and a control module.
  • FIG. 20 shows a block diagram of another example of an AV system that includes two or more synergistically redundant operations pipelines.
  • FIG. 21 shows an example of an architecture for an AV which includes synergistically redundant pipelines, each of which includes three or more of a perception module, a localization module, a planning module, and a control module.
  • FIGs. 22-23 is a flow chart of an example of a process for operating a pair of synergistically redundant four-stage pipelines each of which includes a perception module, a localization module, a planning module, and a control module.
  • FIG. 24 shows a block diagram of an example of an AV system that includes four synergistically redundant operations pipelines, each of which includes a perception module and a planning module, each of the modules includes a solution proposer and a solution scorer.
  • FIG. 25 shows a block diagram of an example of an AV system that includes two synergistically redundant operations pipelines, each of which includes a perception module and a planning module, each of the perception modules includes a solution proposer and a solution scorer, each of the planning modules includes multiple solution proposers and a solution scorer.
  • FIG. 26 shows a block diagram of an example of an AV system that includes two synergistically redundant operations pipelines, each of which includes a perception module and a planning module, each of the perception modules includes a solution proposer and a solution scorer, each of the planning modules includes a solution proposer and multiple solution scorers.
  • FIG. 27 is a flow chart of an example of a process performed by an output mediator for managing AV operation outputs of different AV operations subsystems coupled with the output mediator.
  • FIGs. 28-29 show computational components and data structures used by an output mediator to perform the process of FIG. 27.
  • FIG. 30 shows a redundant control system 2900 for providing redundancy in control systems for an AV.
  • FIG. 31 shows a flowchart representing a method 3000 for providing redundancy in control systems according to at least one implementation of the present disclosure.
  • FIG. 32 shows an example of a sensor-related architecture of an autonomous vehicle for detecting and handling sensor failure.
  • FIG. 33 shows an example of a process to operate an autonomous vehicle and sensors therein.
  • FIG. 34 shows an example of a process to detect a sensor-related abnormal condition.
  • FIG. 35 shows an example of a process to transform a sensor data stream in response to a detection of an abnormal condition.
  • FIG. 36 illustrates example architecture of a teleoperation system.
  • FIG. 37 shows an example architecture of a teleoperation client.
  • FIG. 38 illustrates an example teleoperation system.
  • FIG. 39 shows a flowchart indicating a process for activating teleoperator control.
  • FIG. 40 shows a flowchart representing a process for activating redundant teleoperator and human control.
  • FIG. 41 shows a flowchart.
  • FIG. 42 shows an example exchange of information among a fleet of autonomous vehicles.
  • FIGS. 43-46 show an example exchange of information between autonomous vehicles.
  • FIGS. 47-50 show an example exchange of information between autonomous vehicles, and an example modification to a planned route of travel based on the exchanged information.
  • FIGS. 51-53 show an example formation of a platoon of autonomous vehicles.
  • FIGS. 54-56 show another example formation of a platoon of autonomous vehicles.
  • FIG. 57 is a flow chart diagram showing an example process for exchanging information between autonomous vehicles.
  • FIG. 58 shows a block diagram of a system for implementing redundancy in an autonomous vehicle using one or more external messages provided by one or more external wireless communication devices, according to an embodiment.
  • FIG. 59 shows an external message format, according to an embodiment.
  • FIG. 60 shows an example process for providing redundancy in an autonomous vehicle using external messages provided by one or more external wireless communication devices, according to an embodiment.
  • FIG. 61 shows a block diagram of an example architecture for replacing redundant components in an autonomous vehicle.
  • FIG. 62 shows a flow diagram of an example process of replacing redundant components in an autonomous vehicle.
  • FIG. 63 shows a block diagram of a redundant planning system.
  • FIG. 64 shows a table illustrating actions to be taken by an autonomous vehicle based on in-scope operation, diagnostic coverage and the outputs of two redundant planning modules.
  • FIG. 65 shows a flow diagram of a redundant planning process.
  • FIG. 66 shows a block diagram of system for implementing redundancy using simulations.
  • FIG. 67 shows a flow diagram of a process for redundancy using simulations.
  • FIG. 68 shows a block diagram of a system for unionizing perception inputs to model an operating environment, according to an embodiment.
  • FIG. 69 shows an example process for unionizing perception inputs to model an operating environment, according to an embodiment.
  • connecting elements such as solid or dashed lines or arrows
  • the absence of any such connecting elements is not meant to imply that no connection, relationship or association can exist.
  • some connections, relationships or associations between elements are not shown in the drawings so as not to obscure the disclosure.
  • a single connecting element is used to represent multiple connections, relationships or associations between elements.
  • a connecting element represents a communication of signals, data or instructions
  • such element represents one or multiple signal paths (e.g., a bus), as may be needed, to affect the communication.
  • FIG. 1 shows an example of an autonomous vehicle 100 having autonomous capability.
  • the term 'autonomous capability' refers to a function, feature, or facility that enables a vehicle to be partially or fully operated without real-time human intervention, including without limitation fully autonomous vehicles, highly autonomous vehicles, and conditionally autonomous vehicles.
  • an autonomous vehicle is a vehicle that possesses autonomous capability.
  • vehicle includes means of transportation of goods or people.
  • vehicles for example, cars, buses, trains, airplanes, drones, trucks, boats, ships, submersibles, dirigibles, mobile robots, etc.
  • a driverless car is an example of a vehicle.
  • trajectory refers to a path or route generated to navigate from a first spatiotemporal location to second spatiotemporal location.
  • first spatiotemporal location is referred to as the initial or starting location and the second spatiotemporal location is referred to as the goal or goal position or goal location.
  • the spatiotemporal locations correspond to real world locations.
  • the spatiotemporal locations are pick up or drop-off locations to pick up or drop-off persons or goods.
  • sensor(s) includes one or more hardware components that detect information about the environment surrounding the sensor.
  • Some of the hardware components can include sensing components (e.g., image sensors, biometric sensors), transmitting and/or receiving components (e.g., laser or radio frequency wave transmitters and receivers), electronic components such as analog-to-digital converters, a data storage device (such as a RAM and/or a nonvolatile storage), software or firmware components and data processing components such as an ASIC (application-specific integrated circuit), a microprocessor and/or a microcontroller.
  • sensing components e.g., image sensors, biometric sensors
  • transmitting and/or receiving components e.g., laser or radio frequency wave transmitters and receivers
  • electronic components such as analog-to-digital converters
  • a data storage device such as a RAM and/or a nonvolatile storage
  • software or firmware components and data processing components such as an ASIC (application-specific integrated circuit)
  • microprocessor and/or a microcontroller e.
  • a“scene description” is a data structure (e.g., list) or data stream that includes one or more classified or labeled objects detected by one or more sensors on the AV vehicle or provided by a source external to the AV.
  • One or more includes a function being performed by one element, a function being performed by more than one element, e.g., in a distributed fashion, several functions being performed by one element, several functions being performed by several elements, or any combination of the above.
  • first, second, etc. are, in some instances, used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact.
  • the term“if’ is, optionally, construed to mean“when” or“upon” or“in response to determining” or“in response to detecting,” depending on the context.
  • the phrase“if it is determined” or“if [a stated condition or event] is detected” is, optionally, construed to mean“upon determining” or“in response to determining” or“upon detecting [the stated condition or event]” or“in response to detecting [the stated condition or event],” depending on the context.
  • an AV system refers to the AV along with the array of hardware, software, stored data, and data generated in real-time that supports the operation of the AV.
  • the AV system is incorporated within the AV.
  • the AV system may be spread across several locations.
  • some of the software of the AV system may be implemented on a cloud computing environment similar to cloud computing environment 300 described below with respect to FIG. 3.
  • this document describes technologies applicable to any vehicles that have one or more autonomous capabilities including fully autonomous vehicles, highly autonomous vehicles, and conditionally autonomous vehicles, such as so-called Level 5,
  • Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems which is incorporated by reference in its entirety, for more details on the classification of levels of autonomy in vehicles).
  • Vehicles with Autonomous Capabilities may attempt to control the steering or speed of the vehicles.
  • the technologies descried in this document also can be applied to partially autonomous vehicles and driver assisted vehicles, such as so-called Level 2 and Level 1 vehicles (see SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems).
  • One or more of the Level 1, 2, 3, 4 and 5 vehicle systems may automate certain vehicle operations (e.g., steering, braking, and using maps) under certain operating conditions based on processing of sensor inputs.
  • the technologies described in this document can benefit vehicles in any levels, ranging from fully autonomous vehicles to human-operated vehicles.
  • an AV system 120 operates the AV 100 along a trajectory 198 through an environment 190 to a destination 199 (sometimes referred to as a final location) while avoiding objects (e.g., natural obstructions 191, vehicles 193, pedestrians 192, cyclists, and other obstacles) and obeying rules of the road (e.g., rules of operation or driving preferences).
  • objects e.g., natural obstructions 191, vehicles 193, pedestrians 192, cyclists, and other obstacles
  • rules of the road e.g., rules of operation or driving preferences
  • the AV system 120 includes devices 101 that are instrumented to receive and act on operational commands from the computer processors 146.
  • computing processors 146 are similar to the processor 304 described below in reference to FIG. 3.
  • Examples of devices 101 include a steering control 102, brakes 103, gears, accelerator pedal, windshield wipers, side-door locks, window controls, and turn- indicators.
  • the AV system 120 includes sensors 121 for measuring or inferring properties of state or condition of the AV 100, such as the AV’s position, linear and angular velocity and acceleration, and heading (e.g., an orientation of the leading end of AV 100).
  • sensors 121 are GPS, inertial measurement units (IMU) that measure both vehicle linear accelerations and angular rates, wheel speed sensors for measuring or estimating wheel slip ratios, wheel brake pressure or braking torque sensors, engine torque or wheel torque sensors, and steering angle and angular rate sensors.
  • IMU inertial measurement units
  • the sensors 121 also include sensors for sensing or measuring properties of the AV’s environment.
  • sensors for sensing or measuring properties of the AV’s environment For example, monocular or stereo video cameras 122 in the visible light, infrared or thermal (or both) spectra, LiDAR 123, RADAR, ultrasonic sensors, time-of-flight (TOF) depth sensors, speed sensors, temperature sensors, humidity sensors, and precipitation sensors.
  • monocular or stereo video cameras 122 in the visible light, infrared or thermal (or both) spectra LiDAR 123, RADAR, ultrasonic sensors, time-of-flight (TOF) depth sensors, speed sensors, temperature sensors, humidity sensors, and precipitation sensors.
  • TOF time-of-flight
  • the AV system 120 includes a data storage unit 142 and memory 144 for storing machine instructions associated with computer processors 146 or data collected by sensors 121.
  • the data storage unit 142 is similar to the ROM 308 or storage device 310 described below in relation to FIG. 3.
  • memory 144 is similar to the main memory 306 described below.
  • the data storage unit 142 and memory 144 store historical, real-time, and/or predictive information about the environment 190.
  • the stored information includes maps, driving performance, traffic congestion updates or weather conditions.
  • data relating to the environment 190 is transmitted to the AV 100 via a communications channel from a remotely located database 134.
  • the AV system 120 includes communications devices 140 for communicating measured or inferred properties of other vehicles’ states and conditions, such as positions, linear and angular velocities, linear and angular accelerations, and linear and angular headings to the AV 100.
  • communications devices 140 include Vehicle-to-Vehicle (V2V) and Vehicle-to-Infrastructure (V2I) communication devices and devices for wireless
  • V2V Vehicle-to-Vehicle
  • V2I Vehicle-to-Infrastructure
  • V2X Vehi cl e-to-Every thing
  • the communication devices 140 include communication interfaces. For example, wired, wireless, WiMAX, Wi-Fi, Bluetooth, satellite, cellular, optical, near field, infrared, or radio interfaces.
  • the communication interfaces transmit data from a remotely located database 134 to AV system 120.
  • the remotely located database 134 is embedded in a cloud computing environment 200 as described in FIG. 2.
  • the communication interfaces 140 transmit data collected from sensors 121 or other data related to the operation of AV 100 to the remotely located database 134.
  • communication interfaces 140 transmit information that relates to teleoperations to the AV 100.
  • the AV 100 communicates with other remote (e.g., “cloud”) servers 136.
  • the remotely located database 134 also stores and transmits digital data (e.g., storing data such as road and street locations). Such data may be stored on the memory 144 on the AV 100, or transmitted to the AV 100 via a communications channel from the remotely located database 134.
  • digital data e.g., storing data such as road and street locations.
  • the remotely located database 134 stores and transmits historical information about driving properties (e.g., speed and acceleration profiles) of vehicles that have previously traveled along trajectory 198 at similar times of day. Such data may be stored on the memory 144 on the AV 100, or transmitted to the AV 100 via a communications channel from the remotely located database 134.
  • driving properties e.g., speed and acceleration profiles
  • Computing devices 146 located on the AV 100 algorithmically generate control actions based on both real-time sensor data and prior information, allowing the AV system 120 to execute its autonomous driving capabilities.
  • the AV system 120 may include computer peripherals 132 coupled to computing devices 146 for providing information and alerts to, and receiving input from, a user (e.g., an occupant or a remote user) of the AV 100.
  • peripherals 132 are similar to the display 312, input device 314, and cursor controller 316 discussed below in reference to FIG. 3.
  • the coupling may be wireless or wired. Any two or more of the interface devices may be integrated into a single device.
  • FIG. 2 shows an example“cloud” computing environment.
  • Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services).
  • configurable computing resources e.g. networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services.
  • one or more large cloud data centers house the machines used to deliver the services provided by the cloud.
  • FIG. 2 the cloud computing
  • cloud data centers 204a, 204b, and 204c that are interconnected through the cloud 202.
  • Data centers 204a, 204b, and 204c provide cloud computing services to computer systems 206a, 206b, 206c, 206d, 206e, and 206f connected to cloud 202.
  • the cloud computing environment 200 includes one or more cloud data centers.
  • a cloud data center for example the cloud data center 204a shown in FIG. 2, refers to the physical arrangement of servers that make up a cloud, for example the cloud 202 shown in FIG. 2, or a particular portion of a cloud.
  • servers can be physically arranged in the cloud datacenter into rooms, groups, rows, and racks.
  • a cloud datacenter has one or more zones, which include one or more rooms of servers. Each room has one or more rows of servers, and each row includes one or more racks. Each rack includes one or more individual server nodes.
  • Servers in zones, rooms, racks, and/or rows may be arranged into groups based on physical infrastructure requirements of the datacenter facility, which include power, energy, thermal, heat, and/or other requirements.
  • the server nodes are similar to the computer system described in FIG. 3.
  • the data center 204a has many computing systems distributed through many racks.
  • the cloud 202 includes cloud data centers 204a, 204b, and 204c along with the network and networking resources (for example, networking equipment, nodes, routers, switches, and networking cables) that interconnect the cloud data centers 204a, 204b, and 204c and help facilitate the computing systems’ 206a-f access to cloud computing services.
  • the network and networking resources for example, networking equipment, nodes, routers, switches, and networking cables
  • the network represents any combination of one or more local networks, wide area networks, or internetworks coupled using wired or wireless links deployed using terrestrial or satellite connections. Data exchanged over the network, is transferred using any number of network layer protocols, such as Internet Protocol (IP), Multiprotocol Label Switching (MPLS), Asynchronous Transfer Mode (ATM), Frame Relay, etc.
  • IP Internet Protocol
  • MPLS Multiprotocol Label Switching
  • ATM Asynchronous Transfer Mode
  • Frame Relay etc.
  • IP Internet Protocol
  • MPLS Multiprotocol Label Switching
  • ATM Asynchronous Transfer Mode
  • Frame Relay etc.
  • different network layer protocols are used at each of the underlying sub-networks.
  • the network represents one or more interconnected internetworks, such as the public Internet.
  • the computing systems 206a-f or cloud computing services consumers are connected to the cloud 202 through network links and network adapters.
  • the computing systems 206a-f are implemented as various computing devices, for example servers, desktops, laptops, tablet, smartphones, Internet of Things (IoT) devices, autonomous vehicles (including, cars, drones, shuttles, trains, buses, etc.) and consumer electronics.
  • IoT Internet of Things
  • autonomous vehicles including, cars, drones, shuttles, trains, buses, etc.
  • consumer electronics consumer electronics.
  • the computing systems 206a-f may also be implemented in or as a part of other systems.
  • FIG. 3 shows a computer system 300.
  • the computer system 300 is a special purpose computing device.
  • the special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination.
  • ASICs application-specific integrated circuits
  • FPGAs field programmable gate arrays
  • Such special- purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques.
  • the special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, network devices or any other device that incorporates hard- wired and/or program logic to implement the techniques.
  • the computer system 300 may include a bus 302 or other communication mechanism for communicating information, and a hardware processor 304 coupled with a bus 302 for processing information.
  • the hardware processor 304 may be, for example, a general-purpose microprocessor.
  • the computer system 300 also includes a main memory 306, such as a random-access memory (RAM) or other dynamic storage device, coupled to the bus 302 for storing information and instructions to be executed by processor 304.
  • the main memory 306 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor 304.
  • Such instructions when stored in non-transitory storage media accessible to the processor 304, render the computer system 300 into a special-purpose machine that is customized to perform the operations specified in the instructions.
  • the computer system 300 further includes a read only memory (ROM) 308 or other static storage device coupled to the bus 302 for storing static information and instructions for the processor 304.
  • ROM read only memory
  • a storage device 310 such as a magnetic disk, optical disk, or solid-state drive is provided and coupled to the bus 302 for storing information and instructions.
  • the computer system 300 may be coupled via the bus 302 to a display 312, such as a cathode ray tube (CRT), a liquid crystal display (LCD), plasma display, light emitting diode (LED) display, or an organic light emitting diode (OLED) display for displaying information to a computer user.
  • a display 312 such as a cathode ray tube (CRT), a liquid crystal display (LCD), plasma display, light emitting diode (LED) display, or an organic light emitting diode (OLED) display for displaying information to a computer user.
  • An input device 314, including alphanumeric and other keys, is coupled to bus 302 for communicating information and command selections to the processor 304.
  • a cursor controller 316 such as a mouse, a trackball, a touch-enabled display, or cursor direction keys for communicating direction information and command selections to the processor 304 and for controlling cursor movement on the display 312.
  • This input device typically has two degrees of freedom in two axes, a first axis (e.g., x-axis) and a second axis (e.g., y-axis), that allows the device to specify positions in a plane.
  • a first axis e.g., x-axis
  • a second axis e.g., y-axis
  • the techniques herein are performed by the computer system 300 in response to the processor 304 executing one or more sequences of one or more instructions contained in the main memory 306. Such instructions may be read into the main memory 306 from another storage medium, such as the storage device 310. Execution of the sequences of instructions contained in the main memory 306 causes the processor 304 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.
  • Non-volatile media includes, for example, optical disks, magnetic disks, or solid-state drives, such as the storage device 310.
  • Volatile media includes dynamic memory, such as the main memory 306.
  • Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid-state drive, magnetic tape, or any other magnetic data storage medium, a CD- ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NV-RAM, or any other memory chip or cartridge.
  • Storage media is distinct from but may be used in conjunction with transmission media.
  • Transmission media participates in transferring information between storage media.
  • transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise the bus 302.
  • transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infrared data communications.
  • Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 304 for execution.
  • the instructions may initially be carried on a magnetic disk or solid-state drive of a remote computer.
  • the remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem.
  • a modem local to the computer system 300 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal.
  • An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on the bus 302.
  • the bus 302 carries the data to the main memory 306, from which processor 304 retrieves and executes the instructions.
  • the instructions received by the main memory 306 may optionally be stored on the storage device 310 either before or after execution by processor 304.
  • the computer system 300 also includes a communication interface 318 coupled to the bus 302.
  • the communication interface 318 provides a two-way data communication coupling to a network link 320 that is connected to a local network 322.
  • the communication interface 318 may be an integrated service digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line.
  • the communication interface 318 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN.
  • LAN local area network
  • Wireless links may also be implemented.
  • the communication interface 318 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
  • the network link 320 typically provides data communication through one or more networks to other data devices.
  • the network link 320 may provide a connection through the local network 322 to a host computer 324 or to a cloud data center or equipment operated by an Internet Service Provider (ISP) 326.
  • the ISP 326 in turn provides data communication services through the world-wide packet data communication network now commonly referred to as the“Internet” 328.
  • the local network 322 and Internet 328 both use electrical, electromagnetic or optical signals that carry digital data streams.
  • the signals through the various networks and the signals on the network link 320 and through the communication interface 318, which carry the digital data to and from the computer system 300, are example forms of transmission media.
  • the network 320 may contain or may be a part of the cloud 202 described above.
  • the computer system 300 can send messages and receive data, including program code, through the network(s), the network link 320, and the communication interface 318.
  • the computer system 300 may receive code for processing.
  • the received code may be executed by the processor 304 as it is received, and/or stored in storage device 310, or other non-volatile storage for later execution.
  • Autonomous Vehicle Architecture
  • FIG. 4 shows an example architecture 400 for an autonomous vehicle (e.g., the AV 100 shown in FIG. 1).
  • the architecture 400 includes a perception module 402, a planning module 404, a control module 406, a localization module 408, and a database module 410.
  • Each module plays a role in the operation of the AV 100.
  • the modules 402, 404, 406, 408, and 410 may be part of the AV system 120 shown in FIG. 1.
  • the planning module 404 receives data representing a destination 412 and determines data representing a route 414 that can be traveled by the AV 100 to reach (e.g., arrive at) the destination 412. In order for the planning module 404 to determine the data representing the route 414, the planning module 404 receives data from the perception module 402, the localization module 408, and the database module 410.
  • the perception module 402 identifies nearby physical objects using one or more sensors 121, e.g., as also shown in FIG. 1.
  • the objects are classified (e.g., grouped into types such as pedestrian, bicycle, automobile, traffic sign, etc.) and a scene description including the classified objects 416 is provided to the planning module 404.
  • the planning module 404 also receives data representing the AV position 418 from the localization module 408.
  • the localization module 408 determines the AV position by using data from the sensors 121 and data from the database module 410 (e.g., a geographic data) to calculate a position.
  • the localization module 408 might use data from a GNSS (Global Navigation Satellite System) sensor and geographic data to calculate a longitude and latitude of the AV.
  • GNSS Global Navigation Satellite System
  • data used by the localization module 408 includes high-precision maps of the roadway geometric properties, maps describing road network connectivity properties, maps describing roadway physical properties (such as traffic speed, traffic volume, the number of vehicular and cyclist traffic lanes, lane width, lane traffic directions, or lane marker types and locations, or combinations of them), and maps describing the spatial locations of road features such as crosswalks, traffic signs or other travel signals of various types.
  • the control module 406 receives the data representing the route 414 and the data representing the AV position 418 and operates the control functions 420a-c (e.g., steering, throttling, braking, ignition) of the AV in a manner that will cause the AV 100 to travel the route 414 to the destination 412. For example, if the route 414 includes a left turn, the control module 406 will operate the control functions 420a-c in a manner such that the steering angle of the steering function will cause the AV 100 to turn left and the throttling and braking will cause the AV 100 to pause and wait for passing pedestrians or vehicles before the turn is made.
  • the control functions 420a-c e.g., steering, throttling, braking, ignition
  • FIG. 5 shows an example of inputs 502a-d (e.g., sensors 121 shown in FIG. 1) and outputs 504a-d (e.g., sensor data) that may be used by the perception module 402 (FIG. 4).
  • One input 502a is a LiDAR (Light Detection and Ranging) system (e.g., LiDAR 123 shown in FIG. 1).
  • LiDAR is a technology that uses light (e.g., bursts of light such as infrared light) to obtain data about physical objects in its line of sight.
  • a LiDAR system produces LiDAR data as output 504a.
  • LiDAR data may be collections of 3D or 2D points (also known as a point clouds) that are used to construct a representation of the environment 190.
  • RADAR is a technology that uses radio waves to obtain data about nearby physical objects. RADAR can obtain data about objects not within the line of sight of a LiDAR system.
  • a RADAR system 502b produces RADAR data as output 504b.
  • RADAR data may be one or more radio frequency electromagnetic signals that are used to construct a representation of the environment 190.
  • a camera system uses one or more cameras (e.g., digital cameras using a light sensor such as a charge-coupled device [CCD]) to obtain information about nearby physical objects.
  • a camera system produces camera data as output 504c.
  • Camera data often takes the form of image data (e.g., data in an image data format such as RAW, JPEG, PNG, etc.).
  • the camera system has multiple independent cameras, e.g., for the purpose of stereopsis (stereo vision), which enables the camera system to perceive depth.
  • stereopsis stereo vision
  • the camera system may be configured to“see” objects far, e.g., up to a kilometer or more ahead of the AV. Accordingly, the camera system may have features such as sensors and lenses that are optimized for perceiving objects that are far away.
  • TLD traffic light detection
  • a TLD system uses one or more cameras to obtain information about traffic lights, street signs, and other physical objects that provide visual navigation information.
  • a TLD system produces TLD data as output 504d.
  • TLD data often takes the form of image data (e.g., data in an image data format such as RAW, JPEG, PNG, etc.).
  • a TLD system differs from another system incorporating a camera in that a TLD system uses a camera with a wide field of view (e.g., using a wide- angle lens or a fish-eye lens) in order to obtain information about as many physical objects providing visual navigation information as possible, so that the AV 100 has access to all relevant navigation information provided by these objects.
  • the viewing angle of the TLD system may be about 120 degrees or more.
  • outputs 504a-d can be combined using a sensor fusion technique.
  • the individual outputs 504a-d can be provided to other systems of the AV 100 (e.g., provided to a planning module 404 as shown in FIG. 4), or the combined output can be provided to the other systems, either in the form of a single combined output or multiple combined outputs of the same type (e.g., using the same combination technique or combining the same outputs or both) or different types type (e.g., using different respective combination techniques or combining different respective outputs or both).
  • an early fusion technique is used.
  • An early fusion technique is characterized by combining outputs before one or more data processing steps are applied to the combined output.
  • a late fusion technique is used.
  • a late fusion technique is characterized by combining outputs after one or more data processing steps are applied to the individual outputs.
  • FIG. 6 shows an example of a LiDAR system 602 (e.g., the input 502a shown in FIG. 5).
  • the LiDAR system 602 emits light 604a-c from a light emitter 606 (e.g., a laser transmitter).
  • a light emitter 606 e.g., a laser transmitter.
  • Light emitted by a LiDAR system is typically not in the visible spectrum; for example, infrared light is often used.
  • Some of the light 604b emitted encounters a physical object 608 (e.g., a vehicle) and reflects back to the LiDAR system 602.
  • a physical object 608 e.g., a vehicle
  • the LiDAR system 602 also has one or more light detectors 610, which detect the reflected light.
  • One or more data processing systems associated with the LiDAR system can generate an image 612 representing the field of view 614 of the LiDAR system.
  • the image 612 includes information that represents the boundaries 616 of a physical object 608. In this way, the image 612 can be used to determine the boundaries 616 of one or more physical objects near an AV.
  • FIG. 7 shows the LiDAR system 602 in operation.
  • the AV 100 receives both camera system output 504c in the form of an image 702 and LiDAR system output 504a in the form of LiDAR data points 704.
  • the data processing systems of the AV 100 can compare the image 702 to the data points 704.
  • a physical object 706 identified in the image 702 can also be identified among the data points 704.
  • the AV 100 can perceive the boundaries of the physical object based on the contour and density of the data points 704.
  • FIG. 8 shows the operation of the LiDAR system 602 in additional detail.
  • the AV 100 can detect the boundary of a physical object based on characteristics of the data points detected by the LiDAR system 602. As shown in FIG. 8, a flat object, such as the ground 802, will reflect light 804a-d emitted from a LiDAR system 602 in a consistent manner. Put another way, because the LiDAR system 602 emits light using consistent spacing, the ground 802 will reflect light back to the LiDAR system 602 with the same consistent spacing. As the AV 100 travels over the ground 802, the LiDAR system 602 will continue to detect light reflected by the next valid ground point 806 if nothing is obstructing the road.
  • the AV 100 can determine that the object 808 is present.
  • FIG. 9 shows a block diagram 900 of the relationships between inputs and outputs of a planning module 404 (e.g., as shown in FIG. 4).
  • the output of a planning module 404 is a route 902 from a start point 904 (e.g., source location or initial location), and an end point 906 (e.g., destination or final location).
  • the route 902 is typically defined by one or more segments.
  • a segment may be a distance to be traveled over at least a portion of a street, road, highway, driveway, or other physical area appropriate for automobile travel.
  • the route 902 may include“off-road” segments such as unpaved paths or open fields.
  • a planning module also outputs lane-level route planning data 908.
  • the lane-level route planning data 908 is used to traverse segments of the route 902 based on conditions of the segment at a particular time. For example, if the route 902 includes a multi-lane highway, the lane-level route planning data 908 may include path planning data 910 that the AV 100 can use to choose a lane among the multiple lanes, e.g., based on whether an exit is approaching, whether one or more of the lanes have other vehicles, or other factors that may vary over the course of a few minutes or less. Similarly, the lane-level route planning data 908 may include speed constraints 912 specific to a segment of the route 902.
  • the inputs to the planning module 404 can include database data 914 (e.g., from the database module 410 shown in FIG. 4), current location data 916 (e.g., the AV position 418 shown in FIG. 4), destination data 918 (e.g., for the destination 412 shown in FIG. 4), and object data 920 (e.g., the classified objects 416 as perceived by the perception module 402 as shown in FIG. 4).
  • the database data 914 includes rules used in planning.
  • Rules are specified using a formal language, e.g., using Boolean logic. In any given situation encountered by the AV 100, at least some of the rules will apply to the situation. A rule applies to a given situation if the rule has conditions that are met based on information available to the AV 100, e.g., information about the surrounding environment. Rules can have priority. For example, a rule that says,“if the road is a freeway, move to the leftmost lane” can have a lower priority than“if the exit is approaching within a mile, move to the rightmost lane.”
  • FIG. 10 shows a directed graph 1000 used in path planning, e.g., by the planning module 404 (FIG. 4).
  • a directed graph 1000 like the one shown in FIG. 10 can be used to determine a path between any start point 1002 and end point 1004.
  • the distance separating the start point 1002 and end point 1004 may be relatively large (e.g., in two different metropolitan areas) or may be relatively small (e.g., two intersections abutting a city block or two lanes of a multi-lane road).
  • the directed graph 1000 has nodes l006a-d representing different locations between the start point 1002 and end point 1004 that could be occupied by an AV 100.
  • the nodes l006a-d may represent segments of roads.
  • the nodes l006a-d may represent different positions on that road.
  • the directed graph 1000 may include information at varying levels of granularity.
  • a directed graph having high granularity may also be a subgraph of another directed graph having a larger scale.
  • a directed graph in which the start point 1002 and end point 1004 are far away may have most of its information at a low granularity and is based on stored data, but can also include some high granularity information for the portion of the graph that represents physical locations in the field of view of the AV 100.
  • the nodes l006a-d are distinct from objects l008a-b which cannot overlap with a node.
  • the objects l008a-b may represent regions that cannot be traversed by automobile, e.g., areas that have no streets or roads.
  • the objects l008a-b may represent physical objects in the field of view of the AV 100, e.g., other automobiles, pedestrians, or other entities with which the AV 100 cannot share physical space.
  • any of the objects l008a-b can be a static object (e.g., an object that does not change position such as a street lamp or utility pole) or a dynamic object (e.g., an object that is capable of changing position such as a pedestrian or other car).
  • a static object e.g., an object that does not change position such as a street lamp or utility pole
  • a dynamic object e.g., an object that is capable of changing position such as a pedestrian or other car.
  • the nodes l006a-d are connected by edges lOlOa-c. If two nodes l006a-b are connected by an edge lOlOa, it is possible for an AV 100 to travel between one node l006a and the other node l006b, e.g., without having to travel to an intermediate node before arriving at the other node l006b.
  • edges lOlOa-c are often bidirectional, in the sense that an AV 100 can travel from a first node to a second node, or from the second node to the first node. However, edges lOlOa-c can also be unidirectional, in the sense that an AV 100 can travel from a first node to a second node, but cannot travel from the second node to the first node. Edges lOlOa- c are unidirectional when they represent, for example, one-way streets, individual lanes of a street, road, or highway, or other features that can only be traversed in one direction due to legal or physical constraints.
  • the planning module 404 can use the directed graph 1000 to identify a path 1012 made up of nodes and edges between the start point 1002 and end point 1004.
  • An edge lOlOa-c has an associated cost l0l4a-b.
  • the cost l0l4a-b is a value that represents the resources that will be expended if the AV 100 chooses that edge.
  • a typical resource is time. For example, if one edge lOlOa represents a physical distance that is twice that as another edge lOlOb, then the associated cost l0l4a of the first edge lOlOa may be twice the associated cost l0l4b of the second edge lOlOb. Other factors that can affect time include expected traffic, number of intersections, speed limit, etc. Another typical resource is fuel economy. Two edges lOlOa-b may represent the same physical distance, but one edge lOlOa may require more fuel than another edge lOlOb, e.g., because of road conditions, expected weather, etc.
  • the planning module 404 When the planning module 404 identifies a path 1012 between the start point 1002 and end point 1004, the planning module 404 typically chooses a path optimized for cost, e.g., the path that has the least total cost when the individual costs of the edges are added together.
  • two or more redundant planning modules 404 can be included in an AV, as described in further detail in reference to FIGS. N1-N3. Autonomous Vehicle Control
  • FIG. 11 shows a block diagram 1100 of the inputs and outputs of a control module 406 (e.g., as shown in FIG. 4).
  • a control module operates in accordance with a controller 1102 which includes, for example, one or more processors (e.g., one or more computer processors such as microprocessors or microcontrollers or both), short-term and/or long-term data storage (e.g., memory random-access memory or flash memory or both), and instructions stored in memory that carry out operations of the controller 1102 when the instructions are executed (e.g., by the one or more processors).
  • processors e.g., one or more computer processors such as microprocessors or microcontrollers or both
  • short-term and/or long-term data storage e.g., memory random-access memory or flash memory or both
  • instructions stored in memory that carry out operations of the controller 1102 when the instructions are executed (e.g., by the one or more processors).
  • the controller 1102 receives data representing a desired output 1104.
  • the desired output 1104 typically includes a velocity, e.g., a speed and heading.
  • the desired output 1104 can be based on, for example, data received from a planning module 404 (e.g., as shown in FIG. 4).
  • the controller 1102 produces data usable as a throttle input 1106 and a steering input 1108.
  • the throttle input 1106 represents the magnitude in which to engage the throttle (e.g., acceleration control) of an AV 100, e.g., by engaging the steering pedal, or engaging another throttle control, to achieve the desired output 1104.
  • the throttle input 1106 also includes data usable to engage the brake (e.g., deceleration control) of the AV 100.
  • the steering input 1108 represents a steering angle, e.g., the angle at which the steering control (e.g., steering wheel, steering angle actuator, or other functionality for controlling steering angle) of the AV should be positioned to achieve the desired output 1104.
  • the controller 1102 receives feedback that is used in adjusting the inputs provided to the throttle and steering. For example, if the AV 100 encounters a disturbance 1110, such as a hill, the measured speed 1112 of the AV 100 may lower below the desired output speed. Any measured output 1114 can be provided to the controller 1102 so that the necessary adjustments can be performed, e.g., based on the differential 1113 between the measured speed and desired output.
  • the measured output 1114 can include measured position 1116, measured velocity 1118, (including speed and heading), measured acceleration 1120, and other outputs measurable by sensors of the AV 100.
  • FIG. 12 shows a block diagram 1200 of the inputs, outputs, and components of the controller 1102.
  • the controller 1102 has a speed profiler 1202 which affects the operation of a throttle/brake controller 1204.
  • the speed profiler 1202 can instruct the throttle/brake controller 1204 to engage acceleration or engage deceleration using the throttle/brake 1206 depending on, e.g., feedback received by the controller 1102 and processed by the speed profiler 1202.
  • the controller 1102 also has a lateral tracking controller 1208 which affects the operation of a steering controller 1210.
  • the lateral tracking controller 1208 can instruct the steering controller 1204 to adjust the position of the steering angle actuator 1212 depending on, e.g., feedback received by the controller 1102 and processed by the lateral tracking controller 1208.
  • the controller 1102 receives several inputs used to determine how to control the throttle/brake 1206 and steering angle actuator 1212.
  • a planning module 404 provides information used by the controller 1102, for example, to choose a heading when the AV 100 begins operation and to determine which road segment to traverse when the AV 100 reaches an intersection.
  • a localization module 408 provides information to the controller 1102 describing the current location of the AV 100, for example, so that the controller 1102 can determine if the AV 100 is at a location expected based on the manner in which the throttle/brake 1206 and steering angle actuator 1212 are being controlled.
  • the controller 1102 may also receive information from other inputs 1214, e.g., information received from databases, computer networks, etc.
  • a system 1300 useable to operate an AV includes two or more different autonomous vehicle operations subsystems (S) l3l0a, 1310b, each of the two or more different AV operations subsystems, e.g., l3l0a, being redundant with another of the two or more different AV operations subsystems, e.g., 13 lOb (e.g., redundant versions of the perception module 402, localization module 408, planning module 404, control module 406 or combinations (e.g., pipelines) of at least two of these modules).
  • S autonomous vehicle operations subsystems
  • two different AV operations subsystems l3l0a, 1310b are redundant with each other because each can independently operate the AV in the common/shared region of an operating envelope.
  • Partial redundancy/overlap is applicable, for example, when the modules being integrated with each other address at least one common aspect of AV operation.
  • at least one of the two or more different AV operations subsystems is configured to provide additional AV operations solutions that are not redundant with the AV operations solutions of at least one other of the two or more different AV operations subsystems.
  • either of the two subsystems, or both can provide functionality that is not redundant with that provided by the other, in addition to the redundant aspects of operation.
  • At least one of the two or more different AV operations subsystems is configured to only provide AV operations solutions that are redundant with the AV operations solutions of at least one other of the two or more different AV operations subsystems.
  • the different AV operations subsystems l3l0a, 1310b can be implemented as one or more software algorithms that perform respective functions of the AV operations subsystems l3l0a, 1310b. In some implementations, the different AV operations subsystems l3l0a, 1310b can be implemented as integrated circuits that perform respective functions of the AV operations subsystems l3l0a, 13 lOb.
  • the system 1300 includes an output mediator (A) 1340 coupled with the two or more different AV operations subsystems l3l0a, 1310b through respective connections l3l7a, 13 l7b.
  • the output mediator 1340 can be implemented as one or more software algorithms that perform the function of the output mediator 1340.
  • the output mediator 1340 can be implemented as one or more integrated circuits that perform the function of the output mediator 1340.
  • the output mediator 1340 is configured to manage AV operation outputs from the two or more different AV operations subsystems l3l0a, 13 lOb.
  • the output mediator 1340 can be implemented as an AV operations arbiter that selects one output over another. In general, there are numerous ways for an output mediator to select a“winning” AV operation output from among AV operations outputs of two or more redundant AV operations subsystems.
  • an output mediator can be operated in accordance with“substitution redundancy”
  • this arbiter technique can be applied, based on the“l-out-of-2” (loo2) assumption, when the failure modes of the two redundant AV operations subsystems are independent.
  • the output mediator selects the AV operation output from the one of the two redundant AV operations subsystems which is still working. If AV operation outputs are available from both redundant AV operations subsystems, the output mediator must select one of the two outputs. However, the two AV operation outputs may be quite different from each other.
  • the output mediator can be configured as an“authoritative” arbiter to be capable of selecting the appropriate AV operation output based on predetermined criteria.
  • the output mediator can be configured as a trivial arbiter which uses a“bench-warming” approach to perform the selection.
  • one of the two redundant AV operations subsystems is a designated backup, so its output is ignored unless the prime AV operations subsystem fails. For this reason, the bench-warming approach cannot leverage the backup AV operations subsystem.
  • an output mediator can be operated in accordance with “majority redundancy” in multiple-redundant AV operations subsystems.
  • this arbiter technique can be applied, based on the“triple-redundancy” assumption, when the algorithm / model used to obtain the AV operation outputs is considered to be correct, while its HW and/or SW implementation may be faulty in one of the three redundant AV operations subsystems.
  • the output mediator selects the AV operation output from two of the three redundant AV operations subsystems (or equivalently, drops the AV operation output that is different from the other two).
  • the output mediator can be configured as a trivial arbiter.
  • this approach can provide a form of fault detection, e.g., it can identify the one among the three redundant AV operations subsystems in which the algorithm / model’s HW and/or SW implementation is faulty, the majority redundancy approach does not necessarily increase failure tolerance.
  • an output mediator can be operated in accordance with “mobbing redundancy” when, for N > 3 redundant AV operations subsystems, each of the AV operations subsystems uses different models.
  • the output mediator will select the winning AV operation output as the one that is common among the largest number of AV operations subsystems.
  • the output mediator can be configured as a trivial arbiter.
  • the AV operation output is common between a subset of AV operations subsystems not necessarily because it is the“most correct”, but because the different models used by the subset of AV operations subsystems are highly correlated.
  • the“minority report” may be the correct one, i.e., the AV operation output produced by a number of AV operations subsystems that is smaller than the subset of AV operations subsystems.
  • synergistic redundancy can be used to create highly redundant architectures with improved performance and reliability. It will be shown that the approach of synergistic redundancy can be applied to complex algorithms for perception and decision making. Synergistic redundancy can be applied to most engineering problems, e.g., when a particular engineering problem is cast as a problem-solving algorithm, which includes a proposal mechanism and a scoring mechanism.
  • Table 1 below shows that planning, e.g., as performed by the planning module 404 of the AV architecture 400 - also see FIGs. 9-10, and perception, e.g., as performed by the perception module 402 of the AV architecture 400 - also see FIGs. 5-8, fit the same proposal mechanism & scoring mechanism pattern.
  • FIG. 13 shows that each AV operations subsystem l3l0a,b of the two or more different AV operations subsystems l3l0a, 1310b includes a solution proposer (SP) l3l2a,b configured to propose solutions for AV operation based on current input data, and a solution scorer (SS) l3l4a,b configured to evaluate the proposed solutions for AV operation based on one or more cost assessments.
  • the solution proposer l3l2a,b is coupled through respective connection l3l la,b with corresponding sensors of the system 1300 or another AV operations subsystem, which is disposed“up-stream” on the same stack (or pipeline) as the AV operations subsystem l3l0a,b, to receive the current input data.
  • the solution scorer l3l4a,b of at least one of the two or more different AV operations subsystems l3l0a, 131 Ob is configured to evaluate both the proposed solutions from the solution proposer l3l2a,b of the at least one of the two or more different AV operations subsystems l3l0a, 131 Ob and at least one of the proposed solutions from the solution proposer l3l2b,a of at least one other of the two or more different AV operations subsystems l3l0a, 131 Ob.
  • An intra-inter-stack connection 1315 e.g., implemented as a multi-lane bus, is configured to couple the solution proposer l3l2a,b of an AV operations subsystem l3l0a,b with both the solution scorer l3l4a,b of the same AV operations subsystem l3l0a,b and the solution scorer l3l4b,a of another AV operations subsystem l3l0b,a.
  • the solution scorer l3l4a,b of the AV operations subsystem l3l0a,b is configured to operate in the following manner.
  • a solution scorer l3l4a,b of an AV operations subsystem l3l0a,b receives, through the intra-inter-stack connection 1315, a proposed solution from a solution proposer l3l2a,b of the same AV operations subsystem l3l0a,b, also referred to as the local (or native) proposed solution, and another proposed solution from a solution proposer l3l2b,a of another AV operations subsystem l3l0b,a, also referred to as the remote (or non-native or cross-platform) proposed solution.
  • the solution scorer l3l4a,b performs some translation/normalization between the remotely and locally proposed solutions. In this manner, the solution scorer l3l4a,b can evaluate both the locally proposed solution and the remotely proposed solution using a local cost function (or metric). For instance, the solution scorer l3l4a,b applies the local cost function to both the locally proposed solution and the remotely proposed solution to determine their respective costs. Finally, the solution scorer l3l4a,b selects between the locally proposed solution and the remotely proposed solution as the one which has the smaller of the costs evaluated based on the local cost function. The selected solution corresponds to a proposed model (locally or remotely generated) that maximizes the likelihood of the current input data if the proposed model is correct.
  • a proposed model locally or remotely generated
  • the solution scorer l3l4a provides the solution it has selected, as the AV operations subsystem !3l0a’s output, to the output mediator 1340 through the connection 1317&
  • the solution scorer 13 l4b provides the solution it has selected, as the AV operations subsystem 13 lOb’s output, to the output mediator 1340 through the connection 13 l7b.
  • the output mediator 1340 can implement one or more selection processes, described in detail in the next section, to select either one of the AV operations subsystem l3l0a’s output or the AV operations subsystem 13 lOb’s output.
  • the output mediator 1340 provides, through output connection 1347, a single output from the two or more redundant operations subsystems l3l0a, l3l0b, in the form of the selected output, to one or more“down-stream” modules of the system 1300, or one or more actuators of the AV which use the system 1300.
  • FIG. 14 shows an example of a system 1400 which represents a modified version of the system 400, the modification being that the perception module 402 was replaced by redundant perception modules l4l0a, l4l0b and perception-output mediator 1440.
  • the perception modules l4l0a, l4l0b were implemented like the AV operations subsystems l3l0a, l3l0b, and the perception-output mediator 1440 was implemented like the output mediator 1340.
  • Solutions proposed by the solution proposers (implemented like the solution proposers l3l2a, 13 l2b) of the redundant perception modules l4l0a, l4l0b include world view proposals, for instance.
  • the perception subsystems l4l0a, l4l0b can receive data from one or more sensors 121, e.g., LiDAR, RADAR, video/image data in visible, infrared, ultraviolet or other wavelengths, ultrasonic, time-of-flight (TOF) depth, speed, temperature, humidity, and/or precipitation sensors, and from a database (DB) 410.
  • the respective solution proposers of the redundant perception modules l4l0a, l4l0b can generate respective world-view proposals based on, e.g., perception proposal mechanisms, such as bottom-up perception (object detection), top- down task-driven attention, priors, occupancy grids, etc., as described above in connection with FIGs.
  • the solution proposers of the redundant perception modules l4l0a, l4l0b can generate their respective world-view proposals based on information from current sensor signals received from corresponding subsets of sensors of the AV, for instance. Additionally, respective solution scorers (implemented like the solution scorers l3l4a,
  • the solution scorer of each perception module l4l0a,b uses a respective perception-cost function to evaluate at least one world-view proposal generated by the solution proposer of the perception module !4l0a,b, and at least one world- view proposal received through the intra-inter-stack connection 1415 from the solution proposer of another perception module l4l0b,a.
  • the intra-inter-stack connection 1415 is implemented like the intra-inter-stack connection 1315.
  • the solution scorer of the perception module l4l0a selects one between the world-view proposal from the solution proposer of the perception module l4l0a and the world-view proposal from the solution proposer of the perception module l4l0b, the selected one corresponding to a minimum of a first perception-cost function, and provides the selected world-view 1416a as the perception module l4l0a’s output to the perception-output mediator 1440.
  • the solution scorer of the perception module l4l0b selects one between the world-view proposal from the solution proposer of the perception module 141 Ob and the world-view proposal from the solution proposer of the perception module l4l0a, the selected one corresponding to a minimum of a second perception-cost function different from the first perception-cost function, and provides the selected world-view 1416b as the perception module 141 Ob’s output to the perception-output mediator 1440.
  • a world view proposal avoids being tied to a non-optimal solution in the perception module l4l0a,b, e.g., due to convergence to a local minimum during optimization, because the other perception module l4l0b,a uses different initial conditions, or because the other perception module l4l0b,a uses a different world-view forming approach, even if it were to use the exact same initial conditions.
  • the perception-output mediator 1440 selects one of the two world views l4l6a, l4l6b and provides it down-stream to the planning module 404 and the localization module 408 where it will be used to determine route 414, and AV position 418, respectively.
  • FIG. 15 shows an example of a system 1500 which represents a modified version of the system 400, the modification being that the planning module 404 was replaced by redundant planning modules l5l0a, 15 lOb and planning-output mediator 1540.
  • the planning modules l5l0a, 15 lOb were implemented like the AV operations subsystems l3l0a, l3l0b, and the planning-output mediator 1540 was implemented like the output mediator 1340.
  • Solutions proposed by the solution proposers (implemented like the solution proposers l3l2a, 13 l2b) of the redundant planning modules include route proposals, for instance. As noted above in connection with FIGs.
  • route proposals also referred to as candidate routes
  • route proposals can be determined by inferring behavior of the AV and other AVs in accordance with physics of the environment, and driving rules for a current location 418 (provided by the localization module 408), e.g., by using sampling based methods and/or optimization based methods.
  • the respective solution proposers of the redundant planning modules l5l0a, 15 lOb can generate route proposals, based on, e.g., planning proposal mechanisms, such as random sampling, MPC, deep learning, pre-defmed primitives, etc.
  • the solution proposers of the redundant planning modules l5l0a, 15 lOb can generate their respective solution proposals based on information from a current world-view 416 received from a perception module 402 of the AV, the AV’s position 418, a destination 412 and other data from a database (DB) 410, for instance.
  • respective solution scorers (implemented like the solution scorers l3l4a, 13 l4b) of the redundant planning modules l5l0a, 1510b can evaluate the route proposals based on one or more cost assessments, e.g., using cost function evaluation of respective planning-cost functions, such as trajectory scoring based on trajectory length, safety, comfort, etc.
  • the solution scorer of each planning module l5l0a,b evaluates at least one route proposal generated by the solution proposer of the planning module l5l0a,b, and at least one route proposal received through the intra-inter-stack connection 1515 from the solution proposer of another planning module l5l0b,a.
  • the intra-inter-stack connection 1515 is implemented like the intra-inter stack connection 1315.
  • the solution scorer of the planning module l5l0a selects one between the route proposal from the solution proposer of the planning module l5l0a and the route proposal from the solution proposer of the planning module l5l0b, the selected one corresponding to a minimum of a first planning-cost function, and provides the selected route l5l4a as the planning module l5l0a’s output to the planning-output mediator 1540.
  • the solution scorer of the planning module 1510b selects one between the route proposal from the solution proposer of the planning module 1510b and the route proposal from the solution proposer of the planning module l5l0a, the selected one corresponding to a minimum of a second planning-cost function different from the first planning-cost function, and provides the selected route l5l4b as the planning module l5l0b’s output to the planning-output mediator 1540.
  • a route proposal avoids being tied to a non-optimal solution in the planning module l5l0a,b, e.g., due to convergence to a local minimum during optimization, because the other planning module l5l0b,a uses different initial conditions, or because the other planning module l5l0b,a uses a different route forming approach, even if it were to use the exact same initial conditions.
  • the planning-output mediator 1540 selects one of the two routes l5l4a, 15 l4b and provides it down-stream to the controller module 406 where it will be used to determine control signals for actuating a steering actuator B2l0a, a throttle actuator 420b, and/or a brake actuator 420c.
  • these examples correspond to the different AV operations subsystems l3l0a, l3l0b, etc., that are being used at a single level of operation. In some
  • synergistic redundancy can be implemented for two or more operations pipelines, also referred to as stacks, each of which including multiple levels of operation, e.g., a first level of operation corresponding to perception followed by a second level of operation corresponding to planning. Note that levels of operation in a pipeline are also referred to as stages of the pipeline.
  • a system 1600 useable to operate an AV includes two or more operations pipelines l602a, l602b, each of which including two or more levels l604a, l604b. Synergistic redundancy can be implemented in the system 1600 with cross-evaluation at one or more levels. As described in detail below,
  • AV operations subsystems configured like the AV operations subsystems l3l0a, 1310b are used at various operational stages l604a, l604b of each of two or more operations pipelines l602a, l602b, such that each stage l604a,b in the pipeline l602a,b includes at least one solution scorer configured to evaluate proposed solutions from at least one solution proposer in the stage l604a,b and proposed solutions from the same stage l604a,b of another pipeline 1602b, a.
  • the system 1600 includes an output mediator 1640 connected to the last stage of each of the two or more operations pipelines l602a, l602b.
  • a first pipeline of operational stages l602a includes a first stage l604a implemented as a first AV operations subsystem l6l0a, and a second stage l604b implemented as a second AV operations subsystem l620a.
  • a second pipeline of operational stages l602b includes the first stage l604a implemented as another first AV operations subsystem l6l0b and the second stage l604b implemented as another second AV operations subsystem l620b. Note that, in some implementations, the first AV operations subsystem 1610b and the second AV operations subsystem l620b of the second pipeline l602b share a power supply.
  • the first AV operations subsystem l6l0b and the second AV operations subsystem l620b of the second pipeline l602b have their own respective power supplies.
  • the second AV operations subsystem l620a of the first pipeline l602a communicates with the first AV operations subsystem 1610a of the first pipeline 1602a through an intra-stack connection 162 la, and with the output mediator 1640 through an end-stack connection l627a
  • the second AV operations subsystem l620b of the second pipeline l602b communicates with the first AV operations subsystem l6l0b of the second pipeline l602b through another intra stack connection 162 lb, and with the output mediator 1640 through another end-stack connection l627b.
  • first intra-inter-stack connection 1615 also the second AV operations subsystem l620a of the first pipeline l602a and the second AV operations subsystem l620b of the second pipeline l602b communicate with each other through a second intra-inter-stack connection 1625, as described below.
  • the first AV operations subsystem l6l0a of the first pipeline l602a includes a solution proposer l6l2a and a solution scorer l6l4a.
  • the solution proposer l6l2a of the first AV operations subsystem l6l0a of the first pipeline l602a is configured to use first input data available to the first AV operations subsystem l6l0a of the first pipeline l602a to propose first stage solutions.
  • the first AV operations subsystem 1610b of the second pipeline l602b includes another solution proposer l6l2b and another solution scorer l6l4b.
  • the other solution proposer l6l2b of the first AV operations subsystem l6l0b of the second pipeline l602b is configured to use second input data available to the first AV operations subsystem l6l0b of the second pipeline l602b to propose alternative first stage solutions.
  • the solution scorer l6l4a of the first AV operations subsystem l6l0a of the first pipeline l602a is configured to evaluate the first stage solutions from the solution proposer l6l2a of the first AV operations subsystem l6l0a of the first pipeline l602a and the alternative first stage solutions from the other solution proposer l6l2b of the first AV operations subsystem l6l0b of the second pipeline l602b.
  • the solution scorer l6l4a of the first AV operations subsystem l6l0a of the first pipeline l602a is configured to provide, to the second AV operations subsystem l620a of the first pipeline l602a, first pipeline l602a’s first stage output which consists of, for each first stage solution and corresponding alternative first stage solution, one of either the first stage solution or the alternative first stage solution.
  • the solution scorer l6l4b of the first AV operations subsystem l6l0b of the second pipeline l602b is configured to evaluate the first stage solutions from the solution proposer l6l2a of the first AV operations subsystem 1610a of the first pipeline l602a and the alternative first stage solutions from the other solution proposer l6l2b of the first AV operations subsystem l6l0b of the second pipeline l602b.
  • the solution scorer l6l4b of the first AV operations subsystem 1610b of the second pipeline l602b is configured to provide, to the second AV operations subsystem l620b of the second pipeline l602b, second pipeline l602b’s first stage output which consists of, for each first stage solution and corresponding alternative first stage solution, one of either the first stage solution or the alternative first stage solution.
  • the second AV operations subsystem l620a of the first pipeline l602a includes a solution proposer l622a and a solution scorer l624a.
  • the solution proposer l622a of the second AV operations subsystem l620a of the first pipeline l602a is configured to use first pipeline l602a’s first stage output from the solution scorer l6l4a of the first AV operations subsystem l6l0a of the first pipeline l602a to propose second stage solutions.
  • the second AV operations subsystem l620b of the second pipeline l602b includes another solution proposer l622b and another solution scorer l624b.
  • the other solution proposer l622b of the second AV operations subsystem l620b of the second pipeline l602b is configured to use second pipeline l602b’s first stage output from the solution scorer l6l4b of the first AV operations subsystem 1610b of the second pipeline l602b to propose alternative second stage solutions.
  • the solution scorer l624a of the second AV operations subsystem l620a of the first pipeline l602a is configured to evaluate the second stage solutions from the solution proposer l622a of the second AV operations subsystem l620a of the first pipeline l602a and the alternative second stage solutions from the other solution proposer l622b of the second AV operations subsystem l620b of the second pipeline l602b.
  • the solution scorer l624a of the AV operations subsystem l620a of the first pipeline l602a is configured to provide, to the output mediator 1640, first pipeline l602a’s second stage output which consists of, for each second stage solution and corresponding alternative second stage solution, one of either the second stage solution or the alternative second stage solution.
  • the solution scorer l624b of the second AV operations subsystem l620b of the second pipeline l602b is configured to evaluate the second stage solutions from the solution proposer l622a of the second AV operations subsystem l620a of the first pipeline l602a and the alternative second stage solutions from the other solution proposer l622b of the second AV operations subsystem l620b of the second pipeline l602b.
  • the solution scorer l624b of the second AV operations subsystem l620b of the second pipeline l602b is configured to provide, to the output mediator 1640, second pipeline l602b’s second stage output which consists of, for each second stage solution and corresponding alternative second stage solution, one of either the second stage solution or the alternative second stage solution.
  • the output mediator 1640 can implement one or more selection processes, described in detail in the next section, to select either one of the first pipeline l602a’s second stage output or the second pipeline l602b’s second stage output. In this manner, the output mediator 1640 provides, through output connection 1647, a single output from the two or more redundant pipelines l602a, l602b, in the form of the selected output, to one or more “down-stream” modules of the system 1600, or one or more actuators of the AV which use the system 1600.
  • the system 1600 which implements cross-stack evaluation of intermediate solution proposals from AV modules that share a region of the operating envelope, e.g., implemented as the first AV operations subsystems l6l0a, l6l0b, or as the second AV operations subsystems l620a, l620b, ensure higher failure tolerance, and potentially improved solutions in multi-level AV operation stacks/pipelines, during AV operation. These benefits will become apparent based on the examples described below.
  • FIG. 17 shows an example of a system 1700 which represents a modified version of the system 400, the modification being that a two-stage pipeline having a first stage implemented as the perception module 402 and a second stage implemented as the planning module 404 was replaced by two redundant two-stage pipelines and an output mediator 1740.
  • the first two-stage pipeline has a first stage implemented as a first perception module l7l0a and a second stage implemented as a first planning module l720a
  • the second two-stage pipeline has the first stage implemented as a second perception module l7l0b and the second stage implemented as a second planning module l720b.
  • the perception modules l7l0a and 1710b are implemented like the AV operations subsystems l6l0a of the first pipeline l602a, and l6l0b of the second pipeline l602b. Operation of the perception modules l7l0a and 1710b is similar to the operation of the perception modules l4l0a, l4l0b described above in connection with FIG. 14. For instance, solutions proposed by the solution proposers (implemented like the solution proposers l6l2a, l6l2b) of the perception modules l7l0a, l7l0b include world-view proposals.
  • the solution proposers of the perception modules l7l0a, 1710b can generate their respective world-view proposals based on information from current sensor signals received from corresponding subsets of sensors 121 associated with the system 1700, for instance. Additionally, respective solution scorers (implemented like the solution scorers l6l4a, l6l4b) of the perception modules l7l0a, l7l0b can evaluate the world-view proposals based on one or more cost assessments, e.g., based on evaluation of respective perception-cost functions.
  • the solution scorer of each perception module l7l0a,b evaluates at least one world-view proposal generated by the solution proposer of the perception module l7l0a,b, and at least one world-view proposal received through the intra-inter-stack connection 1715 from the solution proposer of another perception module l7l0b,a.
  • the solution scorer of the first perception module !7l0a selects one between the world-view proposal from the solution proposer of the first perception module l7l0a and the world-view proposal from the solution proposer of the second perception module l7l0b, the selected one corresponding to a minimum of a first perception-cost function, and provides, down-stream the first pipeline, the selected world view l7l6a as the first perception module l7l0a’s output to the first planning module l720a.
  • the solution scorer of the second perception module 171 Ob selects one between the world-view proposal from the solution proposer of the second perception module 171 Ob and the world-view proposal from the solution proposer of the first perception module l7l0a, the selected one corresponding to a minimum of a second perception-cost function different from the first perception-cost function, and provides, down-stream the second pipeline, the selected world-view 1716b as the second perception module 171 Ob’s output to the second planning module l720b.
  • the planning modules l720a, l720b are implemented like the AV operations subsystems l620a of the first pipeline l602a, and l620b of the second pipeline l602b, while the output mediator 1740 is implemented like the output mediator 1640.
  • Operation of the planning modules l720a and l720b and of the output mediator 1740 is similar to the operation of the planning modules l5l0a, 1510b and of the planning-output mediator 1540 described above in connection with FIG. 15.
  • solutions proposed by the solution proposers (implemented like the solution proposers l622a, l622b) of the planning modules l720a, l720b include route proposals.
  • the solution proposer of the first planning module l720a generates its route proposal based on the world view l7l6a output by the first perception module l7l0a
  • the solution proposer of the second planning module l720b generates its route proposal based on the alternative world view 1716b output by the second perception module l7l0b, while both can generate their respective route proposals based on the destination 412, the AV position 418 received from the localization module 408, and further on information received from the database (DB) 410.
  • respective solution scorers (implemented like the solution scorers l624a, l624b) of the planning modules l720a, l720b can evaluate the route proposals based on one or more cost assessments, e.g., based on evaluation of respective planning-cost functions.
  • the solution scorer of each planning module l720a,b evaluates at least one route proposal generated by the solution proposer of the planning module l720a,b, and at least one route proposal received through the intra-inter-stack connection 1725 from the solution proposer of another planning module 1720b, a.
  • the intra-inter-stack connections 1715, 1725 are implemented like the intra-inter-stack connections 1615, 1625.
  • the solution scorer of the first planning module l720a selects one between the route proposal from the solution proposer of the first planning module l720a and the route proposal from the solution proposer of the second planning module l720b, the selected one corresponding to a minimum of a first planning-cost function, and provides the selected route l7l4a as the first pipeline’s planning stage output to the output mediator 1740.
  • the solution scorer of the second planning module l720b selects one between the route proposal from the solution proposer of the second planning module l720b and the route proposal from the first solution proposer of the planning module l720a, the selected one corresponding to a minimum of a second planning-cost function different from the first planning-cost function, and provides the selected route l7l4b as the second pipeline’s planning stage output to the output mediator 1740.
  • the output mediator 1740 selects one of the two routes l7l4a, l7l4b and provides it down-stream to the controller module 406 where it will be used to determine control signals for actuating a steering actuator B2l0a, a throttle actuator 420b, and a brake actuator 420c.
  • cross-evaluation of world-view proposals generated by redundant pipelines can be implemented at the perception stage, and cross-evaluation of route proposals generated by the redundant pipelines can be implemented at the planning stage.
  • cross-evaluation of the world-view proposals generated by redundant pipelines can be implemented at the perception stage, without implementing cross-evaluation of the route proposals generated by the redundant pipelines at the planning stage.
  • this can be accomplished by using an intra-inter-stack connection 1725 which can be automatically reconfigured to function as a pair of intra-module connections, one connecting the route proposer and the route scorer of the first planning module l720a, and the other one connecting the route proposer and the route scorer of the second planning module l720b.
  • the cross-evaluation of the route proposals generated by the redundant pipelines at the planning stage can be restored by automatically reconfiguring the pair of intra-module connections to function as the intra- inter-stack connection 225.
  • cross-evaluation of the route proposals generated by redundant pipelines can be implemented at the planning stage, without implementing cross evaluation of the world-view proposals generated by the redundant pipelines at the perception stage.
  • this can be accomplished by using an intra-inter-stack connection 1715 which can be automatically reconfigured to function as a pair of intra module connections, one connecting the world-view proposer and the world-view scorer of the first perception module l7l0a, and the other one connecting the world-view proposer and the world-view scorer of the second perception module !7l0b.
  • the cross-evaluation of the world-view proposals generated by the redundant pipelines at the perception stage can be restored by automatically reconfiguring the pair of intra-module connections to function as the intra-inter-stack connection 215.
  • These situations which correspond to standard loo2 substitution redundancy, can be accomplished by reconfiguring both intra-inter-stack connections 1715, 1715 as described above, and by using an authoritative output mediator 1740.
  • FIG. 18 shows an example of a system 1800 which represents a modified version of the system 400, the modification being that a two-stage pipeline having a first stage implemented as the planning module 404 and a second stage implemented as the controller module 406 was replaced by two redundant two-stage pipelines and an output mediator 1840.
  • the first two-stage pipeline has a first stage implemented as a first planning module l720a and a second stage implemented as a first controller module l8l0a
  • the second two-stage pipeline has the first stage implemented as a second planning module l720b and the second stage implemented as a second controller module l8l0b.
  • the planning modules l720a, l720b are implemented like the AV operations subsystems l6l0a of the first pipeline l602a, and l6l0b of the second pipeline l602b.
  • solutions proposed by the solution proposers (implemented like the solution proposers l6l2a, l6l2b) of the planning modules l720a, l720b include route proposals.
  • the solution proposers of the planning modules l720a, l720b generate their respective route proposals based on the world view 416 output by the perception module 402, on the AV position 418 received from the localization module 408, the destination 412, and further on information received from the database (DB) 410.
  • respective solution scorers (implemented like the solution scorers l6l4a, 1614b) of the planning modules l720a, l720b can evaluate the route proposals based on one or more cost assessments, e.g., based on evaluation of respective planning-cost functions.
  • the solution scorer of each planning module l720a,b evaluates at least one route proposal generated by the solution proposer of the planning module l720a,b, and at least one route proposal received through the intra-inter-stack connection 1725 from the solution proposer of another planning module 1720b, a.
  • the solution scorer of the first planning module l720a selects one between the route proposal from the solution proposer of the first planning module l720a and the route proposal from the solution proposer of the second planning module l720b, the selected one corresponding to a minimum of a first planning-cost function, and provides, down-stream the first pipeline, the selected route l8l4a as the first planning module l720a’s output to the first controller module l8l0a.
  • the solution scorer of the second planning module l720b selects one between the route proposal from the solution proposer of the second planning module l720b and the route proposal from the solution proposer of the first planning module l720a, the selected one corresponding to a minimum of a second planning-cost function different from the first planning-cost function, and provides, down-stream the second pipeline, the selected route 18 l4b as the second planning module l720b’s output to the second controller module 181 Ob.
  • controller modules l8l0a, 1810b are implemented like the AV operations subsystems l620a of the first pipeline l602a, and l620b of the second pipeline l602b, while the output mediator 1840 is implemented like the output mediator 1640.
  • solutions proposed by the solution proposers (implemented like the solution proposers l622a, l622b) of the controller modules l8l0a, l8l0b include control-signal proposals.
  • the solution proposer of the first controller module l8l0a generates its control-signal proposal based on the route l8l4a output by the first planning module l720a
  • the solution proposer of the second controller module 1810b generates its control-signal proposal based on the alternative route 18 l4b output by the second planning module l720b, while both can generate their respective control-signal proposals based on the AV position 418 received from the localization module 408.
  • respective solution scorers (implemented like the solution scorers l624a, l624b) of the controller modules l8l0a, l8l0b can evaluate the control -signal proposals based on one or more cost assessments, e.g., based on evaluation of respective control-cost functions.
  • the solution scorer of each controller module l8l0a,b evaluates at least one control-signal proposal generated by the solution proposer of the controller module l8l0a,b, and at least one control-signal proposal received through the intra-inter-stack connection 1815 from the solution proposer of another controller module l8l0b,a.
  • the intra-inter-stack connection 1815 is implemented like the intra-inter-stack connection 1625.
  • the solution scorer of the first controller module l8l0a selects one between the control-signal proposal from the solution proposer of the first controller module l8l0a and the control-signal proposal from the solution proposer of the second controller module l8l0b, the selected one corresponding to a minimum of a first control-cost function, and provides the selected control-signal as the first pipeline’s controller stage output to the output mediator 1840.
  • the solution scorer of the controller module 1810b selects one between the control-signal proposal from the solution proposer of the second controller module l8l0b and the control-signal proposal from the solution proposer of the first controller module l8l0a, the selected one corresponding to a minimum of a second control-cost function different from the first control-cost function, and provides the selected control-signal as the second pipeline’s controller stage output to the output mediator 1840.
  • a control-signal proposal avoids being tied to a non- optimal solution in the control module l8l0a,b, e.g., due to convergence to a local minimum during optimization, because the other control module l8l0b,a uses different initial conditions, or because the other control module 181 Ob, a uses a different control-signal forming approach, even if it were to use the exact same initial conditions.
  • the output mediator 1840 selects one of the two control signals and provides it down-stream for actuating a steering actuator B2l0a, a throttle actuator 420b, and/or a brake actuator 420c.
  • FIG. 19 shows an example of a system 1900 which represents a modified version of the system 400, the modification being that a two-stage pipeline having a first stage implemented as the localization module 408 and a second stage implemented as the controller module 406 was replaced by two redundant two-stage pipelines and an output mediator 1840.
  • the first two-stage pipeline has a first stage implemented as a first localization module l9l0a and a second stage implemented as a first controller module l8l0a
  • the second two-stage pipeline has the first stage implemented as a second localization module l9l0b and the second stage implemented as a second controller module 1810b.
  • the localization modules 1910a, l9l0b are implemented like the AV operations subsystems l6l0a of the first pipeline l602a, and l6l0b of the second pipeline l602b.
  • solutions proposed by the solution proposers (implemented like the solution proposers l6l2a, l6l2b) of the localization modules l9l0a, l9l0b include AV position proposals.
  • the solution proposers of the localization modules l9l0a, l9l0b generate their respective AV position proposals based on information from current sensor signals received from corresponding subsets of sensors 121 associated with the system 1900, on the world view 416 output by the perception module 402, and further on information received from a database (DB) 410.
  • DB database
  • the AV position proposals may be constrained by known factors, such as roads, legal/illegal positions, altitude, etc.
  • respective solution scorers (implemented like the solution scorers l6l4a, l6l4b) of the localization modules l9l0a, l9l0b can evaluate the AV location proposals based on one or more cost assessments, e.g., based on evaluation of respective localization-cost functions.
  • the solution scorer of each localization module !9l0a,b evaluates at least one AV location proposal generated by the solution proposer of the localization module l9l0a,b, and at least one AV location proposal received through the intra-inter-stack connection 1915 from the solution proposer of another localization module l9l0b,a.
  • the intra-inter stack connections 1915 is implemented like the intra-inter-stack connection 1615.
  • the solution scorer of the first localization module l9l0a selects one between the AV position proposal from the solution proposer of the first localization module 1910a and the AV position proposal from the solution proposer of the second localization module 191 Ob, the selected one corresponding to a minimum of a first localization-cost function, and provides, down-stream the first pipeline, the selected AV position 1918a as the first localization module l9l0a’s output to the first controller module l8l0a.
  • the solution scorer of the second localization module 191 Ob selects one between the AV location proposal from the solution proposer of the second localization module 191 Ob and the AV location proposal from the solution proposer of the first localization module 1910a, the selected one corresponding to a minimum of a second localization-cost function different from the first localization-cost function, and provides, down-stream the second pipeline, the selected AV position 1918b as the second localization module l9l0b’s output to the second controller module 181 Ob.
  • an AV position proposal avoids being tied to a non-optimal solution in the localization module l9l0a,b, e.g., due to convergence to a local minimum during optimization, because the other localization module 191 Ob, a uses different initial conditions, or because the other localization module l9l0b,a uses a different AV location forming approach, even if it were to use the exact same initial conditions.
  • the first controller module l8l0a at the second stage of the first pipeline and the second controller module 1810b at the second stage of the second pipeline are implemented and operated as described above in connection with FIG. 18, except that the solution proposer of the first controller module l8l0a generates its control-signal proposal based on the AV position 1918a output by the first localization module l9l0a, and the solution proposer of the second controller module 18 lOb generates its control-signal proposal based on the alternative route 1918b output by the second localization module l9l0b.
  • the output mediator 1840 is implemented and operated as described above in connection with FIG. 18.
  • the first and second redundant pipelines l602a, l602b each can include two or more stages l604a, l604b.
  • a system 2000 useable to operate an AV includes the two operations pipelines l602a, l602b, each of which including three stages l604a, l604b, 2004c.
  • the system 2000 also includes the output mediator 1640 connected to the last stage of each of the two operations pipelines l602a, l602b. Synergistic redundancy can be implemented in the system 2000 with cross-evaluation at each of the three stages, as described below.
  • the first and second stages l604a, l604b of the system 2000 were implemented as described above in connection with system 1600.
  • the third stage 2004c of the first pipeline l602a was implemented as a third AV operations subsystem 2030a
  • the third stage 2004c of the second pipeline l602b was implemented as another third AV operations subsystem 2030b.
  • the first AV operations subsystem 1610b, the second AV operations subsystem l620b, and the third AV operations subsystem 2030b of the second pipeline l602b share a power supply.
  • the first AV operations subsystem l6l0b, the second AV operations subsystem l620b, and the third AV operations subsystem 2030b of the second pipeline l602b each have their own power supply.
  • the third AV operations subsystem 2030a communicates with the first AV operations subsystem l6l0a through an intra-stack connection 161 la of the first pipeline l602a
  • the other third AV operations subsystem 2030b communicates with the other first AV operations subsystem l6l0b through another intra-stack connection 161 lb of the second pipeline l602b.
  • the third AV operations subsystem 2030a of the first pipeline l602a and the third AV operations subsystem 2030b of the second pipeline l602b communicate with each other through a third intra-inter-stack connection 2035, as described below.
  • the third AV operations subsystem 2030a of the first pipeline l602a includes a solution proposer 2032a and a solution scorer 2034a.
  • the solution proposer 2032a of the third AV operations subsystem 2030a of the first pipeline l602a is configured to use first input data available to the third AV operations subsystem 2030a of the first pipeline l602a to propose third stage solutions.
  • the third AV operations subsystem 2030b of the second pipeline l602b includes another solution proposer 2032b and another solution scorer 2034b.
  • the other solution proposer 2032b of the third AV operations subsystem 2030b of the second pipeline l602b is configured to use second input data available to the third AV operations subsystem 2030b of the second pipeline l602b to propose alternative third stage solutions.
  • the solution scorer 2034a of the third AV operations subsystem 2030a of the first pipeline l602a is configured to evaluate the third stage solutions from the solution proposer 2032a of the third AV operations subsystem 2030a of the first pipeline l602a and the alternative first stage solutions from the other solution proposer 2032b of the third AV operations subsystem 2030b of the second pipeline l602b.
  • the solution scorer 2034a of the third AV operations subsystem 2030a of the first pipeline l602a is configured to provide, to the first AV operations subsystem l6l0a of the first pipeline l602a, first pipeline l602a’s third stage output which consists of, for each third stage solution and corresponding alternative third stage solution, one of either the third stage solution or the alternative third stage solution.
  • the solution scorer 2034b of the third AV operations subsystem 2030b of the second pipeline l602b is configured to evaluate the third stage solutions from the solution proposer 2032a of the third AV operations subsystem 2030a of the first pipeline l602a and the alternative third stage solutions from the other solution proposer 2032b of the third AV operations subsystem 2030b of the second pipeline l602b.
  • the solution scorer 2034b of the third AV operations subsystem 2030b of the second pipeline l602b is configured to provide, to the first AV operations subsystem l6l0b of the second pipeline l602b, second pipeline l602b’s third stage output which consists of, for each third stage solution and corresponding alternative third stage solution, one of either the third stage solution or the alternative third stage solution.
  • the first stage l604a was implemented, as the first AV operations subsystem l6l0a for the first pipeline l602a, and as the other first AV operations subsystem l6l0b for the second pipeline l602b.
  • the first AV operations subsystem l6l0a of the first pipeline l602a, and the other first AV operations subsystem l6l0b of the second pipeline l602b were implemented and operated as described above in connection with FIG.
  • the second stage l604b was implemented as the second AV operations subsystem l620a for the first pipeline l602a, and as the other second AV operations subsystem l620b for the second pipeline l602b.
  • the second AV operations subsystem l620a of the first pipeline l602a, and the other second AV operations subsystem l620b of the second pipeline l602b were implemented and operated as described above in connection with FIG. 16.
  • the output mediator 1640 was implemented and operated as described above in connection with FIG. 16.
  • FIG. 21 shows an example of a system 2100 which represents a modified version of the system 400, one modification being that a three-stage pipeline having a beginning stage implemented as the perception module 402, an intermediate stage implemented as the planning module 404, and a last stage implemented as the control module 406 was replaced by a first pair of redundant three-stage pipelines and an output mediator 1840.
  • the first three-stage pipeline has a beginning stage implemented as a first perception module l7l0a, an intermediate stage implemented as a first planning module l720a, and a last stage implemented as a first control module l8l0a
  • the second three-stage pipeline has the beginning stage implemented as a second perception module l7l0b, the intermediate stage implemented as a second planning module l720b, and the last stage implemented as a second control module 1810b.
  • the perception modules l7l0a, 1710b were implemented like the AV operations subsystems 2030a of the first pipeline l602a, and 2030b of the second pipeline l602b.
  • the solution proposers of the perception modules l7l0a, l7l0b generate their respective world-view proposals based on information from current sensor signals received from corresponding subsets of sensors 121 associated with the system 2100, for instance.
  • the solution scorer of each perception module l7l0a,b evaluates at least one world-view proposal generated by the solution proposer of the perception module l7l0a,b, and at least one world-view proposal received through the intra-inter-stack connection 1715 from the solution proposer of another perception module l7l0b,a, selects the one of these two world-view proposals which minimizes a perception-cost function corresponding to the perception module l7l0a,b, and outputs, down-stream the respective pipeline, the selected proposal as a world-view l7l6a,b to the planning module l720a,b.
  • the planning modules l720a, l720b were implemented and operated as described above in connection with FIG. 17.
  • the solution proposers of the planning modules l720a, l720b generate their respective route proposals based on the world-views 1716a, l7l6b from the respective perception modules l7l0a, 1710b, for instance.
  • the solution scorer of each planning module l720a,b evaluates at least one route proposal generated by the solution proposer of the planning module l720a,b, and at least one route proposal received through the intra-inter-stack connection 1725 from the solution proposer of another planning module l720b,a, selects the one of these two route proposals which minimizes a planning-cost function corresponding to the planning module l720a,b, and outputs, down-stream the respective pipeline, the selected proposal as a route 21 l4a,b to the control module l8l0a,b.
  • control modules l8l0a, 1810b and the output mediator 1840 were implemented and operated as described above in connection with FIG. 18.
  • the solution proposers of the control modules l8l0a, 1810b generate their respective control-signal proposals based on the routes 21 l4a, 21 l4b from the respective planning modules l720a, l720b, for instance.
  • the solution scorer of each control module l8l0a,b evaluates at least one control-signal proposal generated by the solution proposer of the control module l8l0a,b, and at least one control-signal proposal received through the intra- inter-stack connection 1815 from the solution proposer of another control module l8l0b,a, selects the one of these two control-signal proposals which minimizes a control-cost function corresponding to the control module l8l0a,b, and outputs the selected proposal as the control signal to the output mediator 1840.
  • the output mediator 1840 selects one of the two control signals provided by the control modules l8l0a, 1810b and provides it down-stream for actuating a steering actuator B2l0a, a throttle actuator 420b, and/or a brake actuator 420c.
  • Another modification of the system 400 embodied by the system 2100 is that a three-stage pipeline having a beginning stage implemented as the perception module 402, an intermediate stage implemented as the localization module 408, and a last stage implemented as the control module 406 was replaced by a second pair of redundant three-stage pipelines and the output mediator 1840.
  • the first three-stage pipeline has a beginning stage implemented as a first perception module l7l0a, an intermediate stage implemented as a first localization module l9l0a, and a last stage implemented as a first control module l8l0a
  • the second three-stage pipeline has the beginning stage implemented as a second perception module l7l0b, the intermediate stage implemented as a second localization module l9l0b, and the last stage implemented as a second control module l8l0b.
  • the perception modules l7l0a, 1710b are implemented and operated as described above in connection with the first pair of redundant three-stage pipelines of the system 2100, except that each perception module l7l0a,b outputs, down-stream the respective pipeline, the selected proposal as a world-view l7l6a,b to the localization module l9l0a,b.
  • the localization modules !9l0a, 1910b were implemented and operated as described above in connection with FIG. 19.
  • the solution proposers of the localization modules l9l0a, l9l0b generate their respective AV position proposals based on the world-views l7l6a, l7l6b from the respective perception modules l7l0a, l7l0b, for instance.
  • the solution scorer of each localization module l9l0a,b evaluates at least one AV position proposal generated by the solution proposer of the localization module l9l0a,b, and at least one AV position proposal received through the intra-inter-stack connection 1915 from the solution proposer of another localization module l9l0b,a, selects the one of these two AV position proposals which minimizes a localization-cost function corresponding to the localization module l9l0a,b, and outputs, down-stream the respective pipeline, the selected proposal as an AV position 2l l8a,b to the control module l8l0a,b.
  • control modules l8l0a, l8l0b and the output mediator 1840 are implemented and operated as described above in connection with the first pair of redundant three-stage pipelines of the system 2100.
  • Yet another modification of the system 400 embodied by the system 2100 is that a four-stage pipeline having a beginning stage implemented as the perception module 402, a first intermediate stage implemented as the localization module 408, a second intermediate stage implemented as the planning module 404, and a last stage implemented as the control module 406 was replaced by a pair of redundant four-stage pipelines and the output mediator 1840.
  • the first four-stage pipeline has a beginning stage implemented as a first perception module l7l0a, a first intermediate stage implemented as a first localization module l9l0a, a second intermediate stage implemented as a first planning module l720a, and a last stage implemented as a first control module l8l0a
  • the second four-stage pipeline has the beginning stage implemented as a second perception module 1710b, the first intermediate stage implemented as a second localization module l9l0b, the second intermediate stage implemented as a second planning module l720b, and the last stage implemented as a second control module l8l0b.
  • the perception modules l7l0a, l7l0b are implemented as described above in connection with each of the first and second pairs of redundant three-stage pipelines of the system 2100, except that each perception module l7l0a,b outputs, down-stream the respective pipeline, its selected proposal as a world-view l7l6a,b to the localization module l9l0a,b and the planning module l720a,b.
  • the localization modules !9l0a, 1910b were implemented as described above in connection with the second pair of redundant three-stage pipelines of the system 2100, except that each localization module l9l0a,b outputs, down-stream the respective pipeline, its selected proposal as an AV position 21 l8a,b to the control module l8l0a,b and the planning module l720a,b.
  • the planning modules l720a, l720b are implemented as described above in connection with the first pair of redundant three-stage pipelines of the system 2100.
  • control modules l8l0a, l8l0b and the output mediator 1840 are implemented as described above in connection with the first pair of redundant three-stage pipelines of the system 2100.
  • the pair of redundant four-stage pipelines of the system 2100 can be operated using a process 2200 described below in connection with FIGs. 22-23.
  • the first perception module l7l0a receives first sensor signals from a first set of the sensors 121 of an AV, and generates a first world view proposal based on the first sensor signals.
  • the second perception module 1710b receives second sensor signals from a second set of the sensors 121 of the AV, and generates a second world view proposal based on the second sensor signals.
  • the first set of sensors can be different from the second set of sensors.
  • the two sets are partially overlapping, i.e., they can have at least one sensor in common.
  • the two set have no common sensor.
  • the first sensor signals received from the first set of the sensors 121 include one or more lists of objects detected by corresponding sensors of the first set
  • the second sensor signals received from the second set of the sensors 121 include one or more lists of objects detected by corresponding sensors of the second set.
  • these lists are created by the perception modules.
  • the generating of the first world view proposal by the first perception module l7l0a can include creating one or more first lists of objects detected by corresponding sensors of the first set.
  • the generating of the second world view proposal by the second perception module 1710b can include creating one or more second lists of objects detected by corresponding sensors of the second set.
  • the generating of the first world view proposal can be performed by the first perception module l7l0a based on a first perception proposal mechanism.
  • the generating of the second world view proposal can be performed by the second perception module 1710b based on a second perception proposal mechanism different from the first perception proposal mechanism.
  • the second perception module 171 Ob can generate the second world view proposal based on the first perception proposal mechanism to be different than the first world view proposal. That is because the second sensor signals used by the second perception module l7l0b are different than the first sensor signals used by the first perception module l7l0a to generate their respective world view proposals.
  • the first perception module l7l0a selects one between the first world view proposal and the second world view proposal based on a first perception-cost function, and provides the selected one as a first world view 1716a to the first localization module l9l0a.
  • the second perception module 1710b selects one between the first world view proposal and the second world view proposal based on a second perception-cost function, and provides the selected one as a second world view l7l6b to the second localization module l9l0b.
  • the first world view l7l6a provided to the first localization module l9l0a and to the first planning module l720a can include a first object track of one or more objects detected by the first set of sensors.
  • the second world view l7l6b provided to the second localization module l9l0b and to the second planning module l720b can include a second object track of one or more objects detected by the second set of sensors.
  • the first localization module l9l0a receives the first world view 1716a from the first perception module l7l0a, and generates a first AV position proposal based on the first world view l7l6a.
  • the second localization module l9l0b receives the second world view l7l6b from the second perception module 1710b, and generates a second AV position proposal based on the second world view 1716b.
  • the first localization module l9l0a can receive at least a portion of the first sensor signals from the first set of the sensors 121. In this manner, the generating of the first AV position proposal is performed by the first localization module l9l0a based on a combination of the first sensor signals and the first world view l7l6a. Also note that the second localization module 1910b can receive at least a portion of the second sensor signals from the second set of the sensors 121. In this manner, the generating of the second AV position proposal is performed by the second localization module l9l0b based on another combination of the second sensor signals and the second world view l7l6b.
  • the first and second localization modules !9l0a, !9l0b can use one or more localization algorithms including map-based localization, LiDAR map-based localization, RADAR map-based localization, visual map-based localization, visual odometry, and feature-based localization.
  • the generating of the first AV position proposal can be performed by the first localization module l9l0a based on a first localization algorithm.
  • the generating of the second AV position proposal can be performed by the second localization module 1910b based on a second localization algorithm different from the first localization algorithm.
  • the second localization module 1910b can use the first localization algorithm to generate the second AV position proposal and obtain a second AV position proposal that is different than the first AV position proposal.
  • the first localization module l9l0a selects one between the first AV position proposal and the second AV position proposal based on a first localization-cost function, and provides the selected one as a first AV position 2118a to the first planning module l720a.
  • the second localization module l9l0b selects one between the first AV position proposal and the second AV position proposal based on a second localization- cost function, and provides the selected one as a second AV position 2118b to the second planning module l720b.
  • first AV position 21 l8a provided to the first planning module 220a and to the first control module l8l0a can include a first estimate of a current position of the AV
  • second AV position 2118b provided to the second planning module 220b and to the second control module l8l0b can include a second estimate of the current position of the AV.
  • the first planning module l720a receives the first AV position 21 l8a from the first localization module l9l0a, and generates a first route proposal based on the first AV position 2118a.
  • the second planning module l720b receives the second AV position 2118b from the second localization module l9l0b, and generates a second route proposal based on the second AV position 2118b.
  • the first planning module l720a can receive the first world view 1716a from the first perception module l7l0a. In this manner, the generating of the first route proposal is performed by the first planning module l720a based on a combination of the first AV position 21 l8a and the first world view !7l6a. Also note that the second planning module l720b can receive the second world view l7l6b from the second perception module 171 Ob. In this manner, the generating of the second route proposal is performed by the second planning module l720b based on another combination of the second AV position 2118b and the second world view l7l6b.
  • the generating of the first route proposal can be performed by the first planning module l720a based on a first planning algorithm.
  • the generating of the second route proposal can be performed by the second planning module l720b based on a second planning algorithm different from the first planning algorithm.
  • the second planning module l720b can use the first planning algorithm to generate the second route proposal and obtain a second route proposal that is different than the first route proposal.
  • generating the route proposals by the planning modules l720a, l720b can include proposing respective paths between the AV’s current position and a destination 412 of the AV.
  • generating the route proposals by the planning modules l720a, l720b can include inferring behavior of the AV and one or more other vehicles.
  • the behavior is inferred by comparing a list of detected objects with driving rules associated with a current location of the AV. For example, cars drive on the right side of the road in the USA, and the left side of the road in the UK, and are expected to stay on their legal side of the road.
  • the behavior is inferred by comparing a list of detected objects with locations in which vehicles are permitted to operate by driving rules associated with a current location of the vehicle. For example, cars are not allowed to drive on sidewalks, off road, through buildings, etc.
  • the behavior is inferred through a constant velocity or constant acceleration model for each detected object.
  • generating the route proposals by the planning modules l720a, l720b can include proposing respective paths that conform to the inferred behavior and avoid one or more detected objects.
  • the first planning module l720a selects one between the first route proposal and the second route proposal based on a first planning-cost function, and provides the selected one as a first route 21 l4a to the first control module l8l0a.
  • the second planning module 220b selects one between the first route proposal and the second route proposal based on a second planning-cost function, and provides the selected one as a second route 21 l4b to the second control module l8l0b.
  • selecting one between the first route proposal and the second route proposal can include evaluating collision likelihood based on the respective world view l7l6a,b and a behavior inference model.
  • the first control module l8l0a receives the first route 21 l4a from the first planning module l720a, and generates a first control-signal proposal based on the first route 21 l4a.
  • the second control module l8l0b receives the second route 21 l4b from the second planning module l720b, and generates a second control-signal proposal based on the second route 21 l4b.
  • the first control module 1810a can receive the first AV position 2118a from the first localization module l9l0a. In this manner, the generating of the first control- signal proposal is performed by the first control module l8l0a based on a combination of the first AV position 21 l8a and the first route 21 l4a. Also note that the second control module l8l0b can receive the second AV position 2118b from the second localization module 191 Ob. In this manner, the generating of the second control-signal proposal is performed by the second control module 18 lOb based on another combination of the second AV position 2118b and the second route l7l4b
  • the first control module l8l0a selects one between the first control- signal proposal and the second control-signal proposal based on a first control-cost function, and provides the selected one as a first control signal to the output mediator 1840.
  • the second control module 1810b selects one between the first control-signal proposal and the second control-signal proposal based on a second control-cost function, and provides the selected one as a second control signal to the output mediator 1840.
  • the output mediator 1840 receives, or accesses, the first control signal from the first control module l8l0a, and the second control signal from the second control module 1810b.
  • the output mediator 1840 selects one between the first control signal and the second control signal by using selection procedures described in detail in the next section.
  • the output mediator 1840 provides the selected one as a control signal to one or more actuators, e.g., 420a, 420b, 42c of the AV. Ways in which the output mediator 1840 either transmits, or instructs transmission of, the selected control signal to an appropriate actuator of the AV are described in detail in the next section.
  • each scorer l3l4a,b, l6l4a,b, l624a,b, 2034a,b, of respective AV operation subsystems l3l0a,b, l6l0a,b, l620a,b, 2030a,b can adopt a solution proposed by another AV operation subsystems l3l0b,a, l6l0b,a, 1620b, a, 2030b, a if“convinced” of its superiority.
  • the“convincing” includes performing cost function evaluations of the alternative solutions received from proposers l3l2b,a, l6l2b,a, l622b,a, 2032b, a of the other AV operation subsystems side-by-side to the native solution received from the proposers l3l2a,b, l6l2a,b, l622a,b, 2032a, b of its own AV operation subsystem.
  • each of the AV operation subsystems at the same stage of a pipeline performs beter than if the AV operation subsystems could not evaluate each other’s solution proposal. This leads to potentially higher failure tolerance.
  • FIG. 24 shows a system 2400 which achieves the goal of generating and synergistically evaluating N different route proposals, by using N redundant pipelines PLA, PLB, PLC, PLD and an output mediator A.
  • each redundant pipeline PLA, B, C, D includes a first stage implemented as a respective perception module PA, B, C, D, and a second stage implemented as a respective planning module RA, B, C, D.
  • each perception module PA,B,C,D includes a respective solution proposer SPPA,B,C,D and a respective solution scorer SSPA,B,C,D.
  • each planning module RA,B,C,D includes a respective solution proposer SPRA,B,C,D and a respective solution scorer SSRA,B,C,D.
  • the solution scorer SSPA,B,C,D of the perception module PA,B,C,D communicates with the solution proposer SPRA,B,C,D of the planning module RA,B,C,D through a respective intra-stack connection CPR.
  • the solution scorer SSRA, B, C, D of the planning module RA, B, C, D communicates with the output mediator A through a respective end-stack connection CRA.
  • the solution proposer SPPj of each perception module Pj communicates through an intra-inter-stack connection CP with the solution scorer SSPj of the perception module Pj to which it belongs and with respective solution scorers SSPk1j of the remaining perception modules Pk, where j, k e ⁇ A, B, C, D ⁇ .
  • the solution proposer SPPA communicates with the solution scorer SSPA within the same pipeline PLA, and with each of the solution scorers SSPB, SSPC and SSPD across the redundant pipelines, PLB, PLC and PLD, respectively. And so on.
  • the solution proposer SPRj of each planning module Rj communicates through another intra-inter-stack connection CR with the solution scorer SSRj of the planning module Rj to which it belongs and to respective solution scorers SSRk1j of the remaining planning modules Pk, where j, k e ⁇ A, B, C, D ⁇ .
  • the solution proposer SPRA communicates with the solution scorer SSRA within the same pipeline PLA, and with each of the solution scorers SSRB, SSRC and SSRD across the redundant pipelines, PLB, PLC and PLD, respectively. And so on.
  • intra-inter-stack connections CP, CR can be implemented as respective multi-lane buses, e.g., like the intra-inter-stack connections 1315, 1415, 1515, 1615, 1625, 1715, 1725, 1815, 1915, 2035, etc., described above.
  • Synergistic redundancy can be implemented at the perception stage of the system 2400 in the following manner.
  • the solution proposer SPPj of each perception module Pj generates a respective world-view proposal based on available sensor signals from corresponding subsets of sensors associated with the system 2400 (not shown in FIG. 24.)
  • the solution scorer SSPj of each perception module Pj receives, through the intra-inter-stack connection CP, respective world-view proposals from the solution proposer SPPj of the perception module Pj and from the solution proposers SPPk1j of the remaining perception modules Pk, where j, k e ⁇ A, B, C, D ⁇ , and evaluates all the received proposals by using a perception-cost function associated with the solution scorer SSPj.
  • the solution scorer SSPA of the perception module PA evaluates the world view proposals received from the solution proposers SPPA, SPPB, SPPC, SPPD using a first perception-cost function
  • the solution scorer SSPB of the perception module PB evaluates the world view proposals received from the solution proposers SPPA, SPPB, SPPC, SPPD using a second perception-cost function, and so on and so forth.
  • the solution scorer SSPj of each perception module Pj selects as the winning world view the one from among the received world-view proposals which corresponds to the smallest value of the perception-cost function associated with the solution scorer SSPj.
  • the solution scorer SSPA of the perception module PA applies the first perception-cost function to the world view proposals received from the solution proposers SPPA, SPPB, SPPC, SPPD and can determine that a first perception-cost function value corresponding to the world view proposed by the solution proposer SPPB is smaller than first perception-cost function values corresponding to each of the remaining world views proposed by the solution proposers SPPA, SPPC, SPPD. For this reason, the solution scorer SSPA of the perception module PA will provide, through the intra-stack connection CPR of the pipeline PLA, to the solution proposer SPRA of the planning module RA, the world view proposed by the solution proposer SPPB of the perception module PB.
  • the solution scorer SSPB of the perception module PB applies the second perception-cost function to the world view proposals received from the solution proposers SPPA, SPPB, SPPC, SPPD and can determine that a second perception-cost function value corresponding to the world view proposed by the solution proposer SPPB is smaller than second perception-cost function values corresponding to each of the remaining world views proposed by the solution proposers SPPA, SPPC, SPPD.
  • the solution scorer SSPB of the perception module PB will provide, through the intra-stack connection CPR of the pipeline PLB, to the solution proposer SPRB of the planning module RB, the world view proposed by the solution proposer SPPB of the perception module PB. Note that this situation corresponds to the case where the“local solution” wins over multiple“remote solutions.” And so on, and so forth.
  • Synergistic redundancy can be implemented at the planning stage of the system 2400 in the following manner.
  • the solution proposer SPRj of each planning module Rj generates a respective route proposal based on a respective winning world view received, through the intra-stack connection CPR of the pipeline PLj, from the solution scorer SSPj of the perception module Pj.
  • the solution scorer SSRj of each planning module Rj receives, through the intra-inter-stack connection CR, respective route proposals from the solution proposer SPRj of the planning module Rj and from the solution proposers SPRk1j of the remaining planning modules Rk, where j, k e ⁇ A, B, C, D ⁇ .
  • the solution scorer SSRA of the planning module RA evaluates the route proposals received from the solution proposers SPRA, SPRB, SPRC, SPRD using a first planning-cost function
  • the solution scorer SSRB of the planning module RB evaluates the route proposals received from the solution proposers SPRA, SPRB, SPRC, SPRD using a second planning-cost function, and so on and so forth.
  • the solution scorer SSRj of each planning module Rj selects as the winning route the one from among the received route proposals which corresponds to the smallest value of the planning-cost function associated with the solution scorer SSRj.
  • the solution scorer SSRA of the planning module RA applies the first planning-cost function to the route proposals received from the solution proposers SPRA, SPRB, SPRC, SPRD and can determine that a first planning-cost function value corresponding to the route proposed by the solution proposer SPRB is smaller than first planning-cost function values corresponding to each of the remaining routes proposed by the solution proposers SPRA, SPRC, SPRD. For this reason, the solution scorer SSRA of the planning module RA will provide, through the end-stack connection CRA corresponding to the pipeline PLA, to the output mediator A. the route proposed by the solution proposer SPRB of the planning module RB.
  • the solution scorer SSRB of the planning module RB applies the second planning-cost function to the route proposals received from the solution proposers SPRA, SPRB, SPRC, SPRD and can determine that a second planning-cost function value corresponding to the route proposed by the solution proposer SPRB is smaller than second planning-cost function values corresponding to each of the remaining routes proposed by the solution proposers SPRA, SPRC, SPRD.
  • the solution scorer SSRB of the planning module RB will provide, through the end-stack connection CRA corresponding to the pipeline PLB, to the output mediator A. the route proposed by the solution proposer SPRB of the planning module RB. And so on, and so forth.
  • N 2 different route proposals
  • FIG. 25 shows a system 2500 which achieves the goal of generating, and synergistically evaluating, N different route proposals, by using a pair of redundant pipelines PLi, PL2 and an output mediator A. such that N route proposals are provided by the first pipeline PLi, and N 2 route proposals are provided by the second pipeline PL2, where N +
  • each redundant pipeline PL1 2 includes a first stage implemented as a respective perception module Pi, 2, and a second stage implemented as a respective planning module RI,2.
  • each perception module Pi, 2 includes a respective solution proposer SPPi,2 and a respective solution scorer SSPi,2.
  • each planning module R.1,2 includes a respective number Ni, 2 of solution proposers SPR(i,2)i, and a respective solution scorer SSRI,2, where i e ⁇ A, B, ... ⁇ .
  • the solution scorer SSPi,2 of the perception module Pi, 2 communicates with all NI,2 solution proposers SPR(i,2)i of the planning module RI,2 through an intra-stack connection CPR of the pipeline PLI,2. Also note that the solution scorer SSRI,2 of the planning module RI,2 communicates with the output mediator A through a respective end-stack connection CRA. Moreover, the solution proposer SPPi,2 of each perception module Pi, 2 communicates through an intra-inter-stack connection CP with the solution scorer SSPi,2 of the perception module Pi, 2 and with the solution scorer SSP2.1 of the other perception module P2,i.
  • each solution proposer SPR(i,2)i of each planning module RI,2 communicates through another intra-inter-stack connection CR with the solution scorer SSRI,2 of the planning module RI,2 and to the solution scorer SSR2,I of the other planning module R2,I.
  • Synergistic redundancy can be implemented at the planning stage of the system 2500 in the following manner.
  • Each of the Ni solution proposers SPRii of the planning module Ri generates a respective route proposal based on a first world view received, through the intra-stack connection CPR of the pipeline PLi, from the solution scorer SSPi of the perception module Pi, and each of the N2 solution proposers SPR2 1 of the planning module R2 generates a respective route proposal based on a second world view received, through the intra-stack connection CPR of the pipeline PL2, from the solution scorer SSP2 of the perception module P2.
  • the solution scorer SSRi of the planning module Ri evaluates the route proposals received from the first pipeline PLi’s solution proposers SPRIA, SPRIB and from the second pipeline PL2’s solution proposers SPR2A, SPR2B using a first planning-cost function
  • the solution scorer SSR2 of the planning module R2 evaluates the route proposals received from the second pipeline PL2’s solution proposers SPR2A, SPR2B and from the first pipeline PLi’s solution proposers SPRIA, SPRIB using a second planning-cost function.
  • the solution scorer SSRj of each planning module Rj selects as the winning route the one from among the received route proposals which corresponds to the smallest value of the planning-cost function associated with the solution scorer SSRj.
  • the solution scorer SSRi of the planning module Ri applies the first planning-cost function to the route proposals received from the solution proposers SPRIA, SPRIB, SPR2A, SPR2B and can determine that a first planning-cost function value corresponding to the route proposed by the solution proposer SPRIB is smaller than first planning-cost function values corresponding to each of the remaining routes proposed by the solution proposers SPRIA, SPR2A, SPR2B. For this reason, the solution scorer SSRi of the planning module Ri will provide, through the end- stack connection CRA corresponding to the pipeline PLi, to the output mediator A. the route proposed by the solution proposer SPRIB of the planning module Ri.
  • the solution scorer SSR2 of the planning module R2 applies the second planning-cost function to the route proposals received from the solution proposers SPRIA, SPRIB, SPR2A, SPR2B and can determine that a second planning- cost function value corresponding to the route proposed by the solution proposer SPRIB is smaller than second planning-cost function values corresponding to each of the remaining routes proposed by the solution proposers SPRIA, SPR2A, SPR2B.
  • the solution scorer SSR2 of the planning module R2 will provide, through the end-stack connection CRA corresponding to the pipeline PL2, to the output mediator A. the route proposed by the solution proposer SPRIB of the planning module Ri. Note that this situation corresponds to the case where a“remote solution” wins over multiple“local solutions” and other remote solutions.
  • the solution scorer SSRi, 2 can use its local cost function to compare, and select a preferred one from among, the solutions proposed locally by the Nu local solution proposers SPR(i,2)i. Subsequently, or concurrently, the solution scorer SSR.1,2 can use its local cost function to compare, and select a preferred one from among, the solutions proposed remotely by the N 2,I remote solution proposers SPR(2,i)i. Note that to perform the latter comparisons, the solution scorer SSR.1,2 first translates and/or normalizes the received remote proposed solutions, so it can apply its local cost function to them.
  • the solution scorer SSRI,2 selects between the preferred locally proposed solution and the preferred remotely proposed solution as the one which has the smaller of the cost values evaluated based on the local cost function.
  • the solution scorer SSRI,2 compares among themselves scores of N2,I proposed remote solutions that have gone through a translation / normalization operation, and only the best one of them is then compared to the best one of the NI,2 proposed native solutions that did not need to go through the translation / normalization operation.
  • the number of direct comparisons between translated / normalized proposed remote solutions and proposed local solutions can be reduced to one.
  • the solution scorer SSRI,2 compares the two or more solutions proposed locally by the NI,2 local solution proposers SPR(i,2)i, and the two or more solutions proposed remotely by the N2,I remote solution proposers SPR(2,i)i in the order in which they are received without first grouping them by provenance.
  • the solution scorer SSRI,2 first translates / normalizes each of the remotely proposed solutions before it can apply the local cost functions to it.
  • the solution scorer SSR selects - between (i) the received proposed solution and (ii) the currently preferred proposed solution, the latter having resulted from the previous comparison between proposed solutions - a new preferred proposed solution as the one which has the smaller of the cost values evaluated based on the local cost function.
  • the solution scorer SSRI,2 can proceed immediately with the comparison of the most recently received proposed solution without having to wait for another solution of the same provenance, as described in the forgoing implementations.
  • the solution scorer SSRI,2 can avoid a non-optimal solution without substantially reducing the speed of solution making for the overall system 2500.
  • the solution scorer SSRI,2 selects the preferred one as the proposed solution having the smaller of the costs evaluated based on the local cost function if the difference exceeds a threshold, e.g., 10%, 5%, 1%, 0.5% or 0.1% difference. However, if the difference of the costs of the two proposed solutions does not exceed the threshold difference, then the solution scorer SSR.1,2 is configured to compare and select between the proposed solutions based on an additional cost assessment that favors continuity with one or more prior solutions selected for operation of the AV.
  • a threshold e.g. 10%, 5%, 1%, 0.5% or 0.1% difference.
  • the solution scorer SSR.1,2 can keep a track record of when one proposed solution was preferred over another and share that information around the fleet of AVs to track when the other solution may have been better after all.
  • the first of the pipelines having Ni solution scorers at a particular stage can evaluate each of the native solution and the remote solution in Ni ways
  • the second of the pipelines having N2 solution scorers at the particular stage can evaluate each of the native solution and the remote solution in N2 ways, as described below.
  • FIG. 26 shows a system 2600 which generates two different route proposal and synergistically evaluates them in N > 2 ways, by using a pair of redundant pipelines PLi, PL2 and an output mediator A, such that a first route proposal is generated by the first pipeline PLi and a second route proposal is generated by the second pipeline PL2, where the first and second route proposals are evaluated in Ni ways by the first pipeline PLi, and N2 ways by the second pipeline PL2.
  • each of the redundant pipelines PLI,2 includes a first stage implemented as a respective perception module Pi, 2, and a second stage implemented as a respective planning module RI,2.
  • FIG. 1 each of the redundant pipelines PLI,2 includes a first stage implemented as a respective perception module Pi, 2, and a second stage implemented as a respective planning module RI,2.
  • each perception module Pi,2 includes a respective solution proposer SPPi,2 and a respective solution scorer SSPi,2.
  • each planning module RI,2 includes a respective solution proposer SPRI,2, a respective number Ni, 2 of solution scorers SSR(i,2)i, and a respective planning arbiter ARi,2,where i E ⁇ A, B, ... ⁇ .
  • the solution scorer SSPi,2 of the perception module Pi, 2 communicates with the solution proposer SPRi, 2 of the planning module R.1,2 through an intra-stack connection CPR of the pipeline PLI,2.
  • all NI,2 solution scorers SSR(i,2)i communicate with the planning arbiter ARI,2 through an intramodule connection CRR.
  • the planning arbiter ARI,2 of the planning module RI,2 communicates with the output mediator A through a respective end-stack connection CRA.
  • the solution proposer SPPi,2 of each perception module Pi, 2 communicates through an intra-inter-stack connection CP with the solution scorer SSPi ,2 of the perception module Pi, 2 and with the solution scorer SSP2,i of the other perception module P2,i.
  • the solution proposer SPRI,2 of each planning module RI,2 communicates through another intra- inter-stack connection CR with each solution scorer SSR(i,2)i of the planning module RI,2 and to each solution scorer SSR(2,i)i of the other planning module R2,I.
  • Synergistic redundancy can be implemented at the planning stage of the system 2600 in the following manner.
  • the solution proposer SPRi of the planning module Ri generates a first route proposal based on a first world view received, through the intra-stack connection CPR of the pipeline PLi, from the solution scorer SSPi of the perception module Pi, and the solution proposer SPR2 of the planning module R2 generates a second route proposal based on a second world view received, through the intrastack connection CPR of the pipeline PL2, from the solution scorer SSP2 of the perception module P2.
  • Each of the NI,2 solution scorers SSR(i,2)i of the planning module RI,2 receives, through the intra-inter-stack connection CR, the first route proposal from the solution proposer SPRi of the planning module Ri and the second route proposal from the solution proposer SPR2 of the planning module R2, and evaluates both first and second route proposals by using a planning-cost function associated with the solution scorer SSR (1,2) 1 .
  • the solution scorer SSRIA evaluates the first route proposal and the second route proposal using a first planning-cost function
  • the solution scorer SSRIB evaluates the first route proposal and the second route proposal using a second planning-cost function.
  • the first planning-cost function and the second planning-cost function may evaluate each of the first and second route proposals along different axes, e.g., safety, comfort, etc.
  • the solution scorer SSR.2A evaluates the first route proposal and the second route proposal using a third planning-cost function
  • the solution scorer SSR.2B evaluates the first route proposal and the second route proposal using a fourth planning-cost function.
  • Each solution scorer SSR(i,2)i selects as the winning route the one from among the first and second route proposals which corresponds to the smallest value of the planning-cost function associated with the solution scorer SSR(i,2)i.
  • the third planning-cost function and the fourth planning-cost function may evaluate each of the first and second route proposals along the same axis, but with different models, priors, etc.
  • the solution scorer SSRIA applies the first planning-cost function to the first and second route proposals and can determine that a first planning-cost function value corresponding to the first route proposed by the solution proposer SPRi is smaller than first planning-cost function value corresponding to the second route proposed by the solution proposer SPR2. For this reason, the solution scorer SSRIA of the planning module Ri will provide the first route, through the intra-module connection CRR of the planning module Ri, to the planning arbiter ARi.
  • the solution scorer SSRIB applies the second planning-cost function to the first and second route proposals and can determine that a second planning-cost function value corresponding to the first route proposed by the solution proposer SPRi is smaller than second planning-cost function value corresponding to the second route proposed by the solution proposer SPR2.
  • the solution scorer SSRIB of the planning module Ri will provide the first route, through the intra- module connection CRR of the planning module Ri, to the planning arbiter ARi.
  • the planning arbiter ARi can implement one or more selection processes, e.g., like the ones described in detail in the next section, to select one of the routes provided by the redundant solution scorers SSRIA, SSRIB of the planning module Ri.
  • the solution scorers SSRIA, SSRIB provided the same route, so the planning arbiter ARi simply relays, through the end-stack connection CRA corresponding to the pipeline PLi, the first route to the output mediator A. While these operations are performed at the pipeline PLi, the solution scorer SSR2A applies the third planning-cost function to the first and second route proposals and can determine that a third planning-cost function value corresponding to the second route proposed by the solution proposer SPR2 is smaller than third planning-cost function value corresponding to the first route proposed by the solution proposer SPRi.
  • the solution scorer SSR2A of the planning module R2 will provide the second route, through the intra-module connection CRR of the planning module R2, to the planning arbiter AR2.
  • the solution scorer SSR.2B applies the fourth planning-cost function to the first and second route proposals and can determine that a fourth planning-cost function value corresponding to the first route proposed by the solution proposer SPRi is smaller than fourth planning-cost function value corresponding to the second route proposed by the solution proposer SPR2.
  • the solution scorer SSR2B of the planning module R2 will provide the first route, through the intra-module connection CRR of the planning module R2, to the planning arbiter AR2.
  • the planning arbiter AR2 can implement one or more selection processes, e.g., like the ones described in detail in the next section, to select one of the routes provided by the redundant solution scorers SSR2A, SSR2B of the planning module R2.
  • the solution scorers SSR2A, SSR2B provided different routes, so the planning arbiter AR2 must first apply its own selection process, and then it can relay, through the end-stack connection CRA corresponding to the pipeline PL2, the selected one between the first route and the second route to the output mediator A.
  • the output mediator A can implement one or more selection processes, described in detail in the next section, to select one of the routes provided by the pair of redundant pipelines PLi, PL2. In this manner, the output mediator A can provide to a controller module a single route between the first and second routes generated within the redundant pipelines PLi, PL2, and evaluated N > 2 ways within the redundant pipelines PLi, PL2.
  • each of the AV operations subsystems described above include components that are either pure scorers, e.g., denoted above as X14, or pure proposers, e.g., denoted above as X12, where X e ⁇ F, G, H, I,], K ⁇ .
  • X14 pure scorers
  • X12 pure proposers
  • X12 pure proposers
  • K K
  • AV operations subsystem also referred to as modules
  • OEM components i.e., AV operations subsystem (also referred to as modules) designed and/or fabricated by third parties.
  • AV operations subsystem also referred to as modules
  • an AV system integrator need not fully understand the“under-the-hood” configuration of a third-party module as long as the third-party module is placed in a test pipeline integrated through the disclosed synergistic redundancy with one or more other pipelines which include trusted modules at the corresponding stage.
  • the third-party module can be deemed useful and/or reliable if it contributes proposals which are being selected during cross-evaluations with a selection frequency that meets a target selection frequency. If, however, the selection frequency of the proposals contributed by the third-party module is not met during the disclosed cross evaluations, then the third-party module can be removed from the test pipeline.
  • proposers can be designed and fabricated by any third party as long as the third-party proposers’ union covers the use case.
  • examples of such proposers which can be integrated in synergistically redundant AV operations systems like the ones described above, include third-party proposers for planning stereotypical plans, e.g., stop now, follow lane, follow vehicle ahead, etc.
  • Other examples include third-party proposers for planning any ad-hoc heuristics to solve comer cases, for instance.
  • a third-party proposer can be removed from an AV operations subsystem when it is detected that its proposals are not being selected often enough by one or more scorers - from the same AV operations subsystem or AV operations subsystems disposed at the same stage of other redundant pipelines - with which the third-party proposer communicates.
  • the target selection frequency that must be met by the third-party proposer can be established based on performance of one or more currently used proposers. In this manner, the cross-evaluations implemented in the disclosed systems allow for the computation resources used by the“bad” proposer to be recovered by the AV system upon removal of the bad proposer.
  • a system 1300 (or 1600, 2000, 2400, 2500, 2600) useable to operate an autonomous vehicle (AV) includes two or more different AV operations subsystems l3l0a, 13 lOb (or l620a, l620b, Ri, R2, ...
  • the output mediator 1340 (or 1640, or A ) is configured to selectively promote a single one of the two or more different AV operations subsystems 1310a, 1310b (or 1620a, 1620b, or Ri, R2, ... ) to a prioritized status based on current input data compared with historical performance data for the two or more different AV operations subsystems l3l0a, 1310b (or l620a, l620b, or Ri, R2, ... ).
  • one redundant subsystem may be designed for handling highway driving and the other for urban driving; either of the redundant subsystems may be prioritized based on the driving environment.
  • an AV operations module l3l0a,b (or l620a,b or R1 2) has its output favored over outputs of remaining AV operations subsystems l3l0b,a (or l620b,a or R2,I.)
  • the output mediator 1340 (or 1640) operates as a de facto AV operations arbitrator that selects one AV operation output received from an AV operations subsystem l3l0a,b (or l620a,b, or A) over all other outputs received from the remaining AV operations subsystems l3l0b,a (or 1620b, a, R2,I).
  • FIG. 27 is a flow chart of an example of a process 2700 used by an output mediator coupled with N different AV operations subsystems for managing AV operation outputs, denoted OPi, OP2, ... , OPN, from the N different AV operations subsystems, where N 3 2.
  • the output mediator designates prioritized status to one, and non- prioritized status to remaining ones, of N different AV operations subsystems. This operation is performed at the beginning of the process 100, e.g., when the output mediator is powered ON, reset, or patched with upgraded software, etc., to assign initial statuses to each of the N different AV operations subsystems with which the output mediator communicates.
  • the output mediator 1340 (or 1640, 21) has access to an array 28-05 of AV operations subsystem identifiers (IDs) of the N different AV operations subsystems l3l0a, 1310b, ... , FIG. 11 ON (or l620a, l620b, ...
  • the output mediator 1340 uses a priority pointer to point to the ID of the AV operations subsystem having the prioritized status 28-15, and thus to keep track of the fact that, in this example, it is 1310b who has prioritized status, and not another one from the remaining AV operations subsystems l3l0a, ... , 1310N.
  • the output mediator receives N outputs from the N different AV operations subsystems, respectively, i.e., it receives the I st AV operations subsystem’s output OPi, ... , and the N th AV operations subsystem’s output OPN.
  • the output mediator 1440 receives two versions of the world view l4l6a, l4l6b.
  • the output mediator 1540 receives two versions of the route l4l4a, l4l4b (or l7l4a, l7l4b.)
  • the output mediator A also receives two versions of the route.
  • the output mediator A receives four versions of the route.
  • the output mediator 1840 receives two versions of the control signal for controlling the steering actuator 420a, the throttle actuator 420b, and/or the brake actuator 420c.
  • the output mediator determines whether the I st AV operations subsystem, ... , and the N th AV operations subsystem, each provided the same output OP. Equivalently, the output mediator determines, at 2725, whether the I st AV operations subsystem’s output OPi, ... , and the N th AV operations subsystem’s output OPN are equal to each other.
  • the output mediator 1340 uses an output comparator 2825 to compare the received AV operations subsystem outputs 2822.
  • the output comparator 2825 will compare the received AV operations subsystem outputs 2822 by comparing their respective provenance indicators.
  • the solution proposers l3l2a,b, l622a,b, SPRA,B,C,D mark their respective solution proposals with a solution identifier indicating the ID of the AV operations subsystem to which it belongs. For instance, a solution proposed by the solution proposal l3l2a will be marked with a provenance indicator specifying that the solution originated at the AV operations subsystem l3l0a, while the alternative solution proposed by the solution proposal 13 l2b will be marked with a provenance indicator specifying that the solution originated at the redundant AV operations subsystem 13 lOb.
  • each of the I st AV operations subsystem’s output OPi, ... , and the N th AV operations subsystem’s output OPN received by the output mediator will carry a respective provenance indicator identifying the AV operations subsystem where it originated.
  • the output comparator 2825 of the output mediator will simply check the respective provenance indicators of the received AV operations subsystem outputs 2822 to determine whether they are the same, or at least one of them is different from the other.
  • the output mediator A determines that each of the four routes received from the redundant planning modules RA, RB, RC, RD carries the same provenance indicator, e.g., identifying the planning module RB, then the output mediator A treats the four routes as one and the same route, here the route that originated at the planning module RB and was adopted by all four planning modules RA, RB, RC, RD.
  • the output mediator A determines that at least one of the four routes received from the redundant planning modules RA, RB, RC, RD carries a provenance indicator different from the other provenance indicators, then the output mediator A treats that route as being different from the other three routes.
  • the output mediator 1440 receives from the two redundant perception modules l4l0a, l4l0b, the two world views l4l6a,
  • the output mediator 1440 will treat the two world views 1416a, 1416b to be the same if a distance between the world views is smaller than, or equal to, a threshold world view distance, or different if the distance between the world views is larger than the threshold world-view distance.
  • the output mediator 1540 receives from the two redundant planning modules l5l0a, l5l0b, the two routes l5l4a, 15 l4b.
  • the output mediator 1540 will treat the two routes l5l4a, 15 l4b to be the same if a distance between the routes is smaller than, or equal to, a threshold route distance, or different if the distance between the routes is larger than the threshold route distance.
  • the output mediator determines that the I st AV operations subsystem’s output OPi, ... , and the N th AV operations subsystem’s output OPN are equal to each other, then at 2770, the output mediator controls issuance of the output of the AV operations subsystem which has the prioritized status.
  • the output mediator controls the issuance of the output of the AV operations subsystem, which has the prioritized status, are described in detail below.
  • the output mediator determines that at least one of the I st AV operations subsystem’s output OPi, ... , and the N th AV operations subsystem’s output OPN is different from the remaining ones, then at 2730, the output mediator accesses current input data.
  • FIG. 28 shows that the output mediator 1340 (or 1640, A) has access to current input data L231.
  • the current input data 28-31 includes map data 28-32, e.g., stored by the database module 410 or a remote geo-position system; position data 28-38 provided by the localization module 408, for instance; traffic data 28-36 provided by the perception module 402, for instance; weather data 28-34 provided by local sensors 121 or remote weather monitoring / forecast systems; time of day data 28-35 provided by a local or remote clock; and speed data 28-33 provided by a speedometer of the AV.
  • map data 28-32 e.g., stored by the database module 410 or a remote geo-position system
  • position data 28-38 provided by the localization module 408, for instance
  • traffic data 28-36 provided by the perception module 402, for instance
  • weather data 28-34 provided by local sensors 121 or remote weather monitoring / forecast systems
  • time of day data 28-35 provided by a local or remote clock
  • speed data 28-33 provided by a speedometer of the AV.
  • the output mediator determines a current operational context based on the current input data. For instance, the output mediator can use a mapping of input data to operational contexts to (i) identify a portion of input data of the mapping that encompasses the current input data, and (ii) determine the current operational context as an operational context mapped to the identified input data portion.
  • the mapping of input data to operational contexts can be implemented as a look-up-table (LUT), for instance.
  • the LUT used by the output mediator 1340 (or 1640, A) for this purpose is implemented as an input data / context look-up-table (LUT) 2842.
  • the input data / context LUT 2842 includes M predefined operational contexts, and two or more groupings of input data types and ranges, the groupings being mapped to the M predefined operational contexts, where M 3 2.
  • a grouping which includes position data 2838 and map data 2832 corresponding to freeways, and speed data 2833 in the range of 45-75mph is mapped to an operational context called“freeway driving.”
  • a grouping which includes position data 2838 and map data 2832 corresponding to surface streets, and speed data 2833 in the range of 5-45mph is mapped to an operational context called“surface-street driving.”
  • a grouping which includes traffic data 2838 corresponding to low- to medium-traffic, and time of day data 2835 in the range of l9:00h to 06:00h is mapped to an operational context called“night-time driving.”
  • a grouping which includes traffic data 2838 corresponding to medium- to high-traffic, and time of day data 2835 in the range of 06:00h to l9:00h is mapped to an operational context called“day-time driving.”
  • a grouping which includes weather data 2834 corresponding to lack of precipitation, and speed data 2833 in the range of 30-75mph is mapped to an operational context called“fair-weather driving.”
  • Many other predefined operational context can be defined in the input data / context LUT 2842.
  • the output mediator 1340 (or 1640, A) identifies which of the groupings of input data types and ranges included in the input data / context LUT 2842 encompasses the current input data 2831. For instance, if the current input data 2831 includes position data 2838 and map data 2832 indicating that the AV is currently located on the 405 SANTA MONICA FREEWAY and the AV speed is 55mph, then the output mediator 1340 (or 1640) identifies the input data / context LUT 2842’s grouping of input data types and ranges that
  • the output mediator 1340 determines a current operational context 2845 of the AV, as the operational context mapped to the identified grouping.
  • the output mediator 1340 determines that the current operational context 2845 of the AV is“freeway driving.” Once the output mediator 1340 (or 1640, A) determines the current operational context 2845 in this manner, it can use a context pointer which points to an identifier of the current operational context 2845, to keep track of the fact that, in this example, it is“freeway driving” that is the current operational context, and not another one from the remaining operational contexts referenced in the input data / context LUT 2842.
  • the output mediator identifies the AV operations subsystem
  • the output mediator can use a mapping of operational contexts to IDs of AV operations subsystems to (i) select an operational context of the mapping that matches the current operational context, and (ii) identify the AV operations subsystem corresponding to the current operational context as an AV operations subsystem having an ID mapped to the selected operational context.
  • the mapping of operational contexts to IDs of AV operations subsystems represents historical performance data of the N different AV operations subsystems.
  • the output mediator uses machine learning to determine the mapping of specific operational contexts to IDs of AV operations subsystems. For instance, a machine learning algorithm operates on AV operations subsystems’ historical data to determine one or more specific operational contexts for the AV in which each one of its N different AV operations subsystems performs differently, better or worse, than remaining ones of the N different AV operations subsystems.
  • the historical data include data that is collected on the current trip and the determination of the mapping of operational contexts to IDs of AV operations subsystems is run in real time.
  • the historical data include data that was collected on previous trips and the determination of the mapping of operational contexts to IDs of AV operations subsystems was run, e.g., overnight, before the current trip.
  • the machine learning algorithm maps an AV operations subsystem to a specific operational context only after substantial improvement is determined for the AV operations subsystem. For instance, the AV operations subsystem will be mapped to the specific operational context only once the historical performance data shows a substantially better performance in the specific operational context. As an example, if a particular AV operations subsystem has, 52 out of 100 times, better performance than the AV operations subsystem preferred for the specific operational context, then the particular AV operations subsystem will not be promoted to preferred status for this specific operational context. For example, the performance improvement must be 20% higher for the change in preferred status to be implemented. As such, if the particular AV operations subsystem has,
  • the performance improvement is measured in terms of the solutions provided by the particular AV operations subsystem having costs that are lower by a predetermined delta than solutions provided by a previously preferred AV operations subsystem, but also in terms of distances between the solutions provided by the particular AV operations subsystem and solutions provided by the previously preferred AV are less than a predetermined difference.
  • the results of the determination of the mapping of operational contexts to IDs of AV operations subsystems are shared across a fleet of AVs.
  • the machine learning algorithm operates on historical performance data relating to use of the N different AV operations subsystems in different AVs in a fleet of AVs.
  • the results obtained by the machine learning algorithm in this manner can be shared with other AVs of the fleet either directly, e.g., through ad-hoc communications with AVs that are in the proximity of each other, or through a central control system for coordinating the operation of multiple AVs, e.g., like the one described above in connection with FIG. 2.
  • individual AV performance can be improved by using analyses of data spanning a fleet of AVs using the same subsystems.
  • the mapping of operational contexts to IDs of AV operations subsystems can be implemented as another LUT, for instance.
  • the other LUT used by the output mediator 1340 (or 1640, A) for this purpose is implemented as a context / subsystem LUT 2852.
  • the context / subsystem LUT 2852 includes N AV operations subsystem IDs, and M predefined operational contexts, the N IDs being mapped to the M operational contexts, where M, N 3 2. Note that in this example context / subsystem LUT 2852 shown in FIG. 28, an AV operations subsystem ID is mapped to one or more of the M operational contexts, while an operational context has a single AV operations subsystem ID mapped to it.
  • the ID of AV operations subsystem l3l0a is mapped to the I st operational context, e.g.,“freeway driving,” while the ID of AV operations subsystem 1310N is mapped to the j th operational context, e.g.,“night-time driving”.
  • the ID of AV operations subsystem 1310b is mapped to the 2 nd operational context, e.g., “surface-street driving,” and to the operational M th context, e.g.,“inclement-weather driving.”
  • the ID of the planning module RA can be mapped to an operational context“freeway, fair-weather driving,” the ID of the planning module RB can be mapped to another operational context“freeway, inclement-weather driving,” the ID of the planning module Rc can be mapped to yet another operational context“surface-street, fair- weather driving,” and the ID of the planning module RD can be mapped to yet another operational context“surface-street, inclement- weather driving.”
  • the ID of the planning module RD can be mapped, at the same time, to the operational context“heavy- traffic driving,” for instance.
  • the output mediator 1340 selects the operational context included in the context / subsystem LUT 2852 that matches the current operational context 2845. For instance, if the current operational context 2845 is“surface-street driving,” then the output mediator 1340 (or 1640, A) selects the 2 nd operational context, which is labeled“surface- street driving”, from among the operational contexts included in the context / subsystem LUT 2852.
  • the output mediator 1340 By selecting the operational context included in the context / subsystem LUT 2852 that matches the current operational context 2845, the output mediator 1340 (or 1640, A) identifies an ID of an AV operations subsystem 2855, as the ID of the AV operations subsystem mapped to the selected operational context, and, thus, identifies the mapped AV operations subsystem 2855 as corresponding to the current operational context 2845.
  • the output mediator 1340 by selecting the 2 nd operational context included in the context / subsystem LUT 2852, the output mediator 1340 (or 1640, A) identifies the ID of the AV operations subsystem 1310b from among the IDs of the AV operations subsystems l3l0a, !3l0b, ...
  • the output mediator 1340 (or 1640, A) identifies the AV operations subsystem 2855 in this manner, it can use a subsystem pointer which points to an identifier of the AV operations subsystem 2855, to keep track of the fact that, in this example, it is 13 lOb that is the identified AV operations subsystem, and not another one from the remaining AV operations subsystems l3l0a, ... , 1310N referenced in the context / subsystem LUT 2852.
  • the output mediator verifies whether the identified AV operations subsystem is the AV operations subsystem having prioritized status.
  • the output mediator 1340 (or 1640, A ) can determine that the ID of the AV operations subsystem 2855 from the context / subsystem LUT 2852 corresponding to the current operational context 2845 is the same as the ID of the AV operations subsystem having the prioritized status 2815, and, thus, verifies that the identified AV operations subsystem 2855 has prioritized status.
  • the output mediator 1340 (or 1640) can determine that the ID of the AV operations subsystem 2855 from the context / subsystem LUT 2852 corresponding to the current operational context 2845 is different from the ID of the AV operations subsystem having the prioritized status 2815, and, thus, verifies that the identified AV operations subsystem has non-prioritized status.
  • the output mediator determines that the identified AV operations subsystem is the AV operations subsystem having prioritized status, then at 2770 the output mediator controls issuance of the output of the AV operations subsystem which has the prioritized status.
  • the output mediator controls the issuance of the output of the AV operations subsystem, which has the prioritized status, is described in detail below.
  • the output mediator determines that the identified AV operations subsystem is different from the AV operations subsystem having prioritized status, then, at 2760, the output mediator demotes the AV operations subsystem having prioritized status to non-prioritized status, and promotes the identified AV operations subsystem to prioritized status.
  • the output mediator 1340 (or 1640, A) redirects the priority pointer from pointing to the ID of the AV operations subsystem 2815, which had prioritized status prior to being demoted at 2755N, to pointing to the ID of the AV operations subsystem 2855, which has prioritized status since being promoted, at 2755N.
  • the output mediator e.g., 1340 or 1640, A, promotes an AV operations subsystem based on a type of a street on which the AV currently is.
  • the output mediator is configured to selectively promote the identified AV operations subsystem 2855 from among the N different AV operations subsystems to the prioritized status based on the following two factors.
  • the first factor is the current input data 2831 indicates (based on the input data / context LUT 2842) a current operational context 2845 is either city streets or highway driving conditions.
  • the second factor is the historical performance data, represented in the form of the context / subsystem LUT 2852, indicates that the identified AV operations subsystem 2855 performs better in the current operational context 2845 than remaining ones of the N different AV operations subsystems.
  • the output mediator e.g., 1340 or 1640, A. promotes an AV operations subsystem based on traffic currently experienced by the AV.
  • the output mediator is configured to selectively promote the identified AV operations subsystem 2855 from among the N different AV operations subsystems to the prioritized status based on the following two factors.
  • the first factor is the current input data 2831 indicates (based on the input data / context LUT 2842) a current operational context 2845 involves specific traffic conditions.
  • the second factor is the historical performance data, represented in the form of the context / subsystem LUT 2852, indicates that the identified AV operations subsystem 2855 performs better in the current operational context 2845 than remaining ones of the N different AV operations subsystems.
  • the output mediator e.g., 1340 or 1640, A, promotes an AV operations subsystem based on weather currently experienced by the AV.
  • the output mediator is configured to selectively promote the identified AV operations subsystem 2855 from among the N different AV operations subsystems to the prioritized status based on the following two factors.
  • the first factor is the current input data 2831 indicates (based on the input data / context LUT 2842) a current operational context 2845 involves specific weather conditions.
  • the second factor is the historical performance data, represented in the form of the context / subsystem LUT 2852, indicates that the identified AV operations subsystem 2855 performs better in the current operational context 2845 than remaining ones of the N different AV operations subsystems.
  • the output mediator e.g., 1340 or 1640, A, promotes an AV operations subsystem based on the time of day when the AV is currently operated.
  • the output mediator is configured to selectively promote the identified AV operations subsystem 2855 from among the N different AV operations subsystems to the prioritized status based on the following two factors.
  • the first factor is the current input data 2831 indicates (based on the input data / context LUT 2842) a current operational context 2845 is a particular time of day.
  • the second factor is the historical performance data, represented in the form of the context / subsystem LUT 2852, indicates that the identified AV operations subsystem 2855 performs better in the current operational context 2845 than remaining ones of the N different AV operations subsystems.
  • the output mediator e.g., 1340 or 1640, A. promotes an AV operations subsystem based on the current speed of the AV.
  • the output mediator is configured to selectively promote the identified AV operations subsystem 2855 from among the N different AV operations subsystems to the prioritized status based on the following two factors.
  • the first factor is the current input data 2831 indicates (based on the input data / context LUT 2842) a current operational context 2845 involves specific speed ranges.
  • the second factor is the historical performance data, represented in the form of the context / subsystem LUT 2852, indicates that the identified AV operations subsystem 2855 performs better in the current operational context 2845 than remaining ones of the N different AV operations subsystems.
  • the output mediator controls the issuance of the output of the AV operations subsystem which has the prioritized status.
  • the process 2700 reaches operation 2770 after performing either one of operations 2725Y, 2755Y or 2760.
  • 2770 is performed by the output mediator upon confirming that the AV operations subsystem’s output to be provided down-stream from the output mediator was received, at 2720, from the AV operations subsystem which has prioritized status, now at 2770, i.e., in the current operational context.
  • the output mediator instructs the prioritized AV operations subsystem (e.g., 2815) to provide, down-stream therefrom, its AV operation output directly to the next AV operations subsystem or to an actuator of the AV.
  • the output mediator does not relay the prioritized AV operations subsystem’s output to its destination, instead it is the prioritized AV operations subsystem itself that does so.
  • the output mediator 1740 instructs the planning module l720b to provide, down-stream to the control module 406, the planning module l720b’ route !7l4b.
  • the output mediator e.g., 1340 or 1640, A
  • the prioritized AV subsystem e.g., 28l5’s output, which was received by the output mediator, at 2720.
  • the output mediator 1740 relays, down-stream to the control module 406, the planning module l720b’ route l7l4b.
  • the sequence of operations 2720 through 2770 is performed by the output mediator (e.g., 1340, or 1640, A) in each clock cycle. As such, these operations are performed iteratively during future clock cycles.
  • the output mediator e.g., 1340, or 1640, A
  • these operations are performed iteratively during future clock cycles.
  • FIG. 30 shows a redundant control system 3000 for providing redundancy in control systems for an AV.
  • AVs such as the AV 100 of FIG. 1, may include the redundant control system 3000.
  • the redundant control system 3000 includes computer processors 3010, a first control system 3020, and a second control system 3030.
  • the computer processors 3010 include only one processor.
  • the computer processors 3010 include more than one processors.
  • the computer processors 3010 are configured to algorithmically generate control actions based on both real-time sensor data and prior information.
  • the computer processors 3010 are substantially similar to the computer processors 146 referenced in FIG. 1.
  • the computer processors 3010 may include a diagnostics module 3011 and an arbiter module 3012.
  • the first control system 3020 and the second control system 3030 include control modules 3023, 3033.
  • control modules 3023, 3033 are substantially similar to the control module 406 described previously with reference to FIG. 4.
  • control modules 3023, 3033 include controllers substantially similar to the controller 1102 described previously with reference to FIG. 11.
  • one control system uses the data output by the other control system, e.g., as previously described in reference to FIGS. 13-29.
  • the first control system 3020 and the second control system 3030 are configured to receive and act on operational commands from the computer processors 3010.
  • the first control system 3020 and the second control system 3030 may include various other types of controllers, such as door lock controllers, window controllers, turn-indicator controllers, windshield wiper controllers, and brake controllers.
  • the first control system 3020 and the second control systems 3030 also include control devices 3021, 3031.
  • the control devices 3021, 3031 facilitate the control systems’ 3020, 3030 ability to affect the control operations 3040.
  • Examples of control devices 3021, 3031 include, but are not limited to, a steering mechanism/column, wheels, axels, brake pedals, brakes, fuel systems, gear shifter, gears, throttle mechanisms (e.g., gas pedals), windshield wipers, side-door locks, window controls, and turn-indicators.
  • the first control system 3020 and the second control system 3030 include a steering angle controller and a throttle controller.
  • the first control system 3020 and the second control system 3030 are configured to provide output that affects at least one control operation 3040.
  • the output is data that is used for acceleration control.
  • the output is data used for steering angle control.
  • the control operations 3040 includes affecting the direction of motion of the AV 100.
  • the control operations 3040 includes changing the speed of the AV 100. Examples of control operations include, but are not limited to, causing the AV 100 to accelerate/decelerate and steering the AV 100.
  • control systems 3020, 3030 affects control operations 140 that include managing change in speeds and orientations of the AV 100.
  • speed profile relates to the change in acceleration or jerk to cause the AV 100 to transition from a first speed to at least a second speed.
  • a jagged speed profile describes rapid change in the speed of the AV 100 via acceleration or deceleration.
  • An AV 100 with a jagged speed profile transitions between speeds quickly and therefore, may cause a passenger to experience an unpleasant/uncomfortable amount of force due to the rapid
  • a smooth speed profile describes a gradual change in the speed of the AV 100 to transition the AV 100 from a first speed to a second speed.
  • a smooth speed profile ensures that the AV 100 transitions between speeds at a slower rate and therefore, reduces the force of acceleration/deceleration experienced by a passenger.
  • the control systems 3020, 3030 control various derivatives of speed over time, including acceleration, jerk, jounce, snap, crackle, or other higher-order derivatives of speed with respect to time, or combinations thereof.
  • the control systems 3020, 3030 affects the steering profile of the AV 100.
  • Steering profile relates to the change in steering angle to orient the AV 100 from a first direction to a second direction.
  • a jagged steering profile includes causing the AV 100 to transition between orientations at higher/sharper angles.
  • a jagged steering profile may cause passenger discomfort and may also lead to increased probability of the AV 100 tipping over.
  • a smooth steering profile includes causing the AV 100 to transition between orientations at lower/wider angles.
  • a smooth steering profile leads to increased passenger comfort and safety while operating the AV 100 under varied environmental conditions.
  • the first control system 3020 and the second control system 3030 include different control devices 3021, 3031 that facilitate the control systems’ 3020, 3030 ability to affect substantially similar control operations 3040.
  • the first control system 3020 may include a throttle mechanism, a brake pedal, and a gear shifter to affect throttle control operations
  • the second control system 3030 may include the fuel systems, brakes and gears to affect throttle control operations.
  • the steering mechanism is a steering wheel.
  • the steering mechanism can be any mechanism used to steer the direction of the AV 100, such as joysticks or lever steering apparatuses.
  • the first control system 3020 may include the steering mechanism of the AV 100, while the second control system 3030 may include the wheels or axels.
  • the first control system 3020 and the second control system 3030 may act together to allow for two redundant control systems that can both perform the same control operations (e.g., steering, throttle control, etc.) while controlling separate devices.
  • the first control system 3020 and the second control system 3030 affect the same control operations while including the same devices.
  • the first control system 3020 and the second control system 3030 may both include the steering mechanism, brake pedal, gear shifter, and gas pedal to affect steering and throttle operations.
  • the first control system 3020 and the second control system 3030 may simultaneously include overlapping devices as well as separate devices.
  • first control system 3020 and the second control system 3030 may include the AV’s 100 steering column to control steering operations, while the first control system 3020 may include a throttle mechanism to control throttle operations and the second control system 3030 may include the AV’s 100 wheels to control throttle operations.
  • the first control system 3020 and the second control system 3030 provide their respective output in accordance with at least one input.
  • the control systems 3020, 3030 may receive input from a planning module, such as the planning module 404 discussed previously with reference to FIG. 4, that provides information used by the control systems 3020, 3030 to choose a heading for the AV 100 and determine which road segments to traverse.
  • the input may also correspond to information received from a localization module, such as the localization module 408 discussed previously with reference to FIG.
  • the input may also correspond to feedback modules, such as the predictive feedback module 1122 described earlier with reference to FIG. 11.
  • the input may also include information received from databases, computer networks, etc.
  • the input is a desired output.
  • the desired output may include speed and heading based on the information received by, for example, the planning module 404.
  • the first control system 3020 and the second control system 3030 provide output based on the same input.
  • the first control system 3020 provides output based on a first input, while the second control system 3030 provide output based on a second input.
  • the computer processors 3010 are configured to utilize the arbiter module 3012 to select at least one of the first control system 3020 and the second control system 3030 to affect the control operation of the AV 100. Selection of either control system may be based on various criteria.
  • the arbiter module 3012 is configured to evaluate the performance of the control systems 3020, 3030 and select at least one of the first control system 3020 or the second control system 3030 based on the performance of the first control system 3020 and the second control system 3030 over a period of time.
  • evaluating control system performance may include evaluating the responsiveness of the control systems 3020, 3030 or the accuracy of the control systems’ responses.
  • evaluation of responsiveness includes determining the lag between the control system receiving input, for example to affect a change in acceleration, and the control system 3020 or 3030 acting on the throttle control mechanism to change the acceleration.
  • the evaluation of accuracy includes determining the error or difference between the required actuation of an actuator by a control system and the actual actuation applied by the control system.
  • the computer processors 3010 includes a diagnostics module 3011 configured for identifying a failure of at least one of the first control system 3020 and the second control system 3030. The failure may be partial or complete, or the control systems 3020, 3030 can satisfy at least one failure condition.
  • a partial failure may generally refer to a degradation of service while a complete failure may generally refer to a substantially complete loss of service.
  • a complete failure may be a complete loss of the ability to steer the AV 100, while a partial failure may be a degradation in the AV’s 100 responsiveness to steering controls.
  • a complete failure may be a complete loss of the ability to cause the AV 100 to accelerate, while a partial failure may be a degradation in the AV’s 100 responsiveness to throttle controls.
  • failure conditions include a control system becoming nonresponsive, a potential security threat to the control system, a steering device/throttle device becoming locked/jammed, or various other failure conditions that increases the risk of the AV 100 to deviate from its desired output.
  • the computer processors 3010 may select the second control system 3030 to carry out steering operations if the steering column becomes locked in place (e.g., control system failure condition).
  • the computer processors 3010 may select the second control system 3030 to carry out throttle operations if the gas pedal becomes unresponsive to commands sent from the computer processors 3010 (e.g., control system failure condition).
  • the controllers of the first control system 3020 and the second control system 3030 are configured to receive and utilize feedback from a first and second feedback system, respectively.
  • a feedback system can include sets of sensors, a type of sensor, or feedback algorithms.
  • the first control system 3020 and the second control system 3030 are configured to receive feedback from the same feedback system.
  • the first control system 3020 is configured to receive feedback from a first feedback system
  • the second control system 3030 is configured to receive feedback from a second feedback system.
  • the first control system 3020 may receive feedback from only a Lidar sensor on the AV 100
  • the second control system 3030 may receive feedback from only a camera on the AV 100.
  • the feedback can include measured output feedback, such as the AV’s 100 position, velocity or acceleration.
  • the feedback can also include predictive feedback from a predictive feedback module, such as the predictive feedback module 1122 described above with reference to FIG. 11.
  • the computer processors 3010 are configured to compare the feedback from the first feedback system and the second feedback system to identify a failure, if any, of at least one of the first control system 3020 and the second control system 3030.
  • the first control system 3020 and the second control system 3030 are configured to affect throttle operations of the AV 100 with a desired speed output of 25 MPH within certain bounds of error.
  • the first feedback system which corresponds to the first control system 3020, measures the average speed of the AV 100 to be 15 MPH over a time period of 5 minutes
  • the second feedback module measures the average speed of the AV 100 to be 24 MPH over a time period of 5 minutes
  • the computer processors 3010 may determine that the first control system 3010 is experiencing a failure condition.
  • the computer processors 3010 may select the other control system to affect control operations.
  • control systems 3020, 3030 may use control algorithms 3022, 3032 to affect the control operations 3040.
  • control algorithms for example, in an embodiment, the control algorithms
  • the control algorithms 3022/3032 adjust the throttle control of the AV 100.
  • the first control system 3020 uses a first control algorithm 3022 when affecting the control operations 3040.
  • the second control system 3030 uses a second control algorithm 3032 when affecting the control operations.
  • the first control system 3020 may use a first control algorithm 3022 to adjust the steering angle applied to the AV 100
  • the second control system 3030 may use a second control algorithm 3032 to adjust the throttle applied to the AV 100.
  • both control systems 3020, 3030 use the same algorithm to affect the control operations 3040.
  • the control algorithms 3022, 3032 are control feedback algorithms, which are algorithms corresponding to feedback modules, such as the measured feedback module 1114 and the predictive feedback module 1122 as previously described with reference to FIG. 11.
  • the computer processors 3010 are configured to identify at least one environmental condition that interferes with the operation of one or both of the first control system 3020 and the second control system 3030 based on, for example, information detected by the AV’s 100 sensor.
  • Environmental conditions include rain, snow, fog, dust, insufficient sun light, or other conditions that may cause responsive steering/throttle operations to become more important.
  • slippery conditions caused by rain or snow may increase the importance of responsiveness corresponding to steering control.
  • the computer processors 3010 may select the control system with the highest measured performance pertaining to steering responsiveness.
  • throttle control responsiveness may become more important. In that case, the computer processors 3010 may choose the control system with the highest measured performance for throttle control responsiveness.
  • a redundant control system having two control systems capable of controlling the AV 100 mitigates the risks associated with control failure. Also, because the computer processors may select between control systems based on performance diagnostics, feedback, and environmental conditions, the driving performance of the AV 100 (in terms of accuracy and efficiency) may increase.
  • FIG. 31 shows a flowchart representing a method 3100 for providing redundancy in control systems according to at least one implementation of the present disclosure.
  • the redundant control system 3000 described above with reference to FIG. 30 performs the method 3100 for providing redundancy in control systems.
  • the method 3100 includes receiving operating information (block 3110), determining which control operation to affect (block 3120), and selecting a control system to affect the control operation (block 3130). Once the control system is selected, the method 3100 includes generating control functions (block 3140) and generating output by the selected control system (block 3150).
  • the method 3100 for providing redundancy in control systems includes receiving operating information (block 3110). This includes receiving, by at least one processor, information about an AV system, the AV system’s control systems, and/or the surrounding environment in which the AV is operating.
  • the at least one processor is the computer processors 3010 as previously described with reference to FIG. 30.
  • the computer processors 3010 receive information relating to performance statistics of each control system 3020, 3030 over a period of time. For instances, the performance statistics may relate to the responsiveness and/or the accuracy of each control system 3020, 3030. Diagnostics modules, such as the diagnostics module 3011 of FIG.
  • the received performance information is feedback information received from a feedback system.
  • the feedback systems may correspond to one or more control systems.
  • each control system corresponds to a separate feedback system.
  • a first control system may correspond to a first feedback system, while a second control system can correspond to a second feedback system.
  • the diagnostics module identifies a failure, either full or partial, of at least one control system based on the operating information received.
  • a failure can be based on a failure condition.
  • a failure condition can include a control system becoming at least partially inoperable or a control system consistently failing to provide a desired output.
  • the computer processors 3010 receive information about regarding environmental conditions, such as rain, snow, fog, dust, or other environmental conditions that can affect the AV system’s ability to detect, and navigate through, the surrounding environment.
  • the method 3100 also includes determining which control operation to affect (block 3120).
  • the computer processors determine which control operations to affect. This determination may be based on a planning module, as described previously with reference to FIG. 30.
  • the control operations may include throttle operations and/or steering operations.
  • the method 3100 further includes selecting a control system to affect the control operation (block 3130).
  • control systems such as the control systems 3020, 3030 of FIG. 30, may be configured to affect substantially similar control operations using either the same control devices or they may affect similar control operations using different control devices.
  • the computer processors utilize the received operating information to select which control system to use to affect the control operation. For instance, the computer processors may use the received performance statistics to analyze the performance of each control system and select the control system corresponding to the more desirable performance statistics (e.g., the control system with performance statistics that show a higher responsiveness or accuracy).
  • the computer processors may identify a failure (either full or partial) in one control system, and select another control system to affect control operations based on identifying the failure.
  • the computer processors may also use the received information relating to the environmental condition, and use this information to select which control system to use to affect control operations. For instance, assume that the AV is operating in rainy conditions, the computer processors may select the control system that may be more suitable for operating in rainy conditions.
  • the method 3100 includes generating control functions (block 3140). Once the control system is selected for use, the computer processors algorithmically generate and send control functions to the control systems. These control functions may be based on real time sensor data and/or prior information.
  • the method 3100 also includes generating output by the selected control system (block 3150).
  • the selected control system provides output that affects at least one control operation.
  • the output can be data useable for acceleration control and/or data useable for steering angle control.
  • the output can include control algorithms.
  • the algorithms can be feedback algorithms based on feedback received from feedback systems.
  • a first control system uses a first algorithm to affect control operations while a second control system uses a second algorithm to affect control operations.
  • one algorithm includes a bias towards adjusting steering angle as an adjustment technique.
  • one algorithm includes a bias towards adjusting throttle as an adjustment technique.
  • the output can be generated in accordance with at least one input.
  • the input may be input from a planning module that provides information used by the control system to choose a heading for the AV and determine which road segments to traverse.
  • the input may correspond to information received from a localization module, which provides information describing the AV’s current location so that the control system can determine if the AV is at a location expected based on the manner in which the AV’s devices are being controlled.
  • the input may also correspond to feedback modules, as described earlier with reference to FIG.
  • the input may also include information received from databases, computer networks, etc.
  • the input is a desired output.
  • the desired output may include speed and heading based on the information received by, for example, the planning module.
  • the control systems provide output based on the same input.
  • one control system provides output based on a first input, while another control system provides output based on a second input.
  • FIG. 32 shows an example of a sensor-related architecture of an autonomous vehicle 3205 (e.g., the AV 100 shown in FIG. 1) for detecting and handling sensor failure.
  • an autonomous vehicle 3205 e.g., the AV 100 shown in FIG. 1 for detecting and handling sensor failure.
  • the autonomous vehicle 3205 includes first sensor 32l0a, first buffer 32l5a, first multiplexer 3225a, second sensor 3210b, second buffer 3215b, second multiplexer 3225b, first transformer 3220a, second transformer 3220b, anomaly detector 3240, sensor selector 3235, and autonomous vehicle processor 3250.
  • sensors 32l0a-b include LiDAR, RADAR, camera, radio frequency (RF), ultrasound, infrared, and ultraviolet. Other types of sensors are possible. While two sensors are shown, the autonomous vehicle 3205 can use any number of sensors.
  • the sensors 32l0a-b are configured to produce respective sensor data streams from one or more environmental inputs such as objects, weather conditions, or road conditions external to the autonomous vehicle 3205 while the autonomous vehicle is in an operational driving state.
  • the processor 3250 uses the sensor data streams to detect and avoid objects such as natural obstructions, other vehicle, pedestrians, or cyclists.
  • the sensors 32l0a-b are configured to detect a same type of information.
  • the sensors 32l0a-b use one or more different sensor characteristics such as sensing frequencies, sensor placement, range of sensing signal, or amplitude of sensing signal.
  • the autonomous vehicle is in an operational driving state when the vehicle has been turned on or activated.
  • the processor 3250 is communicatively coupled with the sensors 32l0a-b via buffers 32l5a-b and multiplexers 3225a-b.
  • the sensors 32l0a-b produce sensor data streams that include samples generated by analog-to- digital converters (ADCs) within the sensors 32l0a-b. The samples from different streams are stored in respective buffers 32l5a-b.
  • the sensor selector 3235 is configured to control the multiplexers 3225a-b to switch among sensor data streams.
  • the sensor selector 3235 sends a signal to multiplexer 3225a to cause the stream from sensor 3210a to flow to the processor 3250, and sends a signal to multiplexer 3225b to cause the stream from sensor 3210b to flow to the processor 3250.
  • the anomaly detector 3240 is configured to detect an abnormal condition based on a difference between the sensor data streams being produced by respective sensors 32l0a-b.
  • an abnormal condition is detected based on one or more samples values that are indicative of a sensor malfunction or a sensor blockage such as one caused by dirt or another substance covering a sensor 32l0a-b.
  • an abnormal condition is detectable based on one or more missing samples. For example, the first sensor 32l0a may have produced a sample for a particular time index, but the second sensor 3210b did not produce a sample for the same time index.
  • an abnormal condition is a result of external intrusion or attack from a malicious actor on the AV 100 or sub-systems of the AV 100. For example, a hacker may attempt to gain access to AV 100 to send false data, steal data, cause AV 100 to malfunction, or for other nefarious purposes.
  • a transformer 3220a-b transforms a sensor data stream from a functioning sensor 32l0a-b to generate a replacement stream for a sensor 32l0a-b that is not functioning normally.
  • the sensor selector 3235 can send a signal to multiplexer 3225b to cause the output, e.g., replacement stream, from transformer 3220b to flow to the processor 3250.
  • the sensors 32l0a-b captures video of the road ahead of the autonomous vehicle 3205 at different angles such as from the left and right sides of the autonomous vehicle 3205.
  • transformer 3220b performs an affine transformation of the stream being produced by the left side sensor 32l0a to generate a replacement version of the stream that was being produced by the right-side sensor 3210b.
  • a video processing routine running on processor 3250 that is expecting two different camera angles can continue to function by using the replacement stream.
  • the sensors 32l0a-b captures images at different wavelength ranges such as visual and infrared.
  • a transformer transforms the infrared data into a visual range such that a routine configured to detect pedestrians using visual range image data can continue to function by using the transformed version of the infrared sensor stream.
  • the processor 3250 includes the anomaly detector 3240 and the sensor selector 3235.
  • the processor 3250 is configured to switch among the sensors 32l0a-b as an input to control the autonomous vehicle 3205.
  • the processor 3250 communicates with a diagnostic module to resolve the abnormal condition by performing tests or resets of the sensors 32l0a-b.
  • FIG. 33 shows an example of a process to operate an autonomous vehicle and sensors therein.
  • the autonomous vehicle produces, via a first sensor, a first sensor data stream from one or more environmental inputs external to the autonomous vehicle while the autonomous vehicle is in an operational driving state.
  • sensors include LiDAR, RADAR, camera, RF, ultrasound, infrared, and ultraviolet.
  • environmental inputs include nearby objects, weather conditions, or road conditions. Other types of environmental inputs are possible.
  • a processor performing this process within the autonomous vehicle is configured to send a command to cause a sensor to start producing a sensor data stream.
  • the autonomous vehicle produces, via a second sensor, a second sensor data stream from the one or more environmental inputs external to the autonomous vehicle while the autonomous vehicle is in the operational driving state.
  • the first sensor and the second sensor are configured to detect a same type of information.
  • these sensors can detect the same kinds of inputs such as a nearby object, weather condition, or road conditions.
  • the sensors can use one or more different sensor characteristics to detect the same type of information.
  • sensor characteristics include sensing frequencies, camera placement, range of sensing signal, and amplitude of sensing signal. Other types of sensor characteristics are possible.
  • the second sensor is identical to the first sensor by having the same sensor characteristics.
  • the second sensor operates under one or more different sensor characteristics such as different frequency, different range or amplitude, or different facing angle.
  • two sensors can detect the same type of information, e.g., the presence of a road hazard, by using two different frequency ranges.
  • the autonomous vehicle determines whether there is an abnormal condition based on a difference between the first and second sensor data streams.
  • an abnormal condition include a sensor value variance exceeding a threshold or a sensor or system malfunction. Other types of abnormal conditions are possible.
  • the difference may occur based on one or more missing samples in one of the sensor data streams.
  • the difference is determined by comparing values among two or more sensor data streams.
  • the difference is determined by comparing image frames among two or more sensor data streams. For example, dirt blocking one camera sensor but not the other may produce image frames with mostly black pixels or pixel values that do not change from frame-to-frame, whereas the unblock camera sensor may produce image frames having a higher dynamic range of colors.
  • the difference is determined by comparing values of each stream to historic norms for respective sensors.
  • the difference is determined by counting the number of samples obtained within a sampling window for each stream.
  • the difference is determined by computing a covariance among sensor streams.
  • the autonomous vehicle determines whether an abnormal condition has been detected.
  • a predetermined number of missing sensor samples can trigger an abnormal condition detection.
  • a sample deviation among different streams that is greater than a predetermined threshold triggers an abnormal condition detection.
  • a sensor reports a malfunction code, which in turn, triggers an abnormal condition detection.
  • the autonomous vehicle uses the first sensor and the second sensor to control the autonomous vehicle.
  • the sensor data streams are used to avoid hitting near-by objects, adjust speed, or adjust braking.
  • the autonomous vehicle forwards samples from one or more of the sensors’ streams to an autonomous vehicle’s control routine such as a collision avoidance routine.
  • the autonomous vehicle switches among the first sensor, the second sensor, or both the first and second sensors as an input to control the autonomous vehicle in response to the detected abnormal condition. In some
  • the autonomous vehicle if the first sensor is associated with the abnormal condition, the autonomous vehicle switches to the second sensor’s stream or a replacement version derived from the second sensor’s stream. In some implementations, the autonomous vehicle performs, in response to the detection of the abnormal condition, a diagnostic routine on the first sensor, the second sensor, or both to resolve the abnormal condition.
  • the autonomous vehicle accesses samples from different sensor data streams that correspond to a same time index and computes the difference at 3315 based on the samples.
  • An abnormal condition is detected based on the difference exceeding a predetermined threshold.
  • a difference for each stream is determined based on a comparison to the stream’s expected values.
  • the autonomous vehicle accesses samples from different sensor data streams that correspond to a same time range, computes an average sample value for each stream, and computes the difference at 3315 based on the averages.
  • the difference between the first and second sensor data streams is based on a detection of a missing sample within a sensor data stream.
  • a sensor may experience a temporary or partial failure that results in one or more missing samples, e.g., a camera misses one or more frames.
  • the autonomous vehicle may drop a sample due to events such as vehicle network congestion, a processor slow-down, external attack (for example by a hacker), network intrusion, or a sample storage overflow. Missing samples can trigger the autonomous vehicle to switch to another sensor.
  • one sensor system uses the data output by the other sensor system to detect an abnormal condition, e.g., as previously described in reference to FIGS. 13-29.
  • FIG. 34 shows an example of a process to detect a sensor-related abnormal condition.
  • the autonomous vehicle controls a duration of the sampling time window responsive to a driving condition.
  • driving conditions such as fast speeds, weather conditions, and road conditions such rough or unpaved roads may provide less accurate sensor readings or more variance among samples.
  • the sampling time window is increased.
  • the duration of the sampling time window is predetermined.
  • the autonomous vehicle captures a first set of data values within a first sensor data stream over a sampling time window.
  • data values are stored in a buffer.
  • the autonomous vehicle captures a second set of data values within a second sensor data stream over the sampling time window.
  • the autonomous vehicle detects an abnormal condition based on a deviation between the first set of data values and the second set of data values.
  • the autonomous vehicle operates an anomaly detector to determine a difference among two or more sets of data values.
  • a blocked sensor produces a low-variance series of data values, whereas an unblocked sensor produces a higher dynamic range of data values.
  • the corresponding camera sensor produces values with minimal or no variation in color, brightness, or both.
  • the sensor will produce different values than the mud example, but will still produce values with minimal or no variation in pixel values.
  • the camera lens is free from obstructions or debris, then the camera produces values with more range in values such as more variations in color and brightness. Such a deviation in respective sets of data values may trigger an abnormal condition event.
  • FIG. 35 shows an example of a process to transform a sensor data stream in response to a detection of an abnormal condition.
  • the process provides first and second sensor data streams to a controller of an autonomous vehicle.
  • two data streams are used.
  • additional data streams can be provided to the controller.
  • the process determines whether an abnormal condition is detected within the first sensor data stream. At 3505, if an abnormal condition is not detected, the process continues to provide the sensor data streams. At 3515, if an abnormal condition is detected, the process performs a transformation of the second sensor data stream to produce a replacement version of the first sensor data stream. In an embodiment, performing the transformation of the second sensor data stream includes accessing values within the second sensor data stream and modifying the values to produce a replacement stream that is suitable to replace the first sensor data stream. In some implementations, modifying the values includes applying a transformation such as an affine-transformation. Examples of affine- transformations include translation, scaling, reflection, rotation, shear mapping, similarity transformation, and compositions of them in any combination and sequence.
  • modifying the values includes applying filters to change voltage ranges, frequencies, or both. For example, in some implementations, if the output value range of the second sensor is greater than the first sensor, the second sensor values is compressed to fit within the expected range of values for the first sensor. In some implementations, if the output frequency range of the second sensor is different than the first sensor, the second sensor values are compressed and/or shifted to fit within the expected frequency range for the first sensor.
  • the process provides the second sensor data stream and the replacement version of the first sensor data stream to the controller.
  • the process performs a diagnostic routine on the first sensor.
  • the diagnostic routine includes performing sensor checks, resets, or routines to identify what sensor component has failed, etc.
  • the process determines whether the abnormal condition is resolved.
  • the process receives a sensor status update which reports that the sensor is functioning.
  • the process detects that a sensor is producing samples again.
  • the process detects that the different sensor data streams once again have similar statistical properties. For example, in some implementations, the process computes running averages for each stream and determine whether the averages are within an expected range. In some implementations, the process computes running averages for each stream and determine whether a difference among the averages does not exceed a predetermined threshold. In some implementations, the process computes a deviation for each stream and determines whether the deviation does not exceed a predetermined threshold.
  • the process continues to provide the nominal, untransformed sensor data streams to the controller.
  • the process continues to perform a transformation on the next set of data within the second sensor data stream.
  • an AV includes primary and secondary sensors.
  • an AV controller can determine whether the secondary sensor is identical to the primary sensor or if the secondary sensor has one or more different parametric settings, physical settings, or type. If identical, the AV controller can substitute the primary sensor data stream with the secondary sensor data steam. If different, the AV controller can transform raw sensor data from the secondary sensor to extract the desired information. In some implementations, if two cameras are facing the road at different angles, the data from the secondary camera is affme-transformed to match the primary camera's field of view.
  • the primary sensor is a visual range camera (e.g., for detecting pedestrians) and the secondary sensor is an infrared range camera (e.g., for detecting heat signatures of objects and/or to confirm detection of an object based on heat signature, etc.). If the visual range camera experiences an issue, the AV controller transforms the infrared data into a visual range such that a visual-range-based image processing algorithm can continue to detect pedestrians.
  • FIG. 36 illustrates example architecture of a teleoperation system 3690.
  • a teleoperation system 3690 includes a teleoperation client 3601 (e.g., hardware, software, firmware, or a combination of two or more of them), typically installed on an AV 3600 of an AV system 3692.
  • the teleoperation client 3601 interacts with components (e.g., sensors 3603, communication devices 3604, user interface devices, processor 3606, a controller 3607, or functional devices, or combinations of them) of the AV system 3692, for example, sending and receiving information and commands.
  • the teleoperation client 3601 communicates over a communication network 3605 (e.g., local network 322 and/or Internet 328 that may be at least partly wireless) with a teleoperation server 3610.
  • a communication network 3605 e.g., local network 322 and/or Internet 328 that may be at least partly wireless
  • a teleoperation server 3610 is located in a remote location away from the AV 3600.
  • the teleoperation server 3610 communicates with the teleoperation client 3601 using the communication network 3605.
  • the teleoperation server 3610 communicates simultaneously with multiple teleoperation clients; for example, the teleoperation server 3610 communicates with another teleoperation client 3651 of another AV 3650 that is part of another AV system 3694.
  • the clients 3601 and 3651 communicate with one or more data sources 3620 (e.g., a central server 3622, a remote sensor 3624, and a remote database 3626 or combinations of them) to collect data (e.g., road networks, maps, weather, and traffics) for implementing autonomous driving capabilities.
  • data sources 3620 e.g., a central server 3622, a remote sensor 3624, and a remote database 3626 or combinations of them
  • the teleoperation server 3610 also communicates with the remote data sources 3620 for teleoperations for the AV system 3692 or 3694 or both.
  • a user interface 3612 presented by the teleoperation server 3610 allows a human teleoperator 3614 to engage in teleoperations for the AV system 3692.
  • the interface 3612 renders to the teleoperator 3614 what the AV system 3692 has perceived or is perceiving. The rendering is typically based on sensor signals or based on simulations.
  • the user interface 3612 is replaced by an automatic intervention process 3611 that makes any decisions on behalf of the teleoperator 3614.
  • the human teleoperator 3614 uses augmented reality (AR) or virtual reality (VR) devices to engage in teleoperations for the AV system 3692.
  • AR augmented reality
  • VR virtual reality
  • the human teleoperator 3614 is seated in a VR box or use VR headsets to receive sensor signals in real time.
  • the human teleoperator 3614 utilizes an AR headset to project or
  • the teleoperation client 3601 communicates with two or more teleoperation servers that send and aggregate various information for a single teleoperator 3614 to conduct a teleoperation session on a user interface 3612.
  • the teleoperation client 3601 communicates with two or more teleoperation servers that present individual user interfaces to different teleoperators, allowing the two or more teleoperators to jointly participate in a teleoperation session.
  • the teleoperation client 3601 includes logic for deciding which of the two or more teleoperators to participate in the teleoperation session.
  • automatic processes automate teleoperation on behalf of the interfaces and teleoperators.
  • the two or more teleoperators use AR and VR device to collaboratively teleoperate the AV system 3692.
  • each of the two or more teleoperators teleoperate a separate subsystem of the AV system 3692.
  • a teleoperation request is generated, which requests the teleoperation system to begin an interaction between the AV and a remote operator (a tele-interaction) with the AV system 3692.
  • the teleoperation system allocates an available teleoperator and present the teleoperation request to the teleoperator.
  • the teleoperation request includes information (e.g., a planned trajectory, a perceived environment, a vehicular component, or a combination of them, among other things) of the AV system 3692.
  • the AV system 3692 implements a fallback or default operation.
  • FIG. 37 shows an example architecture of a teleoperation client 3601.
  • the teleoperation client 3601 is implemented as a software module, stored on memory 3722, being executed by a processor 3720, and includes a teleoperation handling process 3736 that requests the teleoperation system to begin a tele-interaction with the AV system.
  • the teleoperation client 3601 is implemented as hardware including one or more of the following: a data bus 3710, a processor 3720, memory 3722, a database 3724, a controller 3734 and a communication interface 3726.
  • the AV system 3692 operates autonomously. Tele-interactions can vary once the teleoperator 3614 accepts the teleoperation request and engages in the tele interaction.
  • the teleoperation server 3610 recommends possible teleoperations through the interface 3612 to the teleoperator 3614, and the teleoperator 3614 selects one or more of the recommended teleoperations and causes the teleoperator sever 3610 to send signals to the AV system 3692 that causes the AV system 3692 to execute the selected teleoperations.
  • the teleoperation server 3610 renders an environment of the AV system through the user interface 3612 to the teleoperator 3614, and the teleoperator 3614 analyzes the environment to select an optimal teleoperation.
  • the teleoperator 3614 enters computer codes to initiate certain teleoperations. For example, the teleoperator 3614 uses the interface 3612 to draw a recommended trajectory for the AV along which to continue its driving.
  • the teleoperator 3614 issues a suitable teleoperation, which is then processed by a teleoperation handling process 3736.
  • the teleoperation handling process 3736 sends the teleoperation request to the AV system 3692 to affect the autonomous driving capabilities of the AV 3600.
  • the AV system completes the execution of the teleoperation (or aborts the teleoperation) or the teleoperation is terminated by the teleoperator 3614, the teleoperation ends.
  • the AV system 3692 returns to autonomous mode and the AV system 3692 listens for another teleoperation event.
  • FIG. 38 illustrates an example teleoperation system 3800.
  • the teleoperation client 3601 (in FIGS. 36 and 37) is integrated as a part of an AV system 3692 (similar to AV system 3810).
  • the teleoperation client 3601 is distinct from the AV system 3692 and maintains communication with the AV system 3692 through a network link.
  • the teleoperation client 3601 includes an AV system monitoring process 3820, a teleoperation event handling process 3830, and a teleoperation command handling process 3840.
  • the AV system monitoring process 3820 reads system information and data 3692 for analysis, for example determining a status of the AV system 3692.
  • An analysis result generates a teleoperation event 3822 to the teleoperation event handling process 3830.
  • the teleoperation event handling process 3830 may send out a teleoperation request 3834 to a teleoperation server 3850 and a fallback request 3832 to the teleoperation command handling process 3840.
  • the teleoperation server 3850 presents a user interface 3860 for a teleoperator 3870 to perform tele-interaction with the AV system 3692.
  • the teleoperation server issues a teleoperation command 3852 that expresses the teleoperation in a form for use by the teleoperation command handling process 3840.
  • the teleoperation command handling process 3840 translates the teleoperation command into an AV system command 3842 expressed in a form useful for the AV system 3692 and sends the command to the AV system 3692.
  • the AV system monitoring process 3820 receives system information and data 3812 to monitor the operation status (e.g., velocity, acceleration, steering, data communications, perception, and trajectory planning) of the AV system 3692.
  • the operation status may be based on outputs of hardware components or software processes or both of the AV system 3692, or indirectly inferring, e.g., computationally or statistically, the outputs by measuring associated quantities, or both.
  • the AV system monitoring process 3820 derives information (e.g., computing a statistic, or comparing monitored conditions with knowledge in a database) from the operation status.
  • the monitoring process 3820 detects a teleoperation event 3822 based on the monitored operation status or derived information or both and generates a request for a teleoperation 3852.
  • a teleoperation event 3822 occurs when one or more components of the AV system 3692 (e.g., 120 in FIG. 1) is in an abnormal or unexpected condition.
  • the abnormal condition is a malfunction in the hardware of the AV system 3692. For instance, a brake malfunctions; a flat tire occurs; the field of view of a vision sensor is blocked or a vision sensor stops functioning; a frame rate of a sensor drops below a threshold; the movement of the AV system 3692 does not match with a current steering angle, a throttle level, a brake level, or a combination of the above.
  • abnormal conditions include malfunctions in software resulting in errors, such as a fault software code; a reduced signal strength such as a reduced ability to communicate with the communication network 3605 and thus with a teleoperator 3870; an increased noise level; an unknown object perceived in the environment of the AV system 3692; a failure of the motion planning process to find a trajectory towards the goal due to a planning error; inaccessibility to a data source (e.g., a database 3602 or 3626, a sensor, or a map data source); or combinations of the above.
  • the abnormal condition is a combination of hardware and software malfunctions.
  • the abnormal conditions occur as a result of abnormal environmental factors, for example heavy rain or snow, extreme weather conditions, presence of unusually high number of reflective surfaces, traffic jams, accidents etc.
  • the AV system 3692 operates autonomously.
  • the control system 3607 (FIG. 36) affects control operations of the AV system 3692.
  • the control system 3607 includes the controller 1102 that controls the throttle/brake 1206 and steering angle actuator 1212 (FIG. 12).
  • the controller 3607 determines instructions for execution by control components such as the throttle/brake 1206 and steering angle actuator 112. These instructions then control the various components, e.g., the steering actuator or other functionality for controlling steering angle; the throttle/brake 1206, the accelerator, or other mobility components of the AV system 3692.
  • the AV system monitoring process 3820 includes a list of errors that generate a teleoperation event 3822. For example, critical errors such as a brake failure or a loss of visual data.
  • the AV system monitoring process 3820 detects a failure or an error and compares the detected error with the list of errors prior to generating a teleoperation event 3822.
  • the teleoperation event 3822 is sent to the teleoperation event handling process 3830 which sends a teleoperation request 3834 to the server 3850.
  • the teleoperator 3870 sends a teleoperation command 3852 to the teleoperation command handling process 3840 which is in communication with the teleoperation client 3601 via the communication interface 3604 that operates with the communication network 3605.
  • the communication interface 3604 can include a network transceiver (a Wi-Fi transceiver, and/or WiMAX transceiver, a Bluetooth transceiver, a BLE transceiver, an IR transceiver, etc.).
  • the communications network 3605 transmits instructions from an external source (e.g., from the teleoperator 3870 and via the server 3850) so that the teleoperation client 3601 receives the instructions.
  • the teleoperation client 3601 uses the instructions received from the external source (e.g., AV system command 3842 relayed from the teleoperator 3870) and determines instructions that are executable by the AV system 3692, such as by the throttle/brake 1206 and steering angle actuator 1212, enabling the teleoperator 3870 to control operations of the AV system 3692.
  • the external source e.g., AV system command 3842 relayed from the teleoperator 3870
  • the teleoperation client 3601 switches to using instructions received from the teleoperator 3870 when one or more specified conditions are detected that trigger a teleoperation event 3822. These specified conditions are based on one or more inputs from one or more of the sensors 3603.
  • the teleoperation client 3601 determines if data received from the sensors 3603 positioned on the vehicle meets the one or more specified conditions, and in accordance with the determination enables the teleoperator 3870 to control the AV system 3692 via the communications network 3605.
  • the specified conditions detected by the teleoperation client 3601 include an emergency condition such as a failure of software and/or hardware of the vehicle. For example, a brake, throttle, or accelerator malfunction, a flat tire, an engine error such as the vehicle running out of gas or battery charge; a sensor ceasing to provide useful data, or detection that the vehicle is not responding to rules or inputs.
  • controller 3607 to control by a teleoperator 3870 via the teleoperation client 3601 include input received from an occupant of the autonomous vehicle.
  • the occupant may be aware of an emergency not detected by the sensors (e.g., a medical emergency, a fire, an accident, a flood).
  • the user or occupant of the vehicle may press a button or activate the teleoperation command using one of the computer peripherals 132 coupled to computing devices 146 (FIG. 1) or in input device 314 or cursor controller 316 such as a mouse, a trackball, a touch-enabled display (FIG. 3).
  • This button is be located within an interior of the autonomous vehicle within easy reach of any occupant. In an embodiment, multiple buttons are available within the interior of the vehicle for multiple passengers.
  • the specified conditions causing activation of teleoperation include environmental conditions. These environmental conditions include weather-related conditions, such as a slippery road due to rain or ice, or loss of visibility due to fog or snow. Environmental conditions can be roadway -related, such as the presence of unknown objects on the road, a loss of lane markers (e.g., due to construction), or uneven surface due to road maintenance.
  • the teleoperation client 3601 determines if the autonomous vehicle is currently located on a previously untraveled road. Presence on a previously unknown road is one of the specified conditions and enables the telecommunications system to provide instructions to the teleoperation client 3601 (e.g., from the teleoperator 3870). A previously unknown or untraveled road can be determined by comparing the current location of the AV with those located in the database 3602 of the AV which includes a listing of traveled roads. The teleoperation client 3601 also communicates via the communications network 3605 to query remote information, such as remotely located database 134 or 3626. The teleoperation client 3601 compares the location of the vehicle to all databases available before determining that the current location of the vehicle is on an unknown road.
  • an autonomous vehicle 3600 includes simply a local controller 3607 that affects control operation of the autonomous vehicle 3600.
  • the second processor 3720 part of the teleoperation client 3601, is in communication with controller 3607.
  • the processor 3720 determines instructions for execution by the controller 3607.
  • the communications network 105 is in communication with the processor 3720 via communication device 3604, the telecommunications device configured to receive instructions from an external source such as the teleoperator 3614.
  • the processor 3720 determines instructions that are executable by the controller 3607 from the instructions received from the external source and is configured to enable the received instructions to control the controller 3607 when one or more specified conditions are detected.
  • the autonomous vehicle 3600 operates autonomously or is operated by a teleoperator 3614.
  • the AV system 3692 switches automatically between teleoperation and autonomous operation.
  • the AV 3600 has a controller 3607 that controls operation of the autonomous vehicle, with a processor 3606 is in communication with the controller 3607.
  • the processor 3606 determines instructions for execution by the controller 3607. These elements are part of the local control system.
  • a telecommunications device 3604 is in communication with the controller 3607.
  • the telecommunications device 3604 receives instructions from an external source such as a teleoperator 3614 (via teleoperation server 3610 on a communications network 3605).
  • the telecommunications device 3604 communicates with the AV system 3692 to send instructions to the teleoperation client 3601, which acts as a second, redundant control software module.
  • a processor 3720 that is part of the teleoperation client 3601 determines instructions that are executable by the controller 3607 from the instructions received from the external source (e.g., from the teleoperator 3614 via teleoperation server 3610).
  • the processor 3720 then takes control from the local controller 3607 when one or more specified conditions are detected.
  • the teleoperation client 3601 acts as a second, redundant control module that is part of and which also can control operation of the autonomous vehicle 3600.
  • the second controller 3734 is in communication with the second processor 3720, which determines instructions for execution by the second controller 3734.
  • the telecommunications network 105 is in communication with the processor 3734 via communication device 3604, which receives instructions from the teleoperator 3614.
  • the processor 3720 determines instructions that are executable by the second controller 3734 from the signals received from the teleoperator 3614 and relays the signals to the second controller 3734 to operate the vehicle when one or more specified conditions are detected.
  • the specified conditions indicating switch of control to the vehicle from local control (e.g., by local controller 3607) to control by a teleoperator 3614 via the teleoperation client 3601 includes input received from an occupant of the autonomous vehicle.
  • the occupant may be aware of an emergency not detected by the sensors (e.g., a medical emergency, a fire, an accident, a flood).
  • the user or occupant of the vehicle may press a button or activate the teleoperation command using one of the computer peripherals 132 coupled to computing devices 146 (FIG. 1) or in input device 314 or cursor controller 316 such as a mouse, a trackball, a touch-enabled display (FIG. 3).
  • This button is located within an interior of the autonomous vehicle within easy reach of any occupant. In an embodiment, multiple buttons are available within the interior of the vehicle.
  • the specified conditions causing activation of teleoperation include environmental conditions. These environmental conditions include weather-related conditions, such as a slippery road due to rain or ice, or loss of visibility due to fog or snow. Environmental conditions can also be roadway-related, such as the presence of unknown objects on the road, a loss of lane markers (e.g., due to construction), or uneven surface due to road maintenance.
  • the teleoperation client 3601 determines if the autonomous vehicle is currently located on a previously untraveled road. Presence on a previously unknown road acts as one of the specified conditions and enables the telecommunications system to provide instructions to the teleoperation client 3601 (e.g., from the teleoperator 3870). A previously unknown or untraveled road can be determined by comparing the current location of the AV with those located in the database 3602 of the AV which includes a listing of traveled roads. The teleoperation client 3601 also communicates via the communications network 3605 to query remote information, such as remotely located database 134 or 3626. The teleoperation client 3601 compares the location of the vehicle to all databases available before determining that the current location of the vehicle is on an unknown road.
  • the AV system 3692 may sometimes not be able to communicate with a teleoperator 3614.
  • This communication failure can occur as a malfunction in the AV system 3692, such as a software malfunction or hardware malfunction (e.g., malfunction or damage of communication device 104).
  • the communication failure can occur as a malfunction of the teleoperation system, such as server 3610 going offline due to software failure or power loss.
  • the communication failure can also occur as a natural consequence of the AV 3600 moving around its environment and travelling into areas of reduced or absent network signal strength of the communications network 3605.
  • the loss of signal strength can occur in“dead zones” that lack, for example, Wi-Fi coverage, in tunnels, parking garages, under bridges, or in places surrounded by signal blocking features such as buildings or mountains.
  • the AV system 3692 employs a connectivity driving mode when in contact with the teleoperation system 3690, and a non-connectivity driving mode when not in contact with the teleoperation system.
  • the AV system 3692 detects that it has lost connection to a teleoperator 3614.
  • the AV system 3692 utilizes the connectivity driving mode and employs driving strategies with lower risk.
  • driving strategies with lower risk include reducing the velocity of the vehicle, increasing a following distance between the AV and a vehicle ahead, reducing the size of an object detected by the sensors that causes the AV vehicle to slow down or stop, etc.
  • the driving strategy may involve a single vehicle operation (e.g., change speed), or multiple vehicle operations.
  • the AV 3600 waits a period of time before switching from connectivity mode to non-connectivity mode, e.g., wait 2 seconds, 5 seconds, 60 seconds.
  • the delay allows the AV system 3692 to run diagnostics, or for the loss of connectivity to otherwise resolve itself (such as the AV 3600 clearing a tunnel) without causing frequent changes in the behavior of the vehicle.
  • the AV system 3692 has a controller 3607 that affects control operation of the AV 3600 during autonomous mode, and a second controller 3734 that affect control operations of the autonomous vehicle when in teleoperator mode.
  • the telecommunications device 104 is communication with the second controller module 3734, the telecommunications device 104 being part of a communications network 105 and configured to receive instructions from a teleoperator 3614 via teleoperation server 3610.
  • the teleoperation client 3601 includes a processor 3720 that relays or converts instructions to be readable by the controller 3734 and affect the control operations from the instructions received from the teleoperator 3614.
  • the processor 3720 also is configured to determine an ability of the telecommunications device 104 to communicate with the external source, e.g., communicate with communication network 3605. If the processor 3720 determines that communication is adequate, it sends a signal that local processor 3606 and controller 3607 controls the control operations, e.g., operate in connectivity mode. In an embodiment, the processor 3720 determines that communication is adequate and that signals are being received from the teleoperator 3614.
  • the processor 3720 relays instructions to the controller 3607, or alternatively, cause the processor 3734 of the teleoperation client 3601 to assume control of the control operations. In an embodiment, the processor 3720 determines that communication is with the communication network 3605 is not adequate. In such a circumstance, the processor 3720 loads non-connectivity driving strategies, e.g., from memory 3722. The processor 3720 sends these non-connectivity driving strategies to the controller 3607 or alternatively to the controller 3734.
  • the AV system 3692 continues to operate, but with a set of instructions different than during normal operations where intervention by a teleoperator 3614 can be expected.
  • the processor 3720 determines the ability of the telecommunications device 104 to communicate with the teleoperator 3614 by determining the signal strength of the wireless network. A threshold signal strength is chosen, and if the detected signal strength falls beneath this threshold the AV system 3692 switches to non-connectivity mode where the processor 3722 sends commands to the vehicle’s operational systems.
  • the processor 3606 uses an algorithm or set of algorithms for determining operations of the AV 3600.
  • the processor 3722 uses the same algorithm or set of algorithms.
  • the processor uses a second algorithm or set of algorithms different from the first.
  • the output of the first algorithms affects the operation of the AV to generate movements and behaviors that are more aggressive than an output of the second algorithms. That is, when in connectivity mode, the controller 3607 executes operations that have a higher risk (e.g., higher speed) than the operations executed when the vehicle is in non connectivity mode (and controlled by the controller 3822 for example).
  • the AV system 3692 When the AV system 3692 has lost human teleoperator intervention, it exhibits behavior that is more conservative (e.g., reduces speed, increases a following distance between the vehicle and a vehicle ahead, reduces the size of an object detected by the sensors that causes the AV vehicle to slow down or stop) than when teleoperation interventions is possible.
  • the output of the first algorithms affects the operation of the AV to generate movements and behaviors that are more conservative than an output of the second algorithms.
  • the AV system 3692 defaults to use of the more conservative set of instructions.
  • FIG. 39 shows a flowchart indicating a process 3900 for activating teleoperator control of an AV 3600 when an error is detected.
  • the process can be carried out by the teleoperation client 360lcomponent of the AV 3600.
  • an autonomous vehicle determines instructions for execution by a control system, at step 3902.
  • the control system is configured to affect a control operation of the autonomous vehicle.
  • a control processor is in communication with the control system and a
  • the control system can be the control system 3607 and the telecommunications system can be the telecommunications system 3605 of FIG. 36.
  • the telecommunications system receives instructions from an external source at step 3904.
  • the control processor determines instructions that are executable by the control system from the instructions received from the external source at step 3906. It also enables the external source in communication with the telecommunications system to control the control system when one or more specified conditions are detected, step 3908.
  • the control processor determines if data received from one or more sensors (e.g., sensors 3603 on FIG.
  • the telecommunications system receives instructions based on inputs made by a teleoperator (e.g. teleoperator 3614).
  • a teleoperator e.g. teleoperator 3614
  • FIG. 39 also shows a flowchart representing a process 3900 for activating redundant teleoperator and human control of an AV 3600.
  • the process can be carried out by the teleoperation client 360lcomponent of the AV3600.
  • an autonomous vehicle determines instructions for execution by a control system, at step 3902.
  • the control system can be the control system 3607 of FIG. 36.
  • the control system is configured to affect a control operation of the autonomous vehicle.
  • a control processor is in communication with the control system and is in communication with a telecommunications system.
  • the telecommunications system can be the telecommunications system 3605 of FIG. 36.
  • the telecommunications system receives instructions from an external source, step 3904, e.g., a teleoperator 3614 via a server 3600.
  • the control processor relays instructions that are executable by the control system from the instructions received from the external source, step 3906. In an embodiment, instructions are relayed or a computation takes place to convert the instructions to a usable format. It also enables the external source in communication with the telecommunications system to control the control system, step 3908. In an embodiment, the control processor enables the telecommunications system to operate the control system when one or more specified conditions are detected.
  • the specified condition is based on data received from one or more sensors on the autonomous vehicle or from an occupant of the autonomous vehicle or from a notification interface within an interior of the autonomous vehicle, and in accordance with the determination enables the telecommunications system to control the control system.
  • the one or more specified conditions detected by the control processor also include an emergency condition, environmental conditions, a failure of the control processor, if the autonomous vehicle is on a previously untraveled road (e.g., using data from a database of traveled roads.
  • the telecommunications system receives instructions based on inputs made by a teleoperator.
  • FIG. 40 shows a flowchart representing a process 4000 for controlling the operations of an AV 3600 according to different driving strategies depending on available connectivity to a teleoperator.
  • the process can be carried out by the teleoperation client 3601 of the AV 3600.
  • an autonomous vehicle receives instructions for execution by a control system from an external source, at step 4002.
  • the control system can be a first or a second control system of the autonomous vehicle, for example, controller 3607 of FIG. 36, or controller 3734 of FIG. 37.
  • a control processor is in communication with the control system and is in communication with a telecommunications system that transmits the instructions from the external source, for example processor 3720 or 3606.
  • the system determines instructions that are executable by the control system from the instructions received from the external source, step 4004.
  • the system determines an ability of the telecommunications system to communicate with the external source, step 4008, and then selects the first control system or the second control system and in accordance with the determination.
  • determining the ability of the telecommunications system to communicate with the external source includes determining a metric of signal strength of a wireless network over which the telecommunications system (e.g., telecommunications system 3605) transmits the instructions (step 4102 of flowchart 4100 in FIG. 41) or determining an indication that a wireless signal receiver on the autonomous vehicle is damaged.
  • the first control system uses a first algorithm and the second control system uses a second algorithm different from the first control system.
  • an output of the first algorithm affects the first control operation to generate a movement of the autonomous vehicle that is more aggressive or more conservative than an output of the second algorithm, and uses one algorithm as a default.
  • multiple autonomous vehicles exchange information with one another, and perform automated tasks based on the exchanged information.
  • each autonomous vehicle can individually generate and/or collect a variety of vehicle telemetry data, such as information regarding the autonomous vehicle itself (e.g., vehicle status, location, speed, heading or orientation, altitude, battery level, etc.), information regarding operations performed or to be performed by the autonomous vehicle (e.g., a route traversed by the autonomous vehicle, a planned route to be traversed by the autonomous vehicle, an intended destination of the autonomous vehicle, a task assigned to the autonomous vehicle, etc.), information regarding the environment of the autonomous vehicle (e.g., sensor data indicating objects in proximity to the autonomous vehicle, traffic information, signage information, etc.), or any other information associated with the operation of an autonomous vehicle.
  • This information can be exchanged between autonomous vehicles, such that each autonomous vehicle has access to a greater amount of information with which to conduct its operations.
  • This exchange of information can provide various technical benefits. For instance, the exchange information between autonomous vehicles can improve the redundancy of a fleet of autonomous vehicles as a whole, thereby improving the efficiency, safety, and effectiveness of their operation. As an example, as a first autonomous vehicle travels along a particular route, it might encounter certain conditions that could impact its operation (e.g., obstructions in the road, traffic congestion, etc.). The first autonomous vehicle can transmit information regarding these conditions to other autonomous vehicles, such that they also have access to this information, even if they have not yet traversed that same route.
  • certain conditions that could impact its operation e.g., obstructions in the road, traffic congestion, etc.
  • the other autonomous vehicles can preemptively adjust their operation to account for the conditions of the route (e.g., avoid that route entirely, traverse more slowly in a particular area, use certain lanes in a particular area, etc.) and/or better anticipate the conditions of the route.
  • the conditions of the route e.g., avoid that route entirely, traverse more slowly in a particular area, use certain lanes in a particular area, etc.
  • one or more additional autonomous vehicles can independently collect additional information regarding those conditions and/or any other conditions that the first autonomous vehicle did not observe, and transmit that information to other autonomous vehicles. Accordingly, redundant information regarding the route is collected and exchanged between the autonomous vehicles, thereby reducing the likelihood that any conditions are missed. Further, the autonomous vehicles can determine a consensus regarding the conditions of the route based on the redundant information, thereby improving the accuracy and reliability of the collective information (e.g., by reducing the likelihood of misidentification or misinterpretation of conditions).
  • the autonomous vehicles can operate in a more effective, safer, and more efficient manner.
  • FIG. 42 shows an example exchange of information among a fleet of autonomous vehicles 4202a-c in a region 4206.
  • one or more of the autonomous vehicles 4202a-c is implemented in a similar manner as the autonomous vehicle 100 described with respect to FIG. 1.
  • the fleet of autonomous vehicles 4202a-c exchange information directly with one another (e.g., via peer-to-peer network connections between them).
  • information is exchanged between autonomous vehicles 4202a and 4202b (e.g., as indicated by line 4204a).
  • information is exchanged between autonomous vehicles 4202b and 4202c (e.g., as indicated by line 4204b).
  • an autonomous vehicle can exchange information any other number of other autonomous vehicles (e.g., one, two, three, four, or more).
  • the fleet of autonomous vehicles 4202a-c exchange information through an intermediary.
  • each of the autonomous vehicles 4202a-c transmits information to a computer system 4200 (e.g., as indicated by lines 4204c- e).
  • the computer system 4200 can transmit some or all of the received information to one or more of the autonomous vehicles 4202a-c.
  • the computer system 4200 is remote from each of the autonomous vehicles 4202a-c (e.g., a remote server system).
  • the computer system 4200 is implemented in a similar manner as the remote servers 136 described with respect to FIG. 1 and/or the cloud computing environment 300 described with respect to FIGS. 1 and 3.
  • an autonomous vehicle can transmit information to another autonomous vehicle.
  • that autonomous vehicle can transmit some or all of the received information to another autonomous vehicle.
  • information from an autonomous vehicle can be transmitted to other multiple autonomous vehicles in a chain, such that the information is sequentially distributed among several autonomous vehicles.
  • the exchange of information is unidirectional (e.g., an autonomous vehicle transmits information to another autonomous vehicle, either directly or indirectly, but not receive any information from that autonomous vehicle in return).
  • the exchange of information is bidirectional (e.g., an autonomous vehicle transmits information to another autonomous vehicle, either directly or indirectly, and also receive information from that autonomous vehicle in return, either directly or indirectly).
  • information from one autonomous vehicle is exchanged with every other autonomous vehicle in a fleet. For instance, as shown in FIG. 42, information from the autonomous vehicle 4202b is shared with each of the other autonomous vehicles l02a and l02c. In some embodiments, information from one autonomous vehicle is exchanged with a subset of the other autonomous vehicle in a fleet. For instance, as shown in FIG. 1, information from the autonomous vehicle 4202a is shared with another autonomous vehicle l02b, but not shared with another autonomous vehicle l02c.
  • information is selectively exchanged between autonomous vehicles in a particular region (e.g., within the region 4206).
  • information can be exchanged between autonomous vehicles in a particular political region (e.g., a particular country, state, county, province, city, town, borough, or other political region), a particular pre-defmed region (e.g., a region having particular pre-defmed boundaries), a transiently- defined region (e.g., a region having dynamic boundaries), or any other region.
  • information is selectively exchanged between autonomous vehicles that are in proximity to each other (e.g., less than a particular threshold distance from one another). In some case, information is exchanged between autonomous vehicles, regardless of the region or their proximity to one another.
  • the autonomous vehicles 4202a-c and/or the computer system 4200 can exchange information via one or more communications networks.
  • a communications network can be any network through which data can be transferred and shared. For example, a
  • communications network can be a local area network (LAN) or a wide-area network (WAN), such as the Internet.
  • a communications network can be implemented using various networking interfaces, for instance wireless networking interfaces (such as Wi-Fi, WiMAX, Bluetooth, infrared, cellular or mobile networking, radio, etc.).
  • wireless networking interfaces such as Wi-Fi, WiMAX, Bluetooth, infrared, cellular or mobile networking, radio, etc.
  • the autonomous vehicles 4202a-c and/or the computer system 4200 exchange information via more than one communications network, using one or more networking interfaces.
  • vehicle telemetry data can include a variety of information.
  • vehicle telemetry data can include data obtained from one or more sensors (e.g., photodetectors, camera modules, LiDAR modules, RADAR modules, traffic light detection modules, microphones, ultrasonic sensors, time-of-flight (TOF) depth sensors, speed sensors, temperature sensors, humidity sensors, and precipitation sensors, etc.) ⁇
  • sensors e.g., photodetectors, camera modules, LiDAR modules, RADAR modules, traffic light detection modules, microphones, ultrasonic sensors, time-of-flight (TOF) depth sensors, speed sensors, temperature sensors, humidity sensors, and precipitation sensors, etc.
  • TOF time-of-flight
  • vehicle telemetry data can include information regarding a current condition of the autonomous vehicle.
  • this can include information regarding the autonomous vehicle’s location (e.g., as determined by a localization module having a GNSS sensor), speed or velocity (e.g., as determined by a speed or velocity sensor), acceleration (e.g., as determined by an accelerometer), altitude (e.g., as determined by an altimeter), and/or heading or orientation (e.g., as determined by a compass or gyroscope).
  • location e.g., as determined by a localization module having a GNSS sensor
  • speed or velocity e.g., as determined by a speed or velocity sensor
  • acceleration e.g., as determined by an accelerometer
  • altitude e.g., as determined by an altimeter
  • heading or orientation e.g., as determined by a compass or gyroscope
  • abnormalities related to the autonomous vehicle e.g., error indications, warnings, failure indications, etc.
  • this can include information indicating that one or more specific subcomponents of the autonomous vehicle are operating normally, or information indicating one or more abnormalities related to those subcomponents.
  • vehicle telemetry data can include information regarding historical conditions of the autonomous vehicle. For instance, this can include information regarding the autonomous vehicle's historical locations, speeds, accelerations, altitude, and/or heading or orientations. This can also include information regarding the historical statuses of the autonomous vehicle and/or one or more of its subcomponents.
  • vehicle telemetry data can include information regarding current and/or historical environmental conditions observed by the autonomous vehicle at a particular location and time. For instance, this can include information regarding a traffic condition of a road observed by the autonomous vehicle, a closure or an obstruction of a road observed by the autonomous vehicle, traffic volume and traffic speed observed by the autonomous vehicle, an object or hazard observed by the autonomous vehicle, weather observed by the autonomous vehicle, or other information.
  • vehicle telemetry data includes indications of a specific location and/or time in which an observation or measurement was obtained.
  • vehicle telemetry data can include geographical coordinates and a time stamp associated with each observation or measurement.
  • vehicle telemetry data also indicates a period of time for which the vehicle telemetry data is valid. This can be useful, for example, as autonomous vehicles can determine whether received data is sufficiently "fresh" (e.g., within 10 seconds, 30 seconds, 1 minute, 5 minutes, 10 minutes, 30 minutes, 1 hour, 2 hours, 3 hours, 12 hours, or 24 hours) for use, such that it can determine the reliability of the data.
  • an autonomous vehicle can indicate that information regarding the detected vehicle is valid for a relatively shorter period of time (e.g., as the detected vehicle is expected to remain at a particular location for a relatively short period of time).
  • the autonomous vehicle can indicate that information regarding the detected signage is valid for a relatively longer period of time (e.g., as signage is expected to remain at a location for a relatively longer period of time).
  • the period of time for which vehicle telemetry data is valid can vary, depending on the nature of the vehicle telemetry data.
  • the autonomous vehicle 4202a-c can exchange information according to different frequency, rates, or patterns.
  • the autonomous vehicles 4202a-c can exchange information periodically (e.g., in a cyclically recurring manner, such as at a particular frequency).
  • the autonomous vehicles 4202a-c can exchange information intermittently or sporadically.
  • the autonomous vehicles 4202a-c can exchange information if one or more trigger conditions are met (e.g., when certain types of information are collected by the autonomous vehicle, at a certain type of time, when certain events occur, etc.).
  • the autonomous vehicles can exchange information on a continuous or substantially continuous basis.
  • the autonomous vehicles 4202a-c exchange a subset of the information that they collect.
  • each autonomous vehicle 4202a-c can collect information (e.g., using one or more sensors), and selectively exchange a subset of the collected information with one or more other autonomous vehicles 4202a-c.
  • the autonomous vehicles 4202a-c exchange all or substantially all of the information that they collect.
  • each autonomous vehicle 4202a-c can collect information (e.g., using one or more sensors), and selectively exchange all or substantially all of the collected information with one or more other autonomous vehicles 4202a-c.
  • the exchange information between autonomous vehicles can improve the redundancy of a fleet of autonomous vehicles as a whole, thereby improving the efficiency, safety, and effectiveness of their operation.
  • autonomous vehicles can exchange information regarding conditions of a particular route, such that other autonomous vehicles can preemptively adjust their operation to account for those conditions and/or better anticipate the conditions of the route.
  • FIG. 43 shows two autonomous vehicles 4202a and 4202b in a region 4206.
  • the autonomous vehicles 4202a and 4202b are both traveling along a road 4300 (e.g., in directions 4302a and 4302b, respectively). As they navigate, the autonomous vehicles 4202a and 4202b each collect information regarding their respective operations and surrounding environments (e.g., vehicle telemetry data).
  • a hazard 4302 is present on the road 4300.
  • the hazard 4304 can be, for example, an obstruction to the road 4300, an object on or near the road 4300, a change in traffic pattern with respect to the road 4300 (e.g., a detour or lane closure), or another other condition that could impact the passage of a vehicle.
  • the leading autonomous vehicle 4202b encounters the hazard 4302, it collects information regarding the hazard 4302 (e.g., sensor data and/or other vehicle telemetry data identifying the nature of the hazard 4302, the location of the hazard, the time at which the observation was made, etc.).
  • the autonomous vehicle 4202b transmits some or all of the collected information to the computer system 4200 (e.g., in the form of one or more data items 4306).
  • the computer system 4200 transmits some or all of the received information to the autonomous vehicle 4202a (e.g., in the form of one or more data items 4308).
  • the autonomous vehicle 4202a is behind the autonomous vehicle 4202a along the road 4300 and has not yet encountered the hazard 4304, it has access to information regarding the hazard 4304.
  • the autonomous vehicle 4202a can take preemptive measures to account for the hazard 4302 (e.g., slow down as it approaches the hazard 4302, perform a lane change to avoid the hazard 4302, actively search for the hazard 4302 using one or more of its sensors, etc.). For example, as shown in FIG. 46, as the autonomous vehicle 4202a approaches the hazard 4302, it has access to the shared information from the autonomous vehicle 4202b, as well as information that the autonomous vehicle 4202a itself collects (e.g., based on its own sensors). Using this combined information, the autonomous vehicle 4202a can traverse the hazard 4302 in a safer and more effective manner.
  • preemptive measures to account for the hazard 4302 (e.g., slow down as it approaches the hazard 4302, perform a lane change to avoid the hazard 4302, actively search for the hazard 4302 using one or more of its sensors, etc.). For example, as shown in FIG. 46, as the autonomous vehicle 4
  • an autonomous vehicle modifies its route based on information received from one or more other autonomous vehicles. For example, if an autonomous vehicle encounters an obstruction, congestion, or any other condition that encumbers navigation over a particular portion of a road in a safe and/or efficient manner, other autonomous vehicles can modify their routes to avoid this particular portion of the road.
  • FIG. 47 shows two autonomous vehicles 4202a and 4202b in a region 4206.
  • the autonomous vehicles 4202a and 4202b are both traveling along a road 4700 (e.g., in directions 4702a and 4702b, respectively). As they navigate, the autonomous vehicles 4202a and 4202b each collect information regarding their respective operations and surrounding environments (e.g., vehicle telemetry data).
  • the autonomous vehicle is planning on navigating to a destination location 4704 along a route 4706 (indicated by a dotted line), using the road 4700.
  • the road 4700 is obstructed by a hazard 4708, preventing the efficient and/or safe flow of traffic past it.
  • the leading autonomous vehicle 4202b encounters the hazard 4708, it collects information regarding the hazard 4708 (e.g., sensor data and/or other vehicle telemetry data identifying the nature of the hazard 4302, the location of the hazard, the time at which the observation was made, etc.).
  • the autonomous vehicle 4202b can determine that the hazard 4708 cannot be traversed in a safe and/or efficient manner (e.g., the hazard 4708 blocks the road 4700 entirely, slows down through traffic to a particular degree, renders the road unsafe for passage, etc.).
  • the autonomous vehicle 4202b transmits some or all of the collected information to the computer system 4200 (e.g., in the form of one or more data items 4710).
  • the computer system 4200 transmits some or all of the received information to the autonomous vehicle 4202a (e.g., in the form of one or more data items 4712).
  • the autonomous vehicle 4202a is behind the autonomous vehicle 4202a along the road 4700 and has not yet encountered the hazard 4708, it has access to information regarding the hazard 4708 (e.g., information indicating that the hazard 4708 cannot be traversed in a safe and/or efficient manner).
  • the autonomous vehicle 4202a can modify its route to the location 4704.
  • the autonomous vehicle 4202a can determine, based on information from the autonomous vehicle 4202b, a length of time needed to navigate to the location 4704 using the original route 4706 (e.g., including a time delay associated with traversing the hazard 4708).
  • the autonomous vehicle 4202a can determine one or more alternative routes for navigating to the location 4704 (e.g., one or more route that avoid the portion of the road having the hazard 478). If a particular alternative route can be traversed in a shorter amount of time, the autonomous vehicle 4202a can modify its planned route to align with the alternative route instead.
  • the autonomous vehicle 4202a can determine, based on information from the autonomous vehicle 4202b, that the portion of the road 4700 having the hazard 4708 is impassible and/or cannot be safely traversed. Further, the autonomous vehicle 4202a can determine one or more alternative routes for navigating to the location 4704 that do not utilize the portion of the road 4700 having the hazard 4708. Based on this information, the autonomous vehicle 4202a can modify its planned route to align with the alternative route instead.
  • the autonomous vehicle 4202a can determine, based on the information received from the autonomous vehicle 4202b, that the portion of the road 4700 having the hazard 4708 is impassible and/or cannot be safely traversed.
  • the autonomous vehicle 4202a can determine an alternative route 4714 that bypasses the portion of the road 4700 having the hazard 4708 (e.g., a route that utilizes other roads 4716). Accordingly, the autonomous vehicle 4202a can navigate to the location 4704 using the route 4714 and avoid the hazard 4708, even if it has not yet encountered the hazard 4708 itself.
  • FIGS. 43-46 and 47-50 show the exchange of information regarding hazards, these are merely illustrative examples.
  • autonomous vehicles can exchange any information regarding any aspect of their surrounding environments to enhance the operation of the autonomous vehicles as a whole.
  • autonomous vehicles can exchange information regarding traffic or congestion observed along a particular route, signage observed along a particular route, landmarks observed along a particular route (e.g., buildings, trees, businesses, intersections, crosswalks, etc.), traffic patterns observed along a particular route (e.g., direction of flow, traffic lanes, detours, lane closures, etc.), weather observed along a particular route (e.g., rain, snow, sleet, ice, wind, fog, etc.) or any other information.
  • traffic or congestion observed along a particular route e.g., signage observed along a particular route, landmarks observed along a particular route (e.g., buildings, trees, businesses, intersections, crosswalks, etc.), traffic patterns observed along a particular route (e.g.,
  • autonomous vehicles can exchange information regarding changes to the environment (e.g., changes in traffic or congestion along a particular route, changes in signage along a particular route, changes to landmarks along a particular route, changes in traffic patterns along a particular route, changes in weather along a particular route, or any other change). Further, the autonomous vehicles can exchange information identifying the location at which the observations were made, the time at which those observations were made, and the period of time for which those observations are valid. Accordingly, each autonomous vehicle has access to not only the information that it collects itself, but also information collected by one or more other autonomous vehicles, thereby enabling it to traverse the environment in a safer and more effective manner. [00414] Further, although FIGS.
  • the autonomous vehicles autonomous 4202a and 4202b can exchange information from another intermediary (e.g., one or more other autonomous vehicles), or directly with one another (e.g., via peer-to-peer network connections).
  • another intermediary e.g., one or more other autonomous vehicles
  • peer-to-peer network connections e.g., via peer-to-peer network connections
  • two or more autonomous vehicles form a“platoon” while navigating to their respective destinations.
  • a platoon of autonomous vehicles can be, for example, a group of two or more autonomous vehicles that travel in proximity with one another over a period of time.
  • a platoon of autonomous vehicles is a group of two or more autonomous vehicles that are similar to one another in certain respects.
  • each of the autonomous vehicles in a platoon can have the same hardware configuration as the other autonomous vehicles in the platoon (e.g., the same vehicle make, vehicle model, vehicle shape, vehicle dimensions, interior layout, sensor configurations, intrinsic parameters, on-vehicle computing infrastructure, vehicle controller, and/or communication bandwidth with another vehicle or with a server.)
  • each of the autonomous vehicles in a platoon can have a particular hardware configuration from a limited or pre-defmed pool of hardware configurations.
  • a platoon of autonomous vehicles can travel such that they occupy one or more common lanes of traffic (e.g., in a single file line along a single lane, or in multiple lines along multiple lanes), travel within a certain area (e.g., a certain district, city, state, country, continent, or other region), travel at a generally similar speed, and/or maintain a generally similar distance from the autonomous vehicle ahead of it or behind it.
  • autonomous vehicles traveling in a platoon expend less power (e.g., consume less fuel and/or less electric power) than autonomous vehicles traveling individually (e.g., due to improved aerodynamic characteristics, fewer slowdowns, etc.).
  • one or more autonomous vehicle in a platoon directs the operation of one or more other autonomous vehicles in the platoon.
  • a leading autonomous vehicle in a platoon can determine a route, rate of speed, lane of travel, etc., on behalf of the platoon, and instruct the other autonomous vehicles in the platoon to operate accordingly.
  • a leading autonomous vehicle in a platoon can determine a route, rate of speed, lane of travel, etc., and the other autonomous vehicles in the platoon can follow the leading autonomous vehicle (e.g., in a single file line, or in multiple lines along multiple lanes).
  • autonomous vehicles form platoons based on certain similarities with one another. For example, autonomous vehicles can form platoons if they are positioned at similar locations, have similar destination locations, are planning on navigating similar routes (either in part, in or their entirety), and/or other similarities.
  • FIG. 51 shows two autonomous vehicles 4202a and 4202b in a region 4206.
  • the autonomous vehicle 4202a is planning on navigating to a location 5100a
  • the autonomous vehicle 4202b is planning on navigating to a location 5100b.
  • the autonomous vehicles 4202a and 4202b exchange vehicle telemetry data regarding their planned travel to their respective destination locations.
  • the autonomous vehicles 4202a and 4202b each transmits vehicle telemetry data to the computer system 4200 (e.g., in the form of one or more data items 5102a and 5102b, respectively).
  • the vehicle telemetry data can include, for example, an autonomous vehicle’s current location, its destination location, its heading or orientation, and a route that it plans on navigating to the destination location.
  • the computer system 4200 determines whether the autonomous vehicles 4202a and 4202b should form a platoon with one another. A variety of factors can be considered in determining whether autonomous vehicles should form a platoon. As an example, if two or more autonomous vehicles are nearer to each other, this can weigh in favor of forming a platoon. In contrast, if two or more autonomous vehicles are further from each other, this can weigh against forming a platoon.
  • the computer system 4200 transmits instructions to the autonomous vehicles 4202a and 4202b to form a platoon with one another (e.g., by transmitting instructions 5104a to the autonomous vehicle 4202a to form a platoon with the autonomous vehicle 4202b, and instructions 5104b to the autonomous vehicle 4202b to form a platoon with the autonomous vehicle 4202a).
  • the autonomous vehicles 4202a and 4202b form a platoon, and collectively navigate towards their respective destination locations (e.g., by convening at a particular location, and collectively heading in a direction 5104).
  • the autonomous vehicles 4202a and 4202b exchange of information through an intermediary computer system 4200.
  • autonomous vehicles exchange information directly with one another, and form platoons with one another without express instructions from a remote computer system.
  • FIG. 54 shows two autonomous vehicles 4202a and 4202b in a region 4206.
  • the autonomous vehicle 4202a is planning on navigating to a location 5400a
  • the autonomous vehicle is planning on navigating to a location 5400b.
  • the autonomous vehicles 4202a and 4202b exchange vehicle telemetry data directly with one another regarding their planned travel to their respective destination locations.
  • the autonomous vehicles 4202a and 4202b each transmits vehicle telemetry data to the other (e.g., in the form of one or more data items 5402a and 5402b, respectively).
  • the vehicle telemetry data can include, for example, an autonomous vehicle’s current location, its destination location, its heading or orientation, and a route that it plans on navigating to the destination location.
  • one or both of the autonomous vehicles 4202a and 4202b can determine whether to form a platoon. As described above, a variety of factors can be considered in determining whether autonomous vehicles should form a platoon (e.g., similarities in the current location of the autonomous vehicles, destination locations of the autonomous vehicles, headings or orientations, and/or planned routes of the autonomous vehicles).
  • an autonomous vehicle determines whether to form a platoon with one or more other autonomous vehicles, and if so, transmits invitations to those autonomous vehicles to join the platoon.
  • Each invited autonomous vehicle can either accept the invitation and join the platoon, or decline the invitation and proceed without the platoon (e.g., travel with another platoon or travel individually).
  • the current locations of the autonomous vehicles 4202a and 4202b, their destination locations, and their planned routes are general similar. Based on this information, the autonomous vehicle 4202b determines that it should form a platoon with the autonomous vehicle 4202a, and transmits an invitation 5106 to the autonomous vehicle 4202a to join the platoon.
  • the autonomous vehicle 4202a in response, can transmit a response 5108 to the autonomous vehicle 4202b accepting the invitation.
  • the autonomous vehicles 4202a and 4202b in response to the acceptance of the invitation, form a platoon, and collectively navigate towards their respective destination locations (e.g., by convening at a particular location, and collectively heading in a direction 5410).
  • FIGS. 51-53 and 54-56 show examples of two autonomous vehicles forming a platoon, these are merely illustrative examples. In practice, any number of autonomous vehicles can form a platoon (e.g., two, three, four, or more).
  • autonomous vehicles dynamically join and/or leave a platoon, depending on the circumstances. For instance, an autonomous vehicle can join a platoon to navigate a particular portion of a route common to the autonomous vehicle and those of the platoon. However, when the route of the autonomous vehicle diverges from others of the platoon, the autonomous vehicle can leave the platoon, and either join another platoon or continue to its destination individually.
  • a platoon can also include one or more vehicles that are not autonomous and/or one or more vehicles that are not fully autonomous.
  • a platoon can include one or more autonomous vehicles that are capable of fully autonomous operation, but are currently being operated in a“manual” mode (e.g., being manually operated by human occupants).
  • instructions can be provided to the human occupant regarding the operation of her vehicle in accordance with the platoon (e.g., instructions to navigate to a certain location at a certain time, await other vehicles, travel in a particular lane of traffic, travel at a particular speed, maintain a particular distance from another vehicle ahead of it or behind it, etc.).
  • the instructions are generated by a computer system (e.g., the computer system 4200) and presented to the occupant of the vehicle for execution (e.g., using the occupant’s mobile electronic device, such as a smartphone, and/or an on-board electronic device in the vehicle).
  • the process 5700 can be performed, at least in part, using one or more of the systems described herein (e.g., using one or more computer systems, AV systems, autonomous vehicles, etc.). In some embodiments, the process 5700 is performed, in part or in its entirety, by an autonomous vehicle having one or more sensors (e.g., one or more LiDAR sensors, RADAR sensors, photodetectors, ultrasonic sensors, etc.).
  • sensors e.g., one or more LiDAR sensors, RADAR sensors, photodetectors, ultrasonic sensors, etc.
  • a first autonomous vehicle determines an aspect of an operation of the first autonomous vehicle based on data received from the one or more sensors (step 5710).
  • the first autonomous vehicle can collect and/or generate vehicle telemetry data regarding the planning a route of travel, the identification an object in the surrounding environment (e.g., another vehicle, a sign, a pedestrian, a landmark, etc.), the evaluation of a condition of a road (e.g., the identification of traffic patterns, congestion, detours, hazards, obstructions, etc. along the road to be traversed by the first autonomous vehicle), the interpretation of signage in the environment of the autonomous vehicle, or any other aspect associated with operating the first autonomous vehicle.
  • the data received from the one or more sensors includes an indication of an object in the environment of the autonomous vehicle (e.g., other vehicles, pedestrians, barriers, traffic control devices, etc.), and/or a condition of the road (e.g., potholes, surface water/ice, etc.).
  • sensors detect objects in proximity to the vehicle and/or road conditions, enabling the vehicle to navigate more safely through the environment. This information can be shared with other vehicles, improving overall operation.
  • the first autonomous vehicle also receives data originating at one or more other autonomous vehicles (step 5720).
  • the first autonomous vehicle can receive vehicle telemetry data from one or more other autonomous vehicles, such as nearby autonomous vehicles, other autonomous vehicles in a particular fleet of autonomous vehicles, and/or autonomous vehicles that traversed a particular section of a road or a particular route in the past.
  • the first autonomous vehicle uses the determination and the received data to carry out the operation (step 5730). For example, information collected or generated by the first autonomous vehicle can be enriched or supplemented with data originating at other autonomous vehicles to improve its overall operation (e.g., plan a more efficient route of travel, identify an object in the surrounding environment more accurately, evaluate a condition of a road more accurately, interpret signage in the environment of the autonomous vehicle more accurately, etc.).
  • information collected or generated by the first autonomous vehicle can be enriched or supplemented with data originating at other autonomous vehicles to improve its overall operation (e.g., plan a more efficient route of travel, identify an object in the surrounding environment more accurately, evaluate a condition of a road more accurately, interpret signage in the environment of the autonomous vehicle more accurately, etc.).
  • the first autonomous vehicle also shares information that it collects or generates with one or more other autonomous vehicles. For instance, the first autonomous vehicle can transmit at least a portion of the data received from the one or more sensors to at least one of the other autonomous vehicles. Accordingly, data available to the first autonomous vehicle can be shared with other autonomous vehicles, improving their overall operation.
  • the data originating at the one or more other autonomous vehicles includes an indication of a period of time for which the data originating at the one or more other autonomous vehicles is valid. This can be useful, for example, as autonomous vehicles can determine whether received data is sufficiently“fresh” for use, such that it can determine the reliability of the data.
  • the one or more other autonomous vehicles from which the first autonomous vehicle receives data may have traversed the road prior to the first autonomous vehicle traversing the road. Further, the data originating at the one or more other autonomous vehicles includes an indication of the condition of the road when the one or more other autonomous vehicles traversed the road. This can be useful, for example, as sensor data is shared among autonomous vehicles that traverse the same road, and thus is more likely to be relevant to each of the vehicles
  • the data originating at the one or more other autonomous vehicles includes an indication of one or more paths traversed by the one or more other autonomous vehicles. This can be usage, for example, as autonomous vehicles can share routing data to improve routing decisions.
  • data originating at the one or more other autonomous vehicles includes an indication of one or more modifications to a traffic pattern along the one or more paths traversed by the one or more other autonomous vehicles. This can be beneficial, for example, as autonomous vehicles can share changes in traffic patterns, such as a one-way street becoming a two-way street, to improve the future routing of other vehicles.
  • the data originating at the one or more other autonomous vehicles further includes an indication of one or more obstacles or obstructions along the one or more paths traversed by the one or more other autonomous vehicles. This can be useful, for example, as autonomous vehicles can share information regarding obstacle or obstructions, such as observed potholes or barriers, to improve the future routing of other autonomous vehicles.
  • the data originating at the one or more other autonomous vehicles includes an indication of a change with respect to one or more objects along the one or more paths traversed by the one or more other autonomous vehicles.
  • vehicles can share information regarding landmarks on the side of the road, such as trees or signs, to improve the future routing of other vehicles.
  • autonomous vehicles form platoons with one or more other autonomous vehicles, and collectively navigate towards their respective destination locations.
  • the first autonomous vehicle can determine, based on the data originating at the one or more other autonomous vehicles, that a destination of the one or more other autonomous vehicles is similar to a destination of the first autonomous vehicle.
  • the first autonomous vehicle can transmit a request or invitation to the one or more other autonomous vehicles to form a vehicular platoon. This can be useful, for example, as vehicles traveling to the same location can“platoon” to that location to expend less power (e.g., consume less fuel and/or less electric power).
  • the data originating at the one or more other autonomous vehicles includes an indication of a condition of the environment of the one or more other autonomous vehicle. Accordingly, autonomous vehicles can receive information regarding their surrounding environment from other vehicles, improving the reliability/redundancy of sensor systems.
  • an autonomous vehicle adjusts its planned route of travel based on information regarding an environmental condition received from one or more other autonomous vehicles.
  • the first autonomous vehicle can modify its route based on the indication of the condition of the environment of the one or more other autonomous vehicles. Accordingly, this enables autonomous vehicles to reroute themselves based on information received from other autonomous vehicles.
  • the data originating at the one or more other autonomous vehicles includes a status of the one or more other autonomous vehicles.
  • the status of the one or more other autonomous vehicles can include information regarding a location of the one or more other autonomous vehicles, a speed or velocity of the one or more other autonomous vehicles, or an acceleration of the one or more other autonomous vehicles. This can be beneficial, for example, as it enables vehicles to share telemetry data, such that vehicles can operate more consistently with respect to one another.
  • the autonomous vehicles exchange information via an intermediary, such as a central computer system.
  • the first autonomous vehicle can use a communications engine (e.g., a Wi-Fi, WiMAX, or cellular transceiver) of the first autonomous vehicle to transmit information to and/or receive information from an external control system configured to control an operation of the first autonomous vehicle and the one or more other autonomous vehicles (e.g., a central control system for
  • the autonomous vehicles directly exchange information (e.g., via peer-to-peer connections).
  • the first autonomous vehicle can use a communications engine (e.g., a Wi-Fi, WiMAX, or cellular transceiver) of the first autonomous vehicle to transmit information to and/or receive information from the one or more autonomous vehicles through one or more peer-to-peer network connections.
  • a communications engine e.g., a Wi-Fi, WiMAX, or cellular transceiver
  • redundancy can be implemented in an autonomous vehicle using information provided by one or more wireless communication devices that are located external to the autonomous vehicle.
  • wireless communication device means any device that transmits and/or receives information to/from one or more autonomous vehicles using one or more wireless communication protocols and technologies, including but not limited to: Bluetooth, Near Field, Wi-Fi, infrared, free-space optical, acoustic, paging, Cellular, satellite, microwave and television, radio broadcasting and dedicated short-range radio communication (DSRC) wireless protocol.
  • DSRC dedicated short-range radio communication
  • Wireless communication devices that are located external to the autonomous vehicle are hereinafter referred to as“external” wireless communication devices
  • wireless communication devices that are located on or in the autonomous vehicle are hereinafter referred to as“internal” wireless communication devices.
  • Wireless communication devices can be installed on or in: physical structures (e.g., buildings, bridges, towers, bridges, traffic lights, traffic signs, billboards), road segments, vehicles, aerial drones, mobile devices (e.g., smart phones, smart watches, fitness bands, tablet computers, identification bracelets) and carried or worn by humans or other animals (e.g., attached to a pet collar).
  • the wireless communication devices can receive and/or send radio frequency (RF) signals in a frequency range from about lMHz to about 10 GHz.
  • RF radio frequency
  • an external wireless communication device is configured to broadcast signals (unidirectional) over a wireless communication medium to one or more autonomous vehicles using one or more wireless communication protocols.
  • the external wireless communication device needs not pair with or “handshake” with the internal wireless communication device of the autonomous vehicle.
  • the external wireless communication device “pairs” with the internal wireless communication device to establish a bi-directional communication session with the internal wireless communication device.
  • the internal wireless communication device includes a receiver that decodes one or more messages in the signal, and parses or extracts one or more payloads from the messages (hereinafter referred to as“external message”).
  • the payloads include content that is used to implement redundancy in the autonomous vehicle, as described in reference to FIGS. 58-60.
  • An external message can have any desired format, including without limitation a header, payload and error detection and correcting codes, as described in reference to FIG.
  • one or more steps of authentication are required before the payload can be extracted from the message by the internal wireless communication device.
  • the payload is encrypted, and therefore must be decrypted before it can be read by the internal wireless communication device using cryptographic keys or other secret information.
  • the payload is accessible to the public without authentication or encryption (e.g., public broadcast messages). The contents of the payload are used to provide redundancy for various functions performed by the autonomous vehicle, including but not limited to: planning, localization, perception and control functions, as described in further detail below.
  • FIG. 58 shows a block diagram of a system 5800 for implementing redundancy in an autonomous vehicle using one or more external messages provided by one or more external wireless communication devices, according to an embodiment.
  • System 5800 includes AV 100 having internal wireless communication device 5801 that communicates with external wireless communication devices 5802-5805.
  • Wireless communication devices 5802-5805 communicate one or more external messages to AV 100 over communication links 5806a-5806b, respectively.
  • device 5802 is installed in another vehicle 5807 following AV 100
  • device 5804 is a cell tower transmitter
  • device 5805 is a roadside RF beacon
  • device 5803 is a mobile device (e.g., a smartphone or wearable computer) carried or worn by user 5808.
  • Each of devices 5802-5805 is wired or wirelessly coupled to one or more information sources that provide content for external messages that are related to the operational domain of the AV 100.
  • information sources include but are not limited to: storage devices, sensors, signaling systems and online services.
  • An example sensor is a stereo camera mounted on a building that captures images of a particular geographic region (e.g., a street intersection) or a speed sensor located on a road segment.
  • An example signaling system is a traffic signal at a street intersection.
  • Some examples of online services include but are not limited to: traffic services, government services, vehicle manufacturer or OEM services, over-the-air (OTA) services for software updates, remote operator services, weather forecast services, entertainment services, navigation assistance services, etc.
  • OTA over-the-air
  • cell tower 5804 is coupled to online service 58l0a through network 5809a
  • roadside RF beacon 5805 is coupled to online service 5810b through network 5809b, and is also coupled to storage device 5811 and speed sensor 5812.
  • external wireless communication device 5805 is a roadside RF beacon that is located on a road segment and is coupled to one or more speed sensors 5812 to detect the speed of the AV 100.
  • the AV 100 receives and decodes an RF signal broadcast by the external wireless communication device 5805 over communication link 5806c.
  • the RF signal includes a payload that includes speed data for AV 100 generated by the one or more speed sensors 5812. The AV 100 compares the speed data received from the wireless communication device 5805 with the speed detected by a speedometer or other sensor onboard the AV 100.
  • the AV 100 infers a failure of an onboard sensor (e.g., a speedometer) or subsystem of the AV 100 and performs a“safe stop” maneuver or other suitable action (e.g., slows down).
  • an onboard sensor e.g., a speedometer
  • a“safe stop” maneuver or other suitable action e.g., slows down.
  • external wireless communication device 5802 installed on the vehicle 5807 can send an external message to AV 100 that includes the driving state of AV 100 as observed by onboard sensors (e.g., LiDAR, stereo cameras) of vehicle 5807.
  • Driving state can include a number of driving parameters of AV 100 that are observed by vehicle 5807, including but are not limited to speed, lane information, unusual steering or braking patterns, etc.
  • This information captured by sensors of vehicle 5807 can be sent in a payload of an external message transmitted to AV 100 over communication line 5806a.
  • AV 100 compares this externally generated driving state with its internally generated driving state to discover any discrepancies between the driving parameters.
  • the AV 100 can initiate a“safe stop” maneuver or another action (e.g., slow down, steer the AV 100 into a different lane).
  • an external message from vehicle 5807 could include a driving state that indicates that the AV 100 is traveling in Lane 1 of a highway, wherein the onboard sensors of the AV 100 could indicate that the AV100 is traveling in Lane 2 of the highway due to a system or sensor failure.
  • the external message provided redundant control information that can be used to steer the AV 100 to the correct to Lane 1 or perform some other action like slow down or perform a“safe stop” maneuver.
  • an external wireless communication device can be used to enforce a speed limit or some other constraint on the operation of the AV 100.
  • law enforcement or state, city, or municipal authorities may enforce a speed limit of 30 mph in school zones or construction zones by transmitting control information to an AV through an external wireless communication device that prevents the AV from bypassing that speed limit while within the school zone or near a construction site.
  • the AV 100 can adjust its venting system automatically to close vents and recirculate air to avoid dust from entering the vehicle.
  • wireless communication device devices are used to safely guide the AV 100 (e.g., guide by wire) into a loading zone, charging station or other stopping places by computing distance measurements.
  • external wireless communication devices 5803-5805 can broadcast information about a particular geographic region in which they are located.
  • external wireless communication devices 5803-5805 can advertise to AV 100 when entering a school zone, construction site, loading zone, drone landing port, train track crossing, bridge, tunnel, etc.
  • Such location external information can be used to update maps, routing, and scene descriptions and to potentially place the AV 100 in an alert mode if necessary.
  • an external wireless communication device located in a school zone can advertise that the school is currently in session and therefore many students may be roaming in the school zone. This information may be different than a scene description provided by a perception module of the AV 100.
  • the AV 100 can be commanded to slow down, change its route or lane and/or adjust its sensors and/or scan rate to avoid collision with students.
  • an external wireless communication device located in a construction zone can advertise that construction activities are in progress, and if the construction zone is not included in the scene description, the AV 100 can be commanded to slow its speed, change lanes and/or compute a detour route to avoid the construction zone and a potential collision with construction workers and/or heavy machinery.
  • an external wireless communication device is coupled to one or more perception sensors such as cameras, LiDARs, RADARs, etc.
  • the external wireless communication device 5804 is positioned at an elevated position to provide an unobstructed view of a portion of the road segment traveled by AV 100.
  • the external wireless communication device 5804 is placed on utility tower provides a scene description to the AV 100.
  • the AV 100 compares the externally generated scene description with its internally generated scene description to determine if an object is missing from the internally generated scene description indicating a potential sensor failure.
  • the internally generated scene description may not include a yield sign on the road segment because the AV’s LiDAR is partially occluded by an object (e.g., a large truck).
  • an object e.g., a large truck.
  • a comparison of the externally and internally generated scene descriptions would discover the missing yield sign, causing the AV 100 to be controlled to obey the yield sign by slowing down or stopping until its onboard sensors indicate that the AV 100 can proceed.
  • an external wireless communication device is coupled to a traffic light and sends a signal indicating the traffic light state to the AV 100.
  • the AV 100 can establish a connection with an external wireless communication device coupled to the traffic light to receive a signal indicating the current state of the traffic light. If the external traffic light state is different from a traffic light state perceived by the AV 100 (e.g., perceived using its onboard camera sensors), the AV 100 can slow down or initiate a“safe stop” maneuver.
  • the external wireless communication device coupled to the traffic light can transmit an external message that indicates a time that the traffic signal will change, allowing the AV 100 to perform operations such as stopping or re-starting its engine in advance of the signal change to conserve power.
  • the external wireless communication device 5803 is a portable device (e.g., mobile phone, smart watch, fitness band, identification device) that is carried or worn by a pedestrian or animal.
  • the external wireless communication processor 5803 can send the location (or distance) and/or a speed of a pedestrian to the AV 100.
  • the AV 100 can compare the pedestrian’s location with an internally generate scene description. If there is a discrepancy, the AV 100 can perform a“safe stop” maneuver or other action.
  • the external wireless communication device 5803 can be programmed to provide identifying information such as indicating that the wearer is a child, a physically impaired person, an elderly person, a pet, etc.
  • signal strengths from a large number of external wireless communication devices received in a wireless signal scan by a vehicle can be used to indicate crowds of people that may not have been included in an internally generated scene description due to sensor failure or because the sensors were compromised (e.g., occluded by an object).
  • the wireless communication device 5801 of the AV 100 establishes a connection with three external wireless communication devices, and uses signal strength measurements and advertised locations of the externally wireless communication devices to determine the position of the AV 100 using, for example, a trilateration algorithm.
  • the position of AV 100 can be estimated by a cellular network or external sensors (e.g., external cameras) and provided to the AV 100 in the payload of an external message.
  • the AV 100 can compare the position generated from information provided by the external wireless communication devices with a position of the AV 100 computed by an onboard GNSS receiver or cameras using visual odometry. If a sensor is failing or providing poor navigation solutions (e.g., high horizontal position error), the position determined using externally generated information can be used by the AV 100 in a “safe stop” maneuver or other action.
  • vehicles that are parked and equipped with wireless communication device devices are used to form an ad hoc wireless network for providing position information to the AV 100.
  • parked or out-of-service vehicles that are located in the same geographic region and belong to the same fleet service can be used to provide short-range-communication-based localization service that is redundant to the GNSS receiver and visual odometer localization techniques performed by the AV 100.
  • the parked or out-of-service vehicles can transmit their locations to the cloud so the fleet can determine their locations or send their locations directly to AV 100.
  • the RF signals transmitted by the parked or out-of-service vehicles can be used by the AV 100, together with the known locations of the parked or out-of-service vehicles, to determine the location of the AV 100.
  • FIG. 59 illustrates an external message format 5900, according to an
  • External message format 5900 includes header 5902, public message 5904, one or more private (e.g., encrypted) messages 5906 and error detection/correction code 5906.
  • the public message 5904 and the one or more private message 5906 are collectively referred to as the“payload” of the external message.
  • the header 5902 includes metadata that can be used by wireless communication receivers to parse and decode the external message, including but not limited to: a timestamp and the number, type and size of each payload.
  • the public message 5904 is unencrypted and includes content that can be consumed by anyone wireless communication receiver, including but not limited to: traffic condition information, Amber alerts, weather reports, public service announcements, etc.
  • the one or more private messages 5906 are encrypted and include content that can be consumed by wireless communication receivers that are authorized to access the content, including but not limited to: more detailed traffic and weather reports, customized entertainment content, URLs to websites or portals, etc.
  • the external message format 5900 includes private messages 5906 that include content provided by different service providers and each private message requires a private key to decrypt that can be provided to subscribers of the services.
  • This feature allows different AV fleet services to use share a single external message to deliver their respective private messages 5906 to their subscriber base. Each fleet service can provide a private key to its subscribers to get enhanced or premium content delivered in a private message 5906 in the external message.
  • This feature allows a single external wireless communication device to deliver contents for a variety of different content providers rather than each content provider installing their own proprietary wireless communication device. For example, a city can install and operate wireless communication devices, and then license private message slots in the external message to the content providers for a license fee.
  • an external message can be received by single vehicle from an external wireless communication device, and then be rebroadcast by the single vehicle to other vehicles within the vicinity of the single vehicle, and therefore propagating the external message in a viral manner in geographic regions that are not within the coverage area of the external wireless communication device.
  • FIG. 60 shows an example process 300 for providing redundancy in an autonomous vehicle using external information provided by one or more external wireless communication devices according to an embodiment.
  • a method comprises: performing, by an AV, an autonomous driving function (e.g., localization, planning, perception, control functions) of the AV in an environment (6001); receiving, by an internal wireless communication device of the AV, an external message from an external wireless communication device (e.g., RF beacon, infrared device, free-space optical device, acoustic device, microwave device) that is located in the environment (6002) (e.g., installed in another vehicle, carried or worn on a pedestrian or animal, installed on a utility tower); comparing, by one or more processors of the AV, an output of the function with content of the external message or with data generated based on the content (6003) (e.g., comparing scene descriptions, comparing position coordinates of the AV, comparing driving states); and in accordance with results of the comparing, causing the AV
  • an autonomous driving function
  • AVs may have redundancy built-in to their critical systems.
  • redundant components are required to be compatible with a redundancy model to ensure the safe operation of the AV.
  • one sensor may use the data output by another sensor to determine if one of the sensors has have failed or will fail in the future, as previously described in reference to FIGS. 13-29. If an incompatible replacement component is installed that is redundant to another component of the AV, and the replacement component relies on data from the other component, the replacement component may cause the AV to malfunction.
  • Compatibility can include but is not limited to: compatibility in specifications (e.g., hardware, software and sensor attributes), version compatibility, compatible data rates, and algorithm compatibility (e.g., matching/detection algorithms).
  • compatibility in specifications e.g., hardware, software and sensor attributes
  • version compatibility e.g., version compatibility
  • compatible data rates e.g., data rates
  • algorithm compatibility e.g., matching/detection algorithms.
  • a replacement stereo camera may use a matching algorithm that is identical to a matching algorithm used in a corresponding LiDAR sensor, where the redundancy model requires that the two algorithms be different.
  • redundancy configuration process includes the basic PnP configuration steps but also performs additional steps to detect if the replacement component violates a redundancy model.
  • the components being added to the AV are PnP compatible, such that the components are capable of identifying themselves to an AV operating system (OS) and able to accept resource assignments from the AV OS.
  • OS AV operating system
  • a list of characteristics can be provided to the AV OS that describes the capabilities of the component in sufficient detail that the AV OS can determine if the component violates a redundancy model.
  • Some example characteristics include but are not limited to: the make, model and version of the hardware, and the software/firmware version for the component if the component uses software/firmware.
  • Other characteristics can be component specific performance specifications, such as range, resolution, accuracy and objection detection algorithm for a LiDAR sensor, or sensor resolution, depth resolution (for z axis), bit depth, pixel size, framerate, focal length, field-of-view (FOV), exposure range and matching algorithm (e.g., OpenCV Block Matcher, OpenCV SGBM matcher) for a stereo camera.
  • non-volatile firmware running on a host computer includes routines that collect information about the different components in the AV and allocate resources to the components.
  • the firmware also communicates this information to the AV OS, which uses the information to configure its drivers and other software to make the AV components work correctly in accordance with the redundancy model.
  • the AV OS sets up device drivers for the components that are necessary for the components to be used by AV applications.
  • the AV OS also communicates with the driver of the AV (or with a technician in a repair shop), notifying her of changes to the configuration and allowing the technician to make changes to resource settings if necessary. This communication may be through a display in the AV, through the display of diagnostic equipment, AV telematics data stream, or through any other suitable output mechanism.
  • FIG. 61 shows a block diagram of an example architecture 6100 for replacing redundant components in an AV.
  • architecture 6100 includes
  • each component hub 6105a, l05b operates as a data concentrator and/or router of data from components to computing platform 6102 (e.g., an automated driving server).
  • communication interface 6101 is a Peripheral Component Interconnect Express (PCIe) switch that provides hardware support for“I/O virtualization”, meaning upper layer protocols are abstracted from physical connections (e.g., HDBaseT connections).
  • PCIe Peripheral Component Interconnect Express
  • Components can be any hardware device with PnP capability, including but not limited to: sensors, actuators, controllers, speakers, I/O devices, etc.
  • the PnP function is performed by the BIOS firmware during a boot process.
  • the BIOS will follow a procedure to discover and configure the PnP components in the AV.
  • An example basic PnP configuration includes the following steps: 1) create a resource table of the available interrupt requests (IRQs), direct memory access (DMA) channels and I/O addresses, excluding any that are reserved for system components; 2) search for and identify PnP and non-PnP devices on AV buses or switches; 3) load the last known system configuration stored in non-volatile memory; 4) compare the current configuration to the last known configuration. If the current and last known configurations are unchanged; 5) continue with the boot.
  • IRQs interrupt requests
  • DMA direct memory access
  • a redundancy configuration is performed that includes searching a redundancy table (e.g., stored in storage device 6104) to determine if the new component forms redundant pair with another component of the AV, where the redundant pair of components must be compatible to not violate the redundancy model of the AV. If the new component 6113 is in the redundancy table, the list of characteristics (e.g., performance specifications, sensor attributes) provided by the new component 6113 are compared to a list of characteristics required by the redundancy model that is stored in storage device 6104.
  • a redundancy table e.g., stored in storage device 6104
  • the driver of the AV or a technician e.g., if the AV is in an auto repair shop
  • the AV may also be disabled so that it cannot be driven until a compatible component has been added that does not violate the redundancy model of the AV.
  • FIG. 62 shows a flow diagram of an example process 6200 of replacing redundant components in an AV.
  • Process 6200 begins by detecting a new component coupled to a data network of an AV (6201).
  • the component can be coupled to the data network through a PCIe switch.
  • Some examples of components include but are not limited to: sensors, actuators, controllers and hubs coupled to multiple components.
  • Process 6200 continues by the AV OS discovering the new component with AV OS (6201), and determining if the new component is a redundant component and has a counterpart redundant component (6202). For example, a redundancy table can be searched to determine if the new component is replacing a redundant component and therefore must be compliant with a redundancy model for the AV, as described in reference to FIG. 61.
  • process 6200 performs a redundancy configuration (6203).
  • process 6200 performs a basic configuration (6204).
  • the basic and redundant configuration steps were previously described with reference to FIG. 61.
  • the redundant configuration includes the basic configuration and additional steps to determine compliance of the new module with the redundancy model of the AV.
  • a perception module provides a scene description into an in scope check module that determines if the scene description is within the operational domain of the autonomous vehicle ("in-scope").
  • the operational domain of the autonomous vehicle is a geographic region in which the autonomous vehicle is operating, including all fixed and dynamic objects in the geographic region that are known to the autonomous vehicle.
  • An "in scope" condition is violated when a scene description includes one or more objects (e.g., new stop sign, construction zone, policeman directing traffic, invalid road network graph) that are not within the operational domain of the autonomous vehicle.
  • the perception module provides the scene description as input to two independent and redundant planning modules.
  • Each planning module includes a behavior inference module and a motion planning module.
  • the motion planning modules each generate a trajectory (or trajectory corridor) for the autonomous vehicle using a motion planning algorithm that takes as input the position of the autonomous vehicle and static map data.
  • the position of the autonomous vehicle is provided by a localization module, such as localization module 408, as described in reference to FIG. 4, or by a source external to the autonomous vehicle.
  • Each planning module receives the trajectory (or trajectory corridor) generated by the other planning module and evaluates the trajectory for a collision with at least one object in the scene description.
  • the behavior inference modules use different behavior inference models. For example, a first behavior inference module implemented by a first planning module can evaluate a trajectory (or trajectory corridor) generated by a second planning module using a constant-velocity (CV) and/or constant-acceleration (CA) model. Similarly, a second behavior inference module implemented in the second planning module can evaluate the first trajectory (or trajectory corridor) generated by the first planning module using a machine learning algorithm.
  • CV constant-velocity
  • CA constant-acceleration
  • data inputs/outputs of each planning modules are subjected to independent diagnostic monitoring and plausibility checks to detect hardware and/or software errors associated with the planning modules. Because there are no common cause failures between the redundant planning modules, it is unlikely that the redundant planning modules will fail at the same time due to hardware and/or software errors.
  • the results of the diagnostic monitoring and plausibility checks and the results of the trajectory evaluations determine an appropriate action for the autonomous vehicle, such as a safe stop maneuver or emergency braking.
  • one of the planning modules is used during nominal operating conditions and the other planning module is used for safe stopping in an ego-lane (hereinafter also referred to as "degraded mode").
  • the planning modules do not perform any functions other than evaluating the trajectory provided by the other planning module for collision with at least one object.
  • FIG. 63 shows a block diagram of a redundant planning system 6300, according to an embodiment.
  • System 6300 includes perception module 6301, in-scope check module 6302 and planning modules 6303a, 6303b.
  • Planning module 6303a further includes behavior inference module 6304a, motion planning module 6305a and onboard diagnostics (OBD) module l06a.
  • Planning module 6303b further includes behavior inference module 6304b, motion planning module 6305b and OBD module 6306a.
  • Perception module 6301 (previously described as perception module 402 in reference to FIG. 4) identifies nearby physical objects using one or more sensors.
  • the objects are classified into types (e.g., pedestrian, bicycle, automobile, traffic sign, etc.), and a scene description including the classified objects 416 (also referred to as a “scene description”) is provided to redundant planning modules 6303a, 6303b.
  • Redundant planning modules 6303a, 6303b also receive data (e.g., latitude, longitude, altitude) representing the AV position 418 from localization module 408 (shown in FIG. 4) or from a source external to the AV.
  • the scene description is provided over a wireless communications medium by a source external to the AV (e.g., a cloud-based source, another AV using V2V).
  • In-scope check module 6302 determines if the scene description is“in-scope” which means the scene description is within the operational domain of the AV. If“in-scope”, the in-scope check module 6302 outputs an in-scope signal. Depending on the defined operational domain of the AV, in-scope check module 6302 looks for“out-of-scope” conditions to determine if the operational domain of the AV has been violated. Some examples of out-of-scope conditions include but are not limited to: constructions zones, some weather conditions (for example, storms, heavy rains, dense fog, etc.), a policeman directing traffic and an invalid road network graph (e.g., a new stop sign, lane closure).
  • out-of-scope conditions include but are not limited to: constructions zones, some weather conditions (for example, storms, heavy rains, dense fog, etc.), a policeman directing traffic and an invalid road network graph (e.g., a new stop sign, lane closure).
  • the autonomous vehicle is unaware that it is operating out-of-scope, safe operation of the autonomous vehicle cannot be guaranteed (e.g., the autonomous vehicle may run a stop sign).
  • the failure of the AV to pass the“in-scope” check results in a safe stop maneuver.
  • the in-scope signal is input to planning modules 6303a, 6303b. If“in-scope,” motion planning modules 6305a, 6305b independently generate trajectories for the AV, which are referred to in this example embodiment as trajectory A and trajectory B, respectively.
  • the motion planning modules 6305a, 6305b use common or different motion planning algorithm, static map and AV position to independently generate the trajectories A and B, as described in reference to FIG. 9.
  • Trajectory A is input into behavior inference module 6304b of planning module 6303b and trajectory B is input into behavior inference module 6304a of planning module 6303a.
  • Behavior inference modules 6304a, 6304b implement different behavior inference models to determine if trajectories A and B will collide with at least one object in the scene description. Any desired behavior inference model can be used to determine a collision with an object in the scene description.
  • behavior inference module 6304a uses a constant-velocity (CV) model and/or a constant-acceleration (CA) model to infer object behavior
  • behavior inference module 6304b uses a machine learning model (e.g., a convolutional neural network, deep learning, support vector machine, classifier) to infer object behavior.
  • machine learning model e.g., a convolutional neural network, deep learning, support vector machine, classifier
  • Other examples of behavior inference models include but are not limited to: game-theoretic models, probabilistic models using partially observable Markov decision processes (POMDP), Gaussian mixture models parameterized by neural networks, nonparametric prediction models, inverse reinforcement learning (IRL) models and generative adversarial imitation learning models.
  • the output signals (e.g., Yes/No) of the behavior inference modules 6304a, 6304b indicate whether or not the trajectories A and/or B collide with at least one object in the scene description.
  • the output signals can be routed to another AV module to affect a“safe stop” maneuver or emergency braking, such as control module 406, as described in reference to FIG. 4.
  • a“safe stop maneuver” is a maneuver performed during an emergency (e.g., a system malfunction, an emergency stop initiated by a passenger in the autonomous vehicle, natural disasters, inclement weather conditions, road accidents involving the autonomous vehicle or other vehicles in the environment etc.) by the autonomous vehicle.
  • OBD 6306a and OBD 6306b provide independent diagnostic coverage for planning modules 6303a, 6303b, respectively, including monitoring their respective inputs/outputs and performing plausibility checks to detect hardware and/or software errors.
  • OBD 6306a and OBD 6306b output signals indicating the results of their respective diagnostic tests (e.g., Pass/Fail).
  • other output signals or data can be provided by OBD 6306a and OBD 6306b, such as codes (e.g., binary codes) indicating a type of failure and a severity level of the failure.
  • the output signals are routed to another AV module to affect a“safe stop” maneuver or emergency braking, such as control module 406 described in reference to FIG. 4.
  • FIG. 64 shows a table illustrating redundant planning logic performed by the redundant planning modules shown in FIG. 63.
  • Each row in the table represents a combination of output signals leading to a particular action to be performed by the AV.
  • the AV maintains a nominal operating condition.
  • rows 2 and 3 of the table if“in-scope” and the diagnostics covering planning module 6303a or 6303b indicate failure, there is a lost degree of redundancy and the AV initiates a“safe stop” maneuver in an ego lane.
  • the diagnostics of both planning modules l03a, l03b pass, and both planning modules l03a, l03b have detected collisions then the AV initiates an AEB using, for example, an Advanced Driver Assistance System (ADAS) component in the AV.
  • ADAS Advanced Driver Assistance System
  • only planning module l03a is used during nominal operating conditions and planning module l03b is used only for safe stopping in the ego lane when the AV is operating in a“degraded” mode.
  • FIG. 65 shows a flow diagram of a redundant planning process 6500.
  • Process 6500 can be implemented by the AV architecture shown in FIGS. 3 and 4.
  • Process 6500 can begin by obtaining a scene description of the operating environment from a perception module or external source, and a description of the AV operational domain (6501).
  • Process 6500 continues by determining (6502) if the scene description is within the operational domain of the AV (6502). If not, process 6500 stops. If yes, process 6500 determines (6503) if the diagnostics of one or both of the redundant planning modules indicate a failure of hardware and/or software. In accordance with determining a failure, a“safe stop” maneuver is initiated by the AV (6510).
  • process 6500 continues by generating, by a first planning module, a first trajectory using the scene description and the AV position (6505), and generating, by a second planning module, a second trajectory using the scene description and the AV position (6506).
  • Process 6500 continues by evaluating the second trajectory using a first behavior inference model of the first planning module for a collision, and evaluating the first trajectory using a second behavior inference model of the second planning module for a collision (6507).
  • determining (6508) that both the first and second trajectory are safe the AV operates under nominal conditions (6509) and redundancy is unaffected.
  • process 6500 determining (6511) that one of the first or second trajectories is unsafe, the AV performs a“safe stop” maneuver in an ego lane (6510). In accordance with process 6500 determining (6508) that the first and the second trajectories are unsafe, the AV performs emergency braking (6512) as a last resort.
  • Simulations of AV processes, subsystems and systems are used to provide redundancy for the processes/subsystems/systems by using the output of a first process/ subsystem/system as input into a simulation of a second process/ subsystem/system, and using the output of the second process/ subsystem/system as input into a simulation of the first process/ subsystem/system. Additionally, each process/ subsystem/system is subjected to independent diagnostic monitoring for software or hardware errors.
  • a redundancy processor takes as inputs the outputs of each process/ subsystem/system, the outputs of each simulation and the results of the diagnostic monitoring to determine if there is a potential failure of one or both of the processes or systems.
  • the autonomous vehicle performs a“safe stop” maneuver or other action (e.g., emergency brake).
  • a“safe stop” maneuver or other action e.g., emergency brake.
  • one or more external factors e.g., environmental conditions, road conditions, traffic conditions, AV characteristics, time of day
  • a driver profile e.g., age, skill level, driving patterns
  • “simulation” means an imitation of the operation of a real-world process or system of an AV sensor or subsystem, which may or may not be represented by a “model” that represents key characteristics, behaviors and functions of the process or system.
  • model means the purposeful abstraction of reality, resulting in a specification of the conceptualization and underlying assumptions and constraints of a real- world process or system.
  • FIG. 66 shows a block diagram of a system 6600 for implementing redundancy using simulations.
  • system 6600 includes interfaces 660la, 660lb, diagnostic modules 6602a, 6602b, simulators 6603a, 6603b and redundancy processor 6604.
  • Diagnostic modules 6602a, 6602b are implemented in hardware and/or software
  • simulators 6603a, 6603b are implemented in software that runs on one or more computer processors.
  • Data A from a first AV process/ subsystem/system is input to interface 101 a, which converts and/or formats Data A into a form that is acceptable to simulator 6603b.
  • the converted/formatted Data A is then input into diagnostic module 6602a, which monitors for hardware and software errors and outputs data or a signal indicating the result of the monitoring (e.g., Pass or Fail).
  • Data A is then input into simulator 6603b (“Simulator B”), which performs a simulation of a second AV process/ subsystem/system using Data A.
  • subsystem/system is input to interface lOlb, which converts and/or formats Data B into a form that is acceptable to simulator 6603a.
  • the converted/formatted Data B is then input into diagnostic module 6602b, which monitors for hardware and software errors and outputs data or a signal indicating the result of the monitoring (e.g., Pass or Fail).
  • Data B is then input into simulator 6603a (“Simulator A”), which performs a simulation of the first AV process/system using Data B.
  • system 6600 is implemented using real-time (RT) simulations and hardware-in-the-Loop (HIL) techniques, where hardware (e.g., sensors, controllers, actuators) is coupled to RT simulators 6603a, 6603b by I/O interfaces 660la, 660lb.
  • I/O interfaces 660la, 660lb include analog-to-digital (AD) and digital-to- analog (DAC) converters that convert analog signals output by the hardware to digital values that can be processed by the RT simulations.
  • the I/O interfaces 660la, 660lb can also provide electrical connections, power and data aggregation (e.g., buffers).
  • Redundancy process 6604 applies logic to these inputs to determine whether or not a failure of the first or second process/system has occurred. In accordance with determining that a failure of the first or second process/system has occurred, the AV performs a“safe stop” maneuver or other action. In accordance with determining that a failure of the first or second process/system has not occurred, the AV continues operating in nominal mode.
  • redundancy processor 6604 the logic implemented by redundancy processor 6604 is shown in Table I below.
  • diagnostic modules A and B do not indicate a failure and simulators A and B do not indicate a failure, the AV continues in a nominal mode of operation. If at least one diagnostic module indicates failure or one simulator indicates failure, the AV performs a safe stop maneuver or other action using the process/system that has not failed. If both simulators indicate failure, the AV performs emergency braking.
  • simulators 6603b, 6603a receive real-time data streams and/or historical data from storage devices 6605b, 6605a.
  • the data streams and storage devices l05a, l05b provide external factors and/or a driver profile to simulators 6603a, 6603b which use the external factors and/or driver profile to adjust one or more models of the
  • Some examples of external factors include but are not limited to: weather conditions (e.g., rain, snow, sleet, foggy, temperature, wind speed), road conditions (e.g., steep grades, closed lanes, detours), traffic conditions (e.g., traffic speed, accidents), time of day (e.g., daytime or nighttime), AV characteristics (e.g., make, model, year, configuration, fuel or battery level, tire pressure) and a driver profile (e.g., age, skill level, driving patterns).
  • the external factors can be used to adjust or“tune” one or more models in simulators 6603a, 6603b.
  • certain sensors e.g., LiDAR
  • An example driver profile includes the driver’s age, skill level and historical driving patterns.
  • the historical driving patterns can include but are not limited to:
  • Driving patterns can be learned over time using a machine learning algorithm (e.g., deep learning algorithm) implemented on a processor of the AV.
  • a machine learning algorithm e.g., deep learning algorithm
  • simulators 6603a, 6603b implement a virtual world using fixed map data and a scene description provided by the perception module 408 that includes the AV and other fixed and dynamic objects (e.g., other vehicles, pedestrians, buildings, traffic lights). Simulators 6603a, 6603b simulate the AV in the virtual world (e.g., 2D or 3D simulation) with the external factors and/or driver profile to determine how the AV will perform and whether a failure is likely to occur.
  • the virtual world e.g., 2D or 3D simulation
  • historical data stored in data storage devices 6605a, 6605b are used to perform data analytics to analyze past failures of AV processes/systems and to predict future failures of AV processes/systems.
  • the AV is traveling on a road segment in a nominal mode of operation.
  • the LiDAR outputs point cloud data that is processed by the perception module 402 shown in FIG. 4.
  • the perception module 402 outputs a first scene description that includes one or more classified objects (e.g., vehicles, pedestrians) detected from the LiDAR point cloud data.
  • the stereo camera captures stereo images which are also input into the perception module 402.
  • the perception module 402 outputs a second scene description of one or more classified objects detected from the stereo image data.
  • a first HIL process includes the LiDAR hardware coupled through the first I/O interface 6601 a to a first RT simulator 6603b that simulates operation of the stereo camera using the first scene description.
  • a second HIL process includes the stereo camera hardware coupled through the second I/O interface 660lb to a second RT simulator 6603a that simulates the LiDAR hardware using the second scene description. Additionally, both the LiDAR and stereo camera are monitored by independent diagnostic modules 6602a, 6602b, respectively, for hardware and/or software errors.
  • the simulators 6603a, 6603b are implemented on one or more hardware processors.
  • the I/O interfaces 660la, 660lb are hardware and/or software or firmware that provide electrical connections, supply power and perform data aggregation, conversion and formatting as needed for the simulators l03a, l03b.
  • the LiDAR simulator 6603b uses the position coordinates of the classified objects in the second scene description generated from the stereo camera data to compute a simulated LiDAR scene description.
  • LiDAR depth data can be simulated using the location of the AV obtained from localization module 408 (FIG. 4) and ray-casting techniques.
  • the stereo camera simulator 6603a uses the position coordinates of the objects in the first scene description generated from the LiDAR point cloud data to compute a simulated stereo camera scene description.
  • Each simulator l03a, l03b provides as output their respective simulated scene descriptions to redundancy processor 6604. Additionally, each of the diagnostic modules 6602a, 6620b outputs a pass/fail indicator to the redundancy processor 6604.
  • the redundancy processor 104 executes the logic shown in Table I above. For example, if the diagnostic modules l02a, l02b do not indicate that the LiDAR or stereo camera hardware or software has failed, the LiDAR scene description matches the simulated LiDAR scene description (e.g., all classified objects are accounted for in both scene descriptions), and the stereo camera scene description matches the simulated stereo camera scene description, then the AV continues to operate in nominal mode. If the LiDAR and stereo camera hardware or software have not failed, and one of the LiDAR or stereo camera scene description does not match its corresponding simulated scene description, the AV performs a“safe stop” maneuver or other action.
  • the AV performs a“safe stop” maneuver or other action. If the LiDAR and stereo camera do not have a hardware or software error, and neither the LiDAR nor the stereo camera scene descriptions match their simulated scene descriptions, the AV applies an emergency brake.
  • a GNSS receiver can be simulated using inertial data (e.g., IMU data), LiDAR map-based localization data, visual odometry data (e.g., using image data), or RADAR or vision-based feature map data (e.g., using non-LiDAR series production sensors).
  • inertial data e.g., IMU data
  • LiDAR map-based localization data e.g., LiDAR map-based localization data
  • visual odometry data e.g., using image data
  • RADAR or vision-based feature map data e.g., using non-LiDAR series production sensors.
  • one simulator uses the data output by the other simulator, e.g., as previously described in reference to FIGS. 13-29.
  • FIG. 67 shows a flow diagram of a process 6700 for redundancy using simulations.
  • Process 6700 can be implemented by system 400 shown in FIG. 4.
  • Process 6700 begins by performing, by a first simulator, a simulation of a first AV process/system (e.g., simulating a LiDAR) using data (e.g., stereo camera data) output by a second AV process/system (e.g., a stereo camera) (6701), as described in reference to FIG.
  • a simulation of a first AV process/system e.g., simulating a LiDAR
  • data e.g., stereo camera data
  • a second AV process/system e.g., a stereo camera
  • Process 6700 continues by performing, by a second simulator, a simulation of the first AV process/system using data output by the second AV process/system (6702).
  • Process 6700 continues by comparing outputs of the first and second processes and systems (e.g., scene descriptions based on LiDAR point cloud data and stereo camera data) with outputs of their corresponding simulated processes and systems (6703), and in accordance with determining (6704) that a failure has occurred (or will occur in the future based on a prediction model), causing the AV perform a“safe stop” maneuver or other action (6705). Otherwise, causing the AV to continue operating in nominal mode (6706).
  • the first and second processes and systems e.g., scene descriptions based on LiDAR point cloud data and stereo camera data
  • process 6700 includes monitoring, by independent diagnostic modules, the redundant processes or systems for hardware or software errors, and using the outputs of the diagnostic modules (e.g., pass/fail indicators) in combination with the outputs of the simulators to determine if a failure of one or both of the redundant processes or systems has occurred or will occur, and causing the AV to take action in response to the failure (e.g.,“safe stop” maneuver, emergency braking, nominal mode).
  • the diagnostic modules e.g., pass/fail indicators
  • FIG. 68 shows a block diagram of a vehicle system for unionizing perception inputs to model an operating environment, according to an embodiment.
  • a vehicle system 6800 includes two or more perception components, e.g., the perception components 6802 and 6803, each capable of independently performing a perception function with respect to the operating environment 6801.
  • Example perception functions include the detection, tracking, and classification of various objects and backgrounds present in the operating environment 6801.
  • the perception components 6802 and 6803 are components of the perception module 402 shown in FIG. 4.
  • the perception components implement both hardware and software-based perception techniques.
  • the perception component 6802 can include a hardware module 6804 consisting of complementary sensors such as LiDARs, RADARs, sonars, stereo vision systems, mono vision systems, etc., e.g., the sensors 121 shown in FIG. 1.
  • the perception component 6802 can further include a software module 6806 executing one or more software algorithms to assist the perception function.
  • the software algorithms can include feedforward neural networks, recurrent neural networks, fully convolutional neural networks, region-based convolutional neural networks, You-Only-Look-Once (YOLO) detection models, single-shot detectors (SDD), stereo matching algorithms, etc.
  • the hardware module 6804 and the software module 6806 can share, compare, and cross-check their respective perception outputs to improve an overall perception accuracy for the perception component 6802.
  • the perception components each perform an independent and complementary perception function. Results from different perception functions can be cross checked and fused (e.g., combined) by a processor 6810. Depending on the operating environment, one perception function may be more suited to detecting certain objects or conditions, and the other perception function may be more suited to detecting other objects or conditions, and data from one can be used to augment data from the other in a
  • the perception component 6802 can perform dense free space detection while the perception component 6803 can perform object-based detection and tracking.
  • a free space is defined as an area in the operating environment 6801 that does not contain an obstacle and where a vehicle can safely drive. For example, unoccupied road surfaces are free space but road shoulders (sometimes referred to as “breakdown lanes”) are not.
  • Free space detection is an essential perception function for autonomous/semi-autonomous driving as it is only safe for a vehicle to drive in free space.
  • the goal of object-based detection and tracking is to discover the current presence and to predict the future trajectory of an object in the operating environment 6801. Accordingly, data obtained using both perception functions can be combined to beter understand the surrounding environment.
  • the processor 6810 compares and fuses the independent outputs from the perception components 6802 and 6803 to produce a unionized model of the operating environment 6814.
  • each perception output from a perception component is associated with a confidence score indicating the probability that the output is accurate.
  • the perception component generates a confidence score based on factors that can affect the accuracy of the associated data, e.g., data generated during a rainstorm may have a lower confidence score than data generated during clear weather.
  • the degree of unionization is based on the confidence scores and the desired level of caution for the unionization. For example, if false positives are much preferred to false negatives, a detected object with a low confidence score will still be added to a detected free space with a high confidence score.
  • the perception component 6802 can use one or more LiDARs or cameras, e.g., mono or stereo cameras, to detect free space in the operating environment 6801.
  • a LiDAR can directly output 3D object maps, but has limited operating range relative to other techniques and may encounter performance degradation in unfavorable weather conditions.
  • a mono or stereo camera can sense different colors, a camera requires illumination for operation and can produce distorted data due to lighting variation.
  • the perception component 6802 can acquire redundant measurements using both types of sensors and fuse the perception data together.
  • the perception component 6802 can use a stereo camera to capture depth data beyond the operating range of a LiDAR.
  • the perception component 6802 can then extend the 3D object map created by the LiDAR by matching spatial structures in the 3D object map with those in the stereo camera output.
  • the perception component can fuse data obtained from LiDARs and mono cameras.
  • Mono cameras typically perceive objects in a two-dimensional image plane which impedes measurement of distance between objects.
  • the outputs from the mono cameras can be first fed to a neural network, e.g., running in the software module 6806.
  • the neural network is trained to detect and estimate a distance between objects from mono camera images.
  • the perception component 6802 combines the distance information produced by the neural network with a 3D object map from the LiDAR.
  • the perception component 6803 can take redundant measurements of the operating environment 6801 using one or more 360° mono cameras and RADARs. For example, an object detected by a RADAR can be overlaid onto a panoramic image output captured by a 360° mono camera.
  • the perception component 6803 uses one or more software algorithms for detecting and tracking objects in the operating environment 6801.
  • the software module 6807 can implement a multi-model object tracker that links objects detected by a category detector, e.g., a neural network classifier, to form an object trajectory.
  • the neural network classifier is trained to classify commonly- seen objects in the operating environment 6801 such as vehicles, pedestrians, road signs, road markings, etc.
  • the object tracker can be a neural network trained to associate objects across a sequence of images. The neural network can use object characteristics such as position, shape, or color to perform the association.
  • the processor 6810 compares the output from the perception component 6802 against the output from the perception component 6803 to detect a failure or failure rate of one of the perception components. For example, each perception component can assign a confidence score to its respective output as different perception functions, e.g., free space detection and object detection and tracking, and produces results with different confidence under different conditions. When an inconsistency appears, the processor 6810 disregards the output from the perception component with the lower confidence score.
  • the vehicle system 6800 has a third perception component implementing a different perception method. In this example, the processor 6810 causes the third perception component to perform a third perception function and rely on the majority result, e.g., based on consistency in output between two of the three perception components.
  • the processor 6810 causes the perception components 6802 and 6803 to provide safety checks on each other.
  • the perception component 6802 is configured to detect free space in the operating environment 6801 using LiDARs
  • the perception component 6803 is configured to detect and track objects using a combination of neural networks and stereo cameras.
  • the processor 6810 can cause the neural networks and the stereo cameras to perform free space detection, and the LiDARS to perform object detection and tracking.
  • FIG. 69 shows an example process 6900 for unionizing perception inputs to create a model of an operation environment, according to an embodiment.
  • the example process 6900 will be described below as performed by a vehicle system, e.g., the vehicle system 6800 of FIG. 68.
  • the vehicle system causes a first component to perform a function (step 6902).
  • the function can be a perception function and the first component can be a hardware perception system including one or more LiDARs, stereo cameras, mono cameras, RADARs, sonars, etc.
  • the first component can be a software program configured to receive and analyze data outputs from a hardware sensor.
  • the software program is a neural network trained to detect and track objects in image data or object maps.
  • the vehicle system concurrently causes a second component to perform the same function as the first component (step 6904).
  • the second component can be a hardware perception system or software program similar to the first component to perform a perception function on the operating environment.
  • the vehicle system After the first and the second components produce respective data outputs, the vehicle system combines and compares the outputs to create a model of the operating environment (steps 6906-6908).
  • the first component can be configured to detect free space in the operating environment while the second component can be configured to detect and track objects in the operating environment.
  • the vehicle systems can compare the outputs from the first and the second components by matching their respective spatial features, and create a unionized model of the operating environment.
  • the unionized model can be a more accurate representation of the operating environment compared to the output by the first or the second component alone.
  • the vehicle system After obtaining a unionized model of the operating environment, the vehicle system initiates an operation based on the characteristics of the model (step 6910). For example, the vehicle system can adjust vehicle speed and trajectory to avoid obstacles present in the model of the operating environment.
  • Item 1 A system comprising:
  • each operations subsystem of the two or more different autonomous vehicle operations subsystems comprises:
  • a solution proposer configured to propose solutions for autonomous vehicle operation based on current input data
  • a solution scorer configured to evaluate the proposed solutions for autonomous vehicle operation based on one or more cost assessments
  • solution scorer of at least one of the two or more different autonomous vehicle operations subsystems is configured to evaluate both the proposed solutions from the solution proposer of the at least one of the two or more different autonomous vehicle operations subsystems and at least one of the proposed solutions from the solution proposer of at least one other of the two or more different autonomous vehicle operations subsystems;
  • an output mediator coupled with the two or more different autonomous vehicle operations subsystems and configured to manage autonomous vehicle operation outputs from the two or more different autonomous vehicle operations subsystems.
  • Item 2 The system of item 1, wherein the two or more different autonomous vehicle operations subsystems are included in a perception stage of autonomous vehicle operation.
  • Item 3 The system of any preceding item, wherein the two or more different autonomous vehicle operations subsystems are included in a localization stage of autonomous vehicle operation.
  • Item 4 The system of any preceding item, wherein the two or more different autonomous vehicle operations subsystems are included in a planning stage of autonomous vehicle operation.
  • Item 5 The system of any preceding item, wherein the two or more different autonomous vehicle operations subsystems are included in a control stage of autonomous vehicle operation.
  • Item 6 The system of any preceding item, wherein the solution scorer of the at least one of the two or more different autonomous vehicle operations subsystems is configured to (i) determine a preferred one of the proposed solutions from two or more of the solution proposers of the at least one of the two or more different autonomous vehicle operations subsystems, and a preferred one of the alternative solutions from at least another one of the two or more different autonomous vehicle operations subsystems, (ii) compare the preferred solution with the preferred alternative solution, and (iii) select between the preferred solution and the preferred alternative solution based on the comparison.
  • Item 7 The system of any preceding item, wherein the solution scorer of the at least one of the two or more different autonomous vehicle operations subsystems is configured to compare and select between the proposed solution and the alternative solution based on a cost assessment that favors continuity with one or more prior solutions selected for operation of the autonomous vehicle.
  • Item 8 The system of any preceding item, wherein the solution scorer of the at least one of the two or more different autonomous vehicle operations subsystems is configured to compare the proposed solutions with more than one alternative solutions received from others of the two or more different autonomous vehicle operations subsystems, and select among the proposed solutions and the alternative solutions.
  • Item 9 The system of any of items 1-8, wherein the at least one other of the two or more different autonomous vehicle operations subsystems is configured to provide additional autonomous vehicle operations solutions that are not redundant with the autonomous vehicle operations solutions of the at least one of the two or more different autonomous vehicle operations subsystems.
  • Item 10 The system of any of items 1-8, wherein the at least one other of the two or more different autonomous vehicle operations subsystems is configured to only provide autonomous vehicle operations solutions that are redundant with the autonomous vehicle operations solutions of the at least one of the two or more different autonomous vehicle operations subsystems.
  • each of the two or more different autonomous vehicle operations subsystems comprises a pipeline of operational stages, each stage in the pipeline comprises at least one solution scorer configured to evaluate proposed solutions from at least one solution proposer in the stage, and at least one solution scorer from each pipeline is configured to evaluate a proposed solution from another pipeline.
  • each pipelines of operational stages comprise:
  • a first stage solution scorer of the first pipeline, configured to evaluate solutions from the first stage first pipeline solution proposer;
  • a second stage solution scorer of the first pipeline, configured to evaluate solutions from the second stage first pipeline solution proposer;
  • a first stage solution scorer of the second pipeline, configured to evaluate solutions from the first stage second pipeline solution proposer
  • a second stage solution scorer of the second pipeline, configured to evaluate solutions from the second stage second pipeline solution proposer;
  • first stage first pipeline solution scorer is configured to evaluate a solution from the first stage second pipeline solution proposer
  • first stage second pipeline solution scorer is configured to evaluate a solution from the first stage first pipeline solution proposer
  • the second stage first pipeline solution scorer is configured to evaluate a solution from the second stage second pipeline solution proposer
  • the second stage second pipeline solution scorer is configured to evaluate a solution from the second stage first pipeline solution proposer.
  • Item 13 The system of item 12, wherein components of the second pipeline including the first stage solution proposer, the first stage solution scorer, the second stage solution proposer, and the second stage solution scorer share a power supply.
  • Item 14 The system of item 12, wherein the first stage comprises a perception stage configured to determine a perceived current state of autonomous vehicle operation based on the current input data, and the second stage comprises a planning stage configured to determine a plan for autonomous vehicle operation based on output from the first stage.
  • Item 15 The system of item 14, wherein the first stage first pipeline solution proposer implements a perception generation mechanism comprising at least one of bottom- up perception (object detection), top-down task-driven attention, priors, or occupancy grids, and wherein the first stage first pipeline solution scorer implements a perception evaluation mechanism comprising at least one of computation of likelihood from sensor models.
  • Item 16 The system of item 12, wherein the first stage comprises a planning stage configured to determine a plan for autonomous vehicle operation based on the current input data, and the second stage comprises a control stage configured to determine a control signal for autonomous vehicle operation based on output from the first stage.
  • Item 17 The system of item 16, wherein the first stage first pipeline solution proposer implements a planning generation mechanism comprising at least one of random sampling, MPC, deep learning, or pre-defined primitives, and wherein the first stage first pipeline solution scorer implements a planning evaluation mechanism comprising at least one of trajectory scoring based on trajectory length, safety, or comfort.
  • Item 18 The system of item 12, wherein the first stage comprises a localization stage configured to determine a current position of an autonomous vehicle based on the current input data, and the second stage comprises a control stage configured to determine a control signal for autonomous vehicle operation based on output from the first stage.
  • Item 19 The system of item 12, wherein the pipelines of operational stages comprise:
  • a third stage solution scorer of the first pipeline, configured to evaluate solutions from the third stage first pipeline solution proposer;
  • a third stage solution scorer of the second pipeline, configured to evaluate solutions from the third stage second pipeline solution proposer;
  • the third stage first pipeline solution scorer is configured to evaluate a solution from the third stage second pipeline solution proposer
  • the third stage second pipeline solution scorer is configured to evaluate a solution from the third stage first pipeline solution proposer.
  • Item 20 A method of operating an autonomous vehicle using the system of any of items 1-19
  • Item 21 A non-transitory computer-readable medium encoding instructions operable to cause data processing apparatus to operate an autonomous vehicle using the system of any of items 1-19.
  • Item 22 A method for operating, within an autonomous vehicle (AV) system of an AV, two or more redundant pipelines coupled with an output mediator, a first pipeline of the two or more redundant pipelines comprising a first perception module, a first localization module, a first planning module, and a first control module, and a second pipeline of the two or more redundant pipelines comprising a second perception module, a second localization module, a second planning module, and a second control module, wherein each of the first and second controller modules are connected with an output mediator, the method comprising:
  • [00600] generating, by the second planning module, a second route proposal based on the second AV position; [00601] selecting, by the first planning module, one between the first route proposal and the second route proposal based on a first planning-cost function, and providing, by the first planning module, the selected one as a first route to the first control module;
  • the first sensor signals received from the first set of sensors comprise one or more lists of objects detected by corresponding sensors of the first set, and
  • the second sensor signals received from the second set of sensors comprise one or more lists of objects detected by corresponding sensors of the first set.
  • Item 24 The method of item 22, wherein
  • the generating of the first world view proposal comprises creating one or more first lists of objects detected by corresponding sensors of the first set, and
  • the generating of the second world view proposal comprises creating one or more lists of objects detected by corresponding sensors of the second set.
  • Item 25 The method of any one of items 22 to 24, wherein
  • the generating of the first world view proposal is performed based on a first perception proposal mechanism
  • the generating of the second world view proposal is performed based on a second perception proposal mechanism different from the first perception proposal mechanism.
  • Item 26 The method of any one of items 22 to 25, wherein
  • the first world view provided at least to the first localization module comprises a first object track of one or more objects detected by the first set of sensors, and
  • the second world view provided at least to the second localization module comprises a second object track of one or more objects detected by the second set of sensors.
  • Item 27 The method of any one of items 22 to 26, wherein the first set of sensors is different from the second set of sensors.
  • Item 29 The method of item 28, wherein the generating of the first and second AV position proposals uses one or more localization algorithms including map-based localization, LiDAR map-based localization, RADAR map-based localization, visual map- based localization, visual odometry, and feature-based localization.
  • Item 30 The method of any one of items 22 and 27-28, wherein
  • the generating of the first AV position proposal is performed based on a first localization algorithm
  • the generating of the second AV position proposal is performed based on a second localization algorithm different from the first localization algorithm.
  • Item 31 The method of any one of items 22 and 28 to 30, wherein
  • the first AV position provided at least to the first planning module comprises a first estimate of a current position of the AV
  • the second AV position provided at least to the second planning module comprises a second estimate of a current position of the AV.
  • Item 32 The method of item 22, further comprising [00632] receiving, by the first planning module, the first world view from the first perception module, wherein the generating of the first route proposal is further based on the first world view, and
  • Item 33 The method of item 22 or 32, wherein
  • the generating of the first route proposal is performed based on a first planning algorithm
  • the generating of the second route proposal is performed based on a second planning algorithm different from the first planning algorithm.
  • Item 34 The method of any one of items 22 and 32-33, wherein the generating of the first and second route proposals comprises proposing respective paths between the AV’s current position and a destination of the AV.
  • Item 35 The method of any one of items 22 and 32 to 34, wherein the generating of the first and second route proposals comprises inferring behavior of the AV and one or more other vehicles.
  • Item 36 The method of item 35, wherein the behavior is inferred by comparing a list of detected objects with driving rules associated with a current location of the AV.
  • Item 37 The method of item 35, wherein the behavior is inferred by comparing a list of detected objects with locations in which vehicles are permitted to operate by driving rules associated with a current location of the vehicle.
  • Item 38 The method of item 35, wherein the behavior is inferred through a constant velocity or constant acceleration model for each detected object.
  • Item 39 The method of item 35, wherein the generating of the first and second route proposals comprises proposing respective paths that conform to the inferred behavior and avoids one or more detected objects.
  • Item 40 The method of item 32, wherein the selecting of the first and second route proposals comprises evaluating collision likelihood based on the respective world view and a behavior inference model.
  • Item 42 The method of item 22 or 41, wherein
  • the generating of the first control-signal proposal is performed based on a first control algorithm
  • the generating of the second control-signal proposal is performed based on a second control algorithm.
  • Item 43 A system comprising:
  • an output mediator coupled with the two or more different autonomous vehicle operations subsystems and configured to manage autonomous vehicle operation outputs from the two or more different autonomous vehicle operations subsystems;
  • the output mediator is configured to selectively promote different ones of the two or more different autonomous vehicle operations subsystems to a prioritized status based on current input data compared with historical performance data for the two or more different autonomous vehicle operations subsystems.
  • Item 44 The system of item 43, wherein the two or more different autonomous vehicle operations subsystems are included in a perception stage of autonomous vehicle operation.
  • Item 45 The system of any preceding item, wherein the two or more different autonomous vehicle operations subsystems are included in a localization stage of autonomous vehicle operation.
  • Item 46 The system of any preceding item, wherein the two or more different autonomous vehicle operations subsystems are included in a planning stage of autonomous vehicle operation.
  • Item 47 The system of any preceding item, wherein the two or more different autonomous vehicle operations subsystems are included in a control stage of autonomous vehicle operation.
  • Item 48 The system of any of items 43-47, wherein a first of the different ones of the two or more different autonomous vehicle operations subsystems is configured to provide additional autonomous vehicle operations decisions that are not redundant with autonomous vehicle operations decisions of a second of the different ones of the two or more different autonomous vehicle operations subsystems.
  • Item 49 The system of any of items 43-47, wherein a first of the different ones of the two or more different autonomous vehicle operations subsystems is configured to only provide autonomous vehicle operations decisions that are redundant with autonomous vehicle operations decisions of a second of the different ones of the two or more different autonomous vehicle operations subsystems.
  • Item 50 The system of any of items 43-47, wherein the output mediator is configured to promote an autonomous vehicle operations subsystem to the prioritized status only once the historical performance data shows a substantially better performance in a specific operational context.
  • Item 51 The system of any of items 43-50, wherein the output mediator is configured to promote an autonomous vehicle operations subsystem to the prioritized status based on results from a machine learning algorithm that operates on the historical performance data to determine one or more specific operational contexts for the autonomous vehicle in which one of the two or more different autonomous vehicle operations subsystems performs differently than remaining ones of the two or more different autonomous vehicle operations subsystems.
  • Item 52 The system of item 51 , wherein the machine learning algorithm operates on historical performance data relating to use of the two or more different autonomous vehicle operations subsystems in different autonomous vehicles in a fleet of autonomous vehicles.
  • Item 53 The system of items 43, 51 or 52, wherein the output mediator is configured to selectively promote the different ones of the two or more different autonomous vehicle operations subsystems to the prioritized status based on the current input data indicating a current operational context is either city streets or highway driving conditions, and based on the historical performance data indicating that the different ones of the two or more different autonomous vehicle operations subsystems perform differently in the current operational context than remaining ones of the two or more different autonomous vehicle operations subsystems.
  • Item 54 The system of items 43, 51 or 52, wherein the output mediator is configured to selectively promote the different ones of the two or more different autonomous vehicle operations subsystems to the prioritized status based on the current input data indicating a current operational context involves specific weather conditions, and based on the historical performance data indicating that the different ones of the two or more different autonomous vehicle operations subsystems perform differently in the current operational context than remaining ones of the two or more different autonomous vehicle operations subsystems.
  • Item 55 The system of items 43, 51 or 52, wherein the output mediator is configured to selectively promote the different ones of the two or more different autonomous vehicle operations subsystems to the prioritized status based on the current input data indicating a current operational context involves specific traffic conditions, and based on the historical performance data indicating that the different ones of the two or more different autonomous vehicle operations subsystems perform differently in the current operational context than remaining ones of the two or more different autonomous vehicle operations subsystems.
  • Item 56 The system of items 43, 51 or 52, wherein the output mediator is configured to selectively promote the different ones of the two or more different autonomous vehicle operations subsystems to the prioritized status based on the current input data indicating a current operational context is during a particular time of day, and based on the historical performance data indicating that the different ones of the two or more different autonomous vehicle operations subsystems perform differently in the current operational context than remaining ones of the two or more different autonomous vehicle operations subsystems.
  • Item 57 The system of items 43, 51 or 52, wherein the output mediator is configured to selectively promote the different ones of the two or more different autonomous vehicle operations subsystems to the prioritized status based on the current input data indicating a current operational context involves specific speed ranges, and based on the historical performance data indicating that the different ones of the two or more different autonomous vehicle operations subsystems perform differently in the current operational context than remaining ones of the two or more different autonomous vehicle operations subsystems.
  • Item 58 The system of any of items 43-57, wherein each of the two or more different autonomous vehicle operations subsystems implement both perception and planning functionality for autonomous vehicle operation.
  • Item 59 The system of any of items 43-57, wherein each of the two or more different autonomous vehicle operations subsystems implement both perception and control functionality for autonomous vehicle operation.
  • Item 60 A method of operating an autonomous vehicle using the system of any of items 43-59.
  • Item 61 A non-transitory computer-readable medium encoding instructions operable to cause data processing apparatus to operate an autonomous vehicle using the system of any of items 43-59.
  • Item 62 A method performed by an output mediator for controlling output of two or more different autonomous vehicle operations subsystems of an autonomous vehicle, one of which having prioritized status, the method comprising:
  • Item 63 The method of item 62, wherein controlling issuance of an output from the autonomous vehicle operations subsystem having the prioritized status comprises instructing the autonomous vehicle operations subsystem having the prioritized status to transmit its output to a component of the autonomous vehicle which is disposed down-stream from the output mediator and uses the transmitted output for operating the autonomous vehicle.
  • Item 64 The method of item 62, wherein controlling issuance of an output from the autonomous vehicle operations subsystem having the prioritized status comprises transmitting the output of the autonomous vehicle operations subsystem having the prioritized status to a component of the autonomous vehicle which is disposed down-stream from the output mediator and uses the transmitted output for operating the autonomous vehicle.
  • Item 65 The method of any one of items 62-64, wherein the promoting is performed in response to determining that the autonomous vehicle operations subsystem corresponding to the current operational context lacks prioritized status.
  • Item 66 The method of any one of items 62-64, further comprising
  • Item 67 The method of any one of items 62-64, further comprising
  • Item 68 The method of any one of items 62-65, wherein prior to promoting one of the autonomous vehicle operations subsystems which corresponds to the current operational context to prioritized status, the method further comprises
  • Item 69 The method of item 68, wherein determining the current operational context based on the current input data is performed by using an input data / context look-up- table.
  • Item 70 The method of item 69, wherein input data referenced by the input data / context look-up-table comprises one or more of traffic data, map data, AV location data, time-of-day data, speed data or weather data.
  • Item 71 The method of item 68, wherein identifying the autonomous vehicle operations subsystem corresponding to the current operational context is performed by using context / subsystem look-up-table.
  • Item 72 The method of any one of items 62-71, wherein
  • the two or more autonomous vehicle operations subsystems are a plurality of perception modules and their outputs are respective world views, and
  • the method comprises controlling issuance, to a planning module disposed down-stream from the output mediator, of the world view provided by the perception module having prioritized status.
  • Item 73 The method of any one of items 62-71, wherein
  • the two or more autonomous vehicle operations subsystems are a plurality of planning modules and their outputs are respective routes, and
  • the method comprises controlling issuance, to a control module disposed down-stream from the output mediator, of the route provided by the planning module having prioritized status.
  • Item 74 The method of any one of items 62-71, wherein
  • the two or more autonomous vehicle operations subsystems are a plurality of localization modules and their outputs are respective AV positions, and
  • the method comprises controlling issuance, to a control module disposed down-stream from the output mediator, of the AV position provided by the localization module having prioritized status.
  • Item 75 The method of any one of items 62-71, wherein
  • the two or more autonomous vehicle operations subsystems are a plurality of control modules and their outputs are respective control signals
  • the method comprises controlling issuance, to an actuator disposed down stream from the output mediator, of the control signal provided by the control module having prioritized status.
  • Item 76 An autonomous vehicle, comprising:
  • a first control system configured to, in accordance with at least one input, provide output that affects a control operation of the autonomous vehicle while the autonomous vehicle is in an autonomous driving mode and while the first control system is selected;
  • a second control system configured to, in accordance with at least one input, provide output that affects the control operation of the autonomous vehicle while the autonomous vehicle is in the autonomous driving mode and while the second control system is selected;
  • At least one processor configured to select at least one of the first control system and the second control system to affect the control operation of the autonomous vehicle.
  • Item 77 The autonomous vehicle of item 76, wherein the at least one processor is configured to select at least one of the first control system and the second control system in accordance with performance of the first control system and the second control system over a period of time.
  • Item 78 The autonomous vehicle of any of items 76-77, wherein the at least one processor is configured for identifying a failure of at least one of the first control system and the second control system.
  • Item 79 The autonomous vehicle of any of items 76-78, wherein the at least one processor is configured for selecting the second control system in accordance with identifying a failure of the first control system.
  • Item 80 The autonomous vehicle of any of items 76-79, wherein the at least one processor is configured for
  • Item 81 The autonomous vehicle of any of items 76-80, wherein the first control system is configured for receiving feedback from a first feedback system and the second control system is configured for receiving feedback from a second feedback system.
  • Item 82 The autonomous vehicle of item 81, wherein the at least one processor is configured to compare the feedback from the first feedback system and the second feedback system to identify a failure of at least one of the first control system and the second control system.
  • Item 83 The autonomous vehicle of any of items 76-82, wherein the first control system operates in accordance with a first input, and the second control system operates in accordance with a second input.
  • Item 84 The autonomous vehicle of any of items 76-82, wherein the first control system operates in accordance with a first input, and the second control system operates in accordance with the first input.
  • Item 85 The autonomous vehicle of item 76-84, wherein the first control system is configured to use a first algorithm when affecting the control operation and the second control system is configured to use a second algorithm when affecting the control operation.
  • Item 86 The autonomous vehicle of item 85, wherein the first algorithm and the second algorithm are control feedback algorithms.
  • Item 87 The autonomous vehicle of any of items 85-86, wherein the first algorithm adjusts steering angle, and the second algorithm adjusts throttle control.
  • Item 88 The autonomous vehicle of any of items 76-86, wherein the first control system is configured to use a steering mechanism to affect steering and the second control system is configured to use functionality other than the steering mechanism to affect steering.
  • Item 89 The autonomous vehicle of item 88, wherein the functionality other than the steering mechanism includes at least one of direct control of the autonomous vehicle’s wheels, and direct control of the autonomous vehicle’s axels.
  • Item 90 The autonomous vehicle of any of items 76-86, wherein the first control system is configured to use a throttle control mechanism to affect acceleration and the second control system is configured to use functionality other than the throttle control mechanism to affect acceleration.
  • Item 91 The autonomous vehicle of item 90, wherein the functionality other than the throttle control mechanism includes at least one of direct control of the autonomous vehicle’s engine and the direct control of the autonomous vehicle’s fuel system.
  • Item 92 The autonomous vehicle of any of items 76-91, wherein the control operation controls at least one of the speed of the autonomous vehicle and the orientation of the autonomous vehicle.
  • Item 93 The autonomous vehicle of any of items 76-92, wherein the control operation controls at least one of the speed smoothness of the autonomous vehicle and the orientation smoothness of the autonomous vehicle.
  • Item 94 The autonomous vehicle of any of items 76-93, wherein the control operation controls at least one of the acceleration, jerk, jounce, snap, and crackle of the autonomous vehicle.
  • Item 95 The autonomous vehicle of any of items 76-94, wherein the at least one processor includes at least one of an arbiter module and a diagnostics module.
  • Item 96 An autonomous vehicle, comprising:
  • a first sensor configured to produce a first sensor data stream from one or more environmental inputs external to the autonomous vehicle while the autonomous vehicle is in an operational driving state
  • a second sensor configured to produce a second sensor data stream from the one or more environmental inputs external to the autonomous vehicle while the autonomous vehicle is in the operational driving state, the first sensor and the second sensor being configured to detect a same type of information; and [00732] a processor coupled with the first sensor and the second sensor, wherein the processor is configured to detect an abnormal condition based on a difference between the first sensor data stream and the second sensor data stream, and wherein the processor is configured to switch among the first sensor, the second sensor, or both as an input to control the autonomous vehicle in response to a detection of the abnormal condition.
  • Item 97 The autonomous vehicle of item 96, wherein the processor is configured to capture a first set of data values within the first sensor data stream over a sampling time window, wherein the processor is configured to capture a second set of data values within the second sensor data stream over the sampling time window, and wherein the processor is configured to detect the abnormal condition by determining a deviation between the first set of data values and the second set of data values.
  • Item 98 The autonomous vehicle of item 97, wherein the processor is configured to control a duration of the sampling time window responsive to a driving condition.
  • Item 99 The autonomous vehicle of item 97, wherein a duration of the sampling time window is predetermined.
  • Item 100 The autonomous vehicle of one of items 96 - 99, wherein the processor is configured to determine the difference based on a first sample of the first sensor data stream and a second sample of the second sensor data stream, the first sample and the second sample corresponding to a same time index.
  • Item 101 The autonomous vehicle of item 100, wherein the processor is configured to detect the abnormal condition based on the difference exceeding a
  • Item 102 The autonomous vehicle of one of items 96 - 101, wherein the processor is configured to determine the difference based on a detection of a missing sample within the first sensor data stream.
  • Item 103 The autonomous vehicle of one of items 96 - 102, wherein the first sensor and the second sensor use one or more different sensor characteristics to detect the same type of information.
  • Item 104 The autonomous vehicle of item 103, wherein the first sensor is associated with the abnormal condition, and wherein the processor, in response to the detection of the abnormal condition, is configured to perform a transformation of the second sensor data stream to produce a replacement version of the first sensor data stream.
  • Item 105 The autonomous vehicle of one of items 96 - 102, wherein the second sensor is a redundant version of the first sensor.
  • Item 106 The autonomous vehicle of one of items 96 - 105, wherein the processor, in response to the detection of the abnormal condition, is configured to perform a diagnostic routine on the first sensor, the second sensor, or both to resolve the abnormal condition.
  • Item 107 A method of operating an autonomous vehicle, comprising:
  • Item 108 The method of item 107, comprising:
  • detecting the abnormal condition comprises determining a deviation between the first set of data values and the second set of data values.
  • Item 109 The method of item 108, comprising:
  • Item 110 The method of item 108, wherein a duration of the sampling time window is predetermined.
  • Item 111 The method of one of items 107-110, wherein the difference is based on a first sample of the first sensor data stream and a second sample of the second sensor data stream, the first sample and the second sample corresponding to a same time index.
  • Item 113 The method of one of items 107 - 112, wherein the difference is based on a detection of a missing sample within the first sensor data stream.
  • Item 114 The method of one of items 107 - 113, wherein the first sensor and the second sensor use one or more different sensor characteristics to detect the same type of information.
  • Item 115 The method of item 114, comprising:
  • Item 116 The method of one of items 107 - 113, wherein the second sensor is a redundant version of the first sensor.
  • Item 117 The method of one of items 107 - 116, comprising:
  • Item 118 An autonomous vehicle, comprising:
  • a control system configured to affect a control operation of the autonomous vehicle
  • control processor in communication with the control system, the control processor configured to determine instructions for execution by the control system
  • a telecommunications system in communication with the control system, the telecommunications system configured to receive instructions from an external source;
  • control processor is configured to determine instructions that are executable by the control system from the instructions received from the external source and is configured to enable the external source in communication with the telecommunications system to control the control system when one or more specified conditions are detected.
  • Item 119 The autonomous vehicle of item 118, wherein the control processor is configured to determine if data received from one or more sensors on the autonomous vehicle meets the one or more specified conditions, and in accordance with the determination enable the telecommunications system to control the control system.
  • Item 120 The autonomous vehicle of item 118, wherein the one or more specified conditions detected by the control processor includes an emergency condition.
  • Item 121 The autonomous vehicle of item 118, wherein the control processor detects the one or more specified conditions from input received from an occupant of the autonomous vehicle.
  • Item 122 The autonomous vehicle of item 121, wherein the input is received from a notification interface within an interior of the autonomous vehicle.
  • Item 123 The autonomous vehicle of item 118, wherein the one or more specified conditions include environmental conditions.
  • Item 124 The autonomous vehicle of item 118, wherein the one or more specified conditions include a failure of the control processor.
  • Item 125 The autonomous vehicle of item 118, wherein the control processor is configured to determine if the autonomous vehicle is on a previously untraveled road as one of the specified conditions, and in accordance with the determination enable the
  • Item 126 The autonomous vehicle of item 125, wherein the determination that the autonomous vehicle is on a previously untraveled road is made using data from a database of traveled roads.
  • Item 127 The autonomous vehicle of item 118, wherein the telecommunications system receives instructions based on inputs made by a teleoperator.
  • An autonomous vehicle comprising:
  • a control system configured to affect a first control operation of the autonomous vehicle
  • control processor in communication with the control system, the control processor configured to determine instructions for execution by the control system
  • a telecommunications system in communication with the control system, the telecommunications system configured to receive instructions from an external source; and [00782] a processor configured to determine instructions that are executable by the control system from the instructions received from the external source and to enable the control processor or the external source in communication with the telecommunications system to operate the control system.
  • Item 129 The autonomous vehicle of item 128, wherein the control processor is configured to enable the telecommunications system to operate the control system when one or more specified conditions are detected.
  • Item 130 The autonomous vehicle of item 129, wherein the one or more specified conditions detected by the control processor includes an emergency condition.
  • Item 131 The autonomous vehicle of item 129, wherein the control processor detects the one or more specified conditions from input received from an occupant of the autonomous vehicle.
  • Item 132 The autonomous vehicle of item 131, wherein the input is received from a notification interface within an interior of the autonomous vehicle.
  • Item 134 The autonomous vehicle of item 129, wherein the one or more specified conditions include a failure of the control processor.
  • Item 135. The autonomous vehicle of item 129, wherein the control processor is configured to determine if the autonomous vehicle is on a previously untraveled road as one of the specified conditions, and in accordance with the determination enable the

Abstract

L'invention concerne, entre autres, des techniques de redondance dans des véhicules autonomes. Par exemple, un véhicule autonome peut comprendre au moins deux sous-systèmes de fonctionnements de véhicule autonome redondants.
PCT/US2019/058949 2018-10-30 2019-10-30 Redondance dans des véhicules autonomes WO2020092635A1 (fr)

Priority Applications (7)

Application Number Priority Date Filing Date Title
GB2017386.0A GB2587275B (en) 2018-10-30 2019-10-30 Redundancy in autonomous vehicles
KR1020237005763A KR20230030029A (ko) 2018-10-30 2019-10-30 자율 주행 차량에서의 리던던시
KR1020207034459A KR20210006926A (ko) 2018-10-30 2019-10-30 자율 주행 차량에서의 리던던시
CN201980072734.1A CN112969622A (zh) 2018-10-30 2019-10-30 自主运载工具中的冗余
US17/058,242 US20210163021A1 (en) 2018-10-30 2019-10-30 Redundancy in autonomous vehicles
DE112019005425.2T DE112019005425T5 (de) 2018-10-30 2019-10-30 Redundanz in autonomen fahrzeugen
DKPA202070218A DK202070218A1 (en) 2018-10-30 2020-04-08 Redundancy in autonomous vehicles

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201862752447P 2018-10-30 2018-10-30
US62/752,447 2018-10-30

Publications (1)

Publication Number Publication Date
WO2020092635A1 true WO2020092635A1 (fr) 2020-05-07

Family

ID=70464217

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2019/058949 WO2020092635A1 (fr) 2018-10-30 2019-10-30 Redondance dans des véhicules autonomes

Country Status (7)

Country Link
US (1) US20210163021A1 (fr)
KR (2) KR20230030029A (fr)
CN (1) CN112969622A (fr)
DE (1) DE112019005425T5 (fr)
DK (1) DK202070218A1 (fr)
GB (5) GB2613509B (fr)
WO (1) WO2020092635A1 (fr)

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111762179A (zh) * 2020-05-11 2020-10-13 广州文远知行科技有限公司 车辆控制方法、装置、车辆和计算机可读存储介质
CN112347906A (zh) * 2020-11-04 2021-02-09 北方工业大学 公交车内异常聚集行为的检测方法
CN112434564A (zh) * 2020-11-04 2021-03-02 北方工业大学 公交车内异常聚集行为的检测系统
WO2021228536A1 (fr) * 2020-05-15 2021-11-18 Robert Bosch Gmbh Procédé de localisation d'un véhicule par rapport à un modèle d'environnement autour d'une trajectoire de déplacement
WO2021233552A1 (fr) * 2020-05-22 2021-11-25 Tsu Gmbh Gesellschaft Für Technik, Sicherheit Und Umweltschutz Mbh Logique de commande redondante pour systèmes d'automatisation critiques en termes de sécurité sur la base de réseaux de neurones artificiels
US20220075388A1 (en) * 2018-07-16 2022-03-10 Phantom Auto Inc. Normalization of intelligent transport system handling characteristics
WO2022063663A1 (fr) * 2020-09-24 2022-03-31 Robert Bosch Gmbh Procédé, module de traitement de données et réseau de traitement de données pour traiter des données
WO2022076058A1 (fr) * 2020-10-05 2022-04-14 Qualcomm Incorporated Procédé et unité de commande pour gérer une anomalie de condition de conduite
WO2022126085A1 (fr) * 2020-12-09 2022-06-16 Baker Hughes Holdings Llc Déclenchement d'appareils photo et photogrammétrie à multiples appareils photo
CN115610346A (zh) * 2022-09-29 2023-01-17 成都赛力斯科技有限公司 汽车风险控制方法、汽车、计算机设备和存储介质
US20230029093A1 (en) * 2021-07-20 2023-01-26 Nissan North America, Inc. Computing Framework for Vehicle Decision Making and Traffic Management
US11644835B2 (en) * 2020-07-29 2023-05-09 Toyota Research Institute, Inc. Game-theoretic planning for risk-aware interactive agents
EP4198573A1 (fr) * 2021-12-14 2023-06-21 Tusimple, Inc. Système et procédé de détection de précipitations pour un véhicule autonome
US20230266139A1 (en) * 2022-02-21 2023-08-24 Motional Ad Llc Passenger preference route and alternative destination estimator
CN115610346B (zh) * 2022-09-29 2024-04-12 重庆赛力斯凤凰智创科技有限公司 汽车风险控制方法、汽车、计算机设备和存储介质

Families Citing this family (62)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11466998B1 (en) 2019-02-15 2022-10-11 State Farm Mutual Automobile Insurance Company Systems and methods for dynamically generating optimal routes for management of multiple vehicles
US11466997B1 (en) 2019-02-15 2022-10-11 State Fram Mutual Automobile Insurance Company Systems and methods for dynamically generating optimal routes for vehicle operation management
US11560153B2 (en) * 2019-03-07 2023-01-24 6 River Systems, Llc Systems and methods for collision avoidance by autonomous vehicles
DE102019107443A1 (de) * 2019-03-22 2020-09-24 Robert Bosch Gmbh Verfahren und Vorrichtung zum Betrieb eines Roboters mit verbesserter Objektdetektion
DE112019007111T5 (de) * 2019-03-29 2022-01-27 Honda Motor Co., Ltd. Steuervorrichtung, Steuerverfahren und Programm
EP3966652A1 (fr) * 2019-05-07 2022-03-16 Kontrol GmbH Vérification formelle pour le développement et l'application en temps réel de systèmes autonomes
AT522167B1 (de) * 2019-06-13 2020-09-15 Avl List Gmbh Verfahren und Vorrichtung zur vorausschauenden Fahrzeugkontrolle
CN116009909A (zh) * 2019-06-21 2023-04-25 华为技术有限公司 软件升级方法、装置及系统
US11549815B2 (en) * 2019-06-28 2023-01-10 GM Cruise Holdings LLC. Map change detection
JP2021015565A (ja) * 2019-07-16 2021-02-12 トヨタ自動車株式会社 車両制御装置
US11927955B2 (en) * 2019-07-29 2024-03-12 Waymo Llc Methods for transitioning between autonomous driving modes in large vehicles
US11301700B2 (en) * 2019-08-22 2022-04-12 Wipro Limited System and method for safely parking an autonomous vehicle on sensor anomaly
US11900244B1 (en) * 2019-09-30 2024-02-13 Amazon Technologies, Inc. Attention-based deep reinforcement learning for autonomous agents
US11619942B2 (en) * 2019-10-15 2023-04-04 Robert Bosch Gmbh Controlling an autonomous vehicle when the autonomous vehicle is outside of its operational design domain
US11370419B2 (en) * 2019-11-13 2022-06-28 Robert Bosch Gmbh Use of driver assistance collision mitigation systems with autonomous driving systems
KR20210066984A (ko) * 2019-11-28 2021-06-08 현대자동차주식회사 자율 발렛 주차를 지원하는 시스템 및 방법, 그리고 이를 위한 인프라 및 차량
DE102019218718B4 (de) * 2019-12-02 2023-11-16 Volkswagen Aktiengesellschaft Steuerungssystem zur Steuerung eines Betriebs eines selbstfahrenden Fahrzeugs sowie Kraftfahrzeug
EP4081993A4 (fr) * 2019-12-23 2023-08-30 Nokia Technologies Oy Peloton dynamique virtuel
KR20210095359A (ko) * 2020-01-23 2021-08-02 엘지전자 주식회사 로봇, 로봇의 제어 방법 및 로봇을 제어하기 위한 서버
US20210232913A1 (en) * 2020-01-27 2021-07-29 Honda Motor Co., Ltd. Interpretable autonomous driving system and method thereof
JP7234967B2 (ja) * 2020-02-17 2023-03-08 トヨタ自動車株式会社 衝突回避支援装置
US11661895B2 (en) * 2020-02-24 2023-05-30 General Electric Comapny Autonomous safety mode for distributed control of turbomachines
US11210869B2 (en) * 2020-03-31 2021-12-28 Calpro Adas Solutions, Llc Vehicle safety feature identification and calibration
US11644846B2 (en) * 2020-03-31 2023-05-09 GM Cruise Holdings LLC. System and method for real-time lane validation
US11453409B2 (en) * 2020-04-21 2022-09-27 Baidu Usa Llc Extended model reference adaptive control algorithm for the vehicle actuation time-latency
KR20210138201A (ko) * 2020-05-11 2021-11-19 현대자동차주식회사 자율 주행 제어 방법 및 장치
JP2023533225A (ja) 2020-07-01 2023-08-02 メイ モビリティー，インコーポレイテッド 自律走行車ポリシーを動的にキュレーションする方法及びシステム
US11643082B2 (en) * 2020-08-05 2023-05-09 Toyota Motor Engineering & Manufacturing North America, Inc. Systems and methods for determining real-time lane level snow accumulation
US20230022896A1 (en) * 2020-08-10 2023-01-26 Jun Luo System and method for managing flexible control of vehicles by diverse agents in autonomous driving simulation
US11687094B2 (en) 2020-08-27 2023-06-27 Here Global B.V. Method, apparatus, and computer program product for organizing autonomous vehicles in an autonomous transition region
US11713979B2 (en) * 2020-08-27 2023-08-01 Here Global B.V. Method, apparatus, and computer program product for generating a transition variability index related to autonomous driving
US11691643B2 (en) 2020-08-27 2023-07-04 Here Global B.V. Method and apparatus to improve interaction models and user experience for autonomous driving in transition regions
US11610412B2 (en) * 2020-09-18 2023-03-21 Ford Global Technologies, Llc Vehicle neural network training
WO2022115713A1 (fr) * 2020-11-30 2022-06-02 Nuro, Inc. Systèmes matériels pour un véhicule autonome
US11827243B2 (en) * 2020-12-13 2023-11-28 Pony Ai Inc. Automated vehicle safety response methods and corresponding vehicle safety systems with serial-parallel computing architectures
WO2022132774A1 (fr) 2020-12-14 2022-06-23 May Mobility, Inc. Système et procédé de plateforme de sécurité de véhicule autonome
US20220196830A1 (en) * 2020-12-17 2022-06-23 Aptiv Technologies Limited Vehicle Routing Based on Availability of Radar-Localization Objects
US11738777B2 (en) 2020-12-21 2023-08-29 Zoox, Inc. Dynamic autonomous control engagement
US11912302B2 (en) * 2020-12-21 2024-02-27 Zoox, Inc. Autonomous control engagement
WO2022144959A1 (fr) * 2020-12-28 2022-07-07 本田技研工業株式会社 Appareil de commande de véhicule, système de véhicule, procédé de commande de véhicule et programme
US11708066B2 (en) * 2021-01-21 2023-07-25 Motional Ad Llc Road surface condition guided decision making and prediction
US20220281478A1 (en) * 2021-03-02 2022-09-08 Steering Solutions Ip Holding Corporation Motion monitoring safety diagnostic for the detection of erroneous autonomous motion requests
EP4063222A1 (fr) * 2021-03-24 2022-09-28 Zenseact AB Planification préventive de trajet de véhicule
US20220306119A1 (en) * 2021-03-25 2022-09-29 Ford Global Technologies, Llc Location-based vehicle operation
JP2024512980A (ja) 2021-04-02 2024-03-21 メイ モビリティー，インコーポレイテッド 不完全な環境情報で自律エージェントを動作させる方法及びシステム
JP2022174596A (ja) * 2021-05-11 2022-11-24 トヨタ自動車株式会社 自動運転システム、自動運転制御方法、及び自動運転制御プログラム
US11639180B1 (en) * 2021-06-30 2023-05-02 Gm Cruise Holdings Llc Notifications from an autonomous vehicle to a driver
CN113386796A (zh) * 2021-07-08 2021-09-14 北京三快在线科技有限公司 无人车控制方法、装置、系统、存储介质及电子设备
DE102021208005A1 (de) 2021-07-26 2023-01-26 Robert Bosch Gesellschaft mit beschränkter Haftung Verarbeitung von Satellitendaten zum Erweitern oder Vervollständigen von Messdaten
CN113370721B (zh) * 2021-07-29 2023-06-20 中国人民解放军国防科技大学 三轴无人车应对野外特种任务的控制策略及系统
WO2023028277A1 (fr) * 2021-08-25 2023-03-02 Cyngn, Inc. Système et procédé de calcul de conduite autonome basés sur un système non embarqué
DE102021211257A1 (de) 2021-10-06 2023-04-06 Zf Friedrichshafen Ag Angriffen auf ein künstliches neuronales Netzwerk vorbeugen
CN113885330B (zh) * 2021-10-26 2022-06-17 哈尔滨工业大学 一种基于深度强化学习的信息物理系统安全控制方法
CN113895451B (zh) * 2021-10-27 2023-07-18 东风汽车集团股份有限公司 一种基于自动驾驶系统的安全冗余与故障诊断系统及方法
US11880428B2 (en) 2021-11-12 2024-01-23 Toyota Motor Engineering & Manufacturing North America, Inc. Methods and systems for updating perception models based on geolocation features
US20230182772A1 (en) * 2021-12-14 2023-06-15 Zoox, Inc. Autonomous vehicle operations related to detection of an unsafe passenger pickup/delivery condition
CN114132337B (zh) * 2021-12-31 2024-03-26 阿维塔科技(重庆)有限公司 车辆的故障管理方法、装置以及车辆
WO2023154568A1 (fr) 2022-02-14 2023-08-17 May Mobility, Inc. Procédé et système de fonctionnement conditionnel d'un agent autonome
CN116767264A (zh) * 2022-03-09 2023-09-19 北京图森智途科技有限公司 具有传感器冗余的车辆
US20230350354A1 (en) * 2022-04-28 2023-11-02 Woven By Toyota, Inc. Method of optimizing execution of a function on a control system and apparatus for the same
US11810459B1 (en) 2022-05-09 2023-11-07 Aptiv Technologies Limited Vehicle localization based on radar detections in garages
US20240062478A1 (en) * 2022-08-15 2024-02-22 Middle Chart, LLC Spatial navigation to digital content

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100019964A1 (en) * 2008-07-24 2010-01-28 Gm Global Technology Operations, Inc. Adaptive vehicle control system with driving style recognition and road condition recognition
US20120259464A1 (en) * 2011-04-06 2012-10-11 Fanuc Corporation Robot system having error detection function of robot and control method thereof
US20170023945A1 (en) * 2014-04-04 2017-01-26 Koninklijke Philips N.V. System and methods to support autonomous vehicles via environmental perception and sensor calibration and verification
US20170076227A1 (en) * 2014-03-03 2017-03-16 Inrix Inc., Traffic obstruction detection
US20170113641A1 (en) * 2015-10-26 2017-04-27 Active Knowledge Ltd. Moveable internal shock-absorbing energy dissipation padding in an autonomous vehicle
US20170131712A1 (en) * 2012-03-14 2017-05-11 Autoconnect Holdings Llc Relay and exchange protocol in an automated zone-based vehicular traffic control environment
US20170206464A1 (en) * 2016-01-14 2017-07-20 Preferred Networks, Inc. Time series data adaptation and sensor fusion systems, methods, and apparatus
US20180017968A1 (en) * 2016-07-14 2018-01-18 Baidu Usa Llc Autonomous vehicle human driver takeover mechanism using electrodes

Family Cites Families (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4594714A (en) * 1983-05-02 1986-06-10 United Technologies Corporation Dual-actuator monitor
US6493618B2 (en) * 2000-03-15 2002-12-10 Toyota Jidosha Kabushiki Kaisha Vehicle control using multiple sensors
DE112011102158A5 (de) * 2010-06-28 2013-05-02 Schaeffler Technologies AG & Co. KG Verfahren zur Erkennung der Anwesenheit eines Fahrers in einem Kraftfahrzeug
WO2012050473A1 (fr) * 2010-10-11 2012-04-19 General Electric Company Systèmes, procédés et appareils de détection de concordance dans des canaux individuels entre des signaux de capteurs redondants
US9151786B2 (en) * 2010-10-11 2015-10-06 General Electric Company Systems, methods, and apparatus for detecting shifts in redundant sensor signals
SG10201407100PA (en) * 2014-10-30 2016-05-30 Nec Asia Pacific Pte Ltd System For Monitoring Event Related Data
CN107113479B (zh) * 2015-02-27 2020-04-14 株式会社藤仓 传感器节点和传感器节点的控制方法
US9785145B2 (en) * 2015-08-07 2017-10-10 International Business Machines Corporation Controlling driving modes of self-driving vehicles
EP3979028A1 (fr) * 2015-09-28 2022-04-06 Uatc, Llc Méthode d'exploitation d'un véhicule autonome avec unité de contrôle auxiliaire indépendante
US9630619B1 (en) * 2015-11-04 2017-04-25 Zoox, Inc. Robotic vehicle active safety systems and methods
US9916703B2 (en) * 2015-11-04 2018-03-13 Zoox, Inc. Calibration for autonomous vehicle operation
US9632502B1 (en) * 2015-11-04 2017-04-25 Zoox, Inc. Machine-learning systems and techniques to optimize teleoperation and/or planner decisions
US10598797B2 (en) * 2015-12-29 2020-03-24 Huawei Technologies Co., Ltd. Switching method and portable electronic device
US9883403B2 (en) * 2016-05-15 2018-01-30 Fmr Llc Monitoring presence of authorized user during user session based upon mobile computing device motion
AT519164A3 (de) * 2016-08-16 2018-10-15 Fts Computertechnik Gmbh Fehlertolerantes Verfahren und Vorrichtung zur Steuerung einer autonomen technischen Anlage auf der Basis eines konsolidierten Umweltmodells
DE102017216083B4 (de) * 2016-09-13 2023-08-17 Hl Klemove Corp. Vorrichtung und Verfahren zur Stoßabsorption für ein Fahrzeug
EP3828657A1 (fr) * 2016-12-23 2021-06-02 Mobileye Vision Technologies Ltd. Système de navigation
WO2018170074A1 (fr) * 2017-03-14 2018-09-20 Starsky Robotics, Inc. Système de capteur de véhicule et procédé d'utilisation
US10479376B2 (en) * 2017-03-23 2019-11-19 Uatc, Llc Dynamic sensor selection for self-driving vehicles
US11377108B2 (en) * 2017-04-03 2022-07-05 Motional Ad Llc Processing a request signal regarding operation of an autonomous vehicle
US10526992B2 (en) * 2017-04-05 2020-01-07 GM Global Technology Operations LLC Method and system to detect and mitigate sensor degradation
US10883436B2 (en) * 2017-04-12 2021-01-05 GM Global Technology Operations LLC Method and system to control propulsion systems having sensor or actuator degradation
DE102017206485A1 (de) * 2017-04-18 2018-10-18 Robert Bosch Gmbh Vorrichtung und Verfahren zur Steuerung eines Fahrzeugs
JP6841162B2 (ja) * 2017-05-25 2021-03-10 株式会社デンソー 電子制御装置
JP6848769B2 (ja) * 2017-08-29 2021-03-24 トヨタ自動車株式会社 車載中継装置、情報処理システム、中継装置、情報処理方法、及びプログラム
IL308640A (en) * 2018-03-18 2024-01-01 Driveu Tech Ltd Device, system and method for autonomous driving and remotely controlled vehicles
CN111919089A (zh) * 2018-03-27 2020-11-10 松下知识产权经营株式会社 自动驾驶控制装置、车辆以及需求仲裁系统
US20220194412A1 (en) * 2020-12-18 2022-06-23 Lyft, Inc. Validating Vehicle Sensor Calibration

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100019964A1 (en) * 2008-07-24 2010-01-28 Gm Global Technology Operations, Inc. Adaptive vehicle control system with driving style recognition and road condition recognition
US20120259464A1 (en) * 2011-04-06 2012-10-11 Fanuc Corporation Robot system having error detection function of robot and control method thereof
US20170131712A1 (en) * 2012-03-14 2017-05-11 Autoconnect Holdings Llc Relay and exchange protocol in an automated zone-based vehicular traffic control environment
US20170076227A1 (en) * 2014-03-03 2017-03-16 Inrix Inc., Traffic obstruction detection
US20170023945A1 (en) * 2014-04-04 2017-01-26 Koninklijke Philips N.V. System and methods to support autonomous vehicles via environmental perception and sensor calibration and verification
US20170113641A1 (en) * 2015-10-26 2017-04-27 Active Knowledge Ltd. Moveable internal shock-absorbing energy dissipation padding in an autonomous vehicle
US20170206464A1 (en) * 2016-01-14 2017-07-20 Preferred Networks, Inc. Time series data adaptation and sensor fusion systems, methods, and apparatus
US20180017968A1 (en) * 2016-07-14 2018-01-18 Baidu Usa Llc Autonomous vehicle human driver takeover mechanism using electrodes

Cited By (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220075388A1 (en) * 2018-07-16 2022-03-10 Phantom Auto Inc. Normalization of intelligent transport system handling characteristics
US11675367B2 (en) * 2018-07-16 2023-06-13 Phantom Auto Inc. Normalization of intelligent transport system handling characteristics
CN111762179A (zh) * 2020-05-11 2020-10-13 广州文远知行科技有限公司 车辆控制方法、装置、车辆和计算机可读存储介质
WO2021228536A1 (fr) * 2020-05-15 2021-11-18 Robert Bosch Gmbh Procédé de localisation d'un véhicule par rapport à un modèle d'environnement autour d'une trajectoire de déplacement
WO2021233552A1 (fr) * 2020-05-22 2021-11-25 Tsu Gmbh Gesellschaft Für Technik, Sicherheit Und Umweltschutz Mbh Logique de commande redondante pour systèmes d'automatisation critiques en termes de sécurité sur la base de réseaux de neurones artificiels
US11644835B2 (en) * 2020-07-29 2023-05-09 Toyota Research Institute, Inc. Game-theoretic planning for risk-aware interactive agents
WO2022063663A1 (fr) * 2020-09-24 2022-03-31 Robert Bosch Gmbh Procédé, module de traitement de données et réseau de traitement de données pour traiter des données
US11715370B2 (en) 2020-10-05 2023-08-01 Qualcomm Incorporated Managing a driving condition anomaly
WO2022076058A1 (fr) * 2020-10-05 2022-04-14 Qualcomm Incorporated Procédé et unité de commande pour gérer une anomalie de condition de conduite
US11386776B2 (en) 2020-10-05 2022-07-12 Qualcomm Incorporated Managing a driving condition anomaly
CN112434564B (zh) * 2020-11-04 2023-06-27 北方工业大学 公交车内异常聚集行为的检测系统
CN112434564A (zh) * 2020-11-04 2021-03-02 北方工业大学 公交车内异常聚集行为的检测系统
CN112347906B (zh) * 2020-11-04 2023-06-27 北方工业大学 公交车内异常聚集行为的检测方法
CN112347906A (zh) * 2020-11-04 2021-02-09 北方工业大学 公交车内异常聚集行为的检测方法
WO2022126085A1 (fr) * 2020-12-09 2022-06-16 Baker Hughes Holdings Llc Déclenchement d'appareils photo et photogrammétrie à multiples appareils photo
US20230029093A1 (en) * 2021-07-20 2023-01-26 Nissan North America, Inc. Computing Framework for Vehicle Decision Making and Traffic Management
EP4198573A1 (fr) * 2021-12-14 2023-06-21 Tusimple, Inc. Système et procédé de détection de précipitations pour un véhicule autonome
US20230266139A1 (en) * 2022-02-21 2023-08-24 Motional Ad Llc Passenger preference route and alternative destination estimator
US11959760B2 (en) * 2022-02-21 2024-04-16 Motional Ad Llc Passenger preference route and alternative destination estimator
CN115610346A (zh) * 2022-09-29 2023-01-17 成都赛力斯科技有限公司 汽车风险控制方法、汽车、计算机设备和存储介质
CN115610346B (zh) * 2022-09-29 2024-04-12 重庆赛力斯凤凰智创科技有限公司 汽车风险控制方法、汽车、计算机设备和存储介质

Also Published As

Publication number Publication date
KR20210006926A (ko) 2021-01-19
GB202017386D0 (en) 2020-12-16
GB2613298B (en) 2023-12-20
KR20230030029A (ko) 2023-03-03
GB2610938B (en) 2023-09-06
CN112969622A (zh) 2021-06-15
GB202303553D0 (en) 2023-04-26
GB2613298A (en) 2023-05-31
GB2613740B (en) 2023-12-06
GB2613509B (en) 2023-11-22
GB2587275A (en) 2021-03-24
DE112019005425T5 (de) 2021-07-22
GB2587275B (en) 2022-10-26
GB2613740A (en) 2023-06-14
GB2610938A (en) 2023-03-22
GB202213300D0 (en) 2022-10-26
US20210163021A1 (en) 2021-06-03
GB202303153D0 (en) 2023-04-19
DK202070218A1 (en) 2020-07-13
GB2613509A (en) 2023-06-07
GB202303756D0 (en) 2023-04-26

Similar Documents

Publication Publication Date Title
US20210163021A1 (en) Redundancy in autonomous vehicles
US11548526B2 (en) Systems and methods for implementing an autonomous vehicle response to sensor failure
CN111915917B (zh) 计算机实现的方法、存储介质和运载工具
KR102551208B1 (ko) 차량의 교통 신호등 검출 시스템
CN111121776B (zh) 用于对运载工具进行导航的最佳轨迹的生成
KR20210055617A (ko) 미리 계산되거나 동적으로 생성된 궤적 뱅크로부터의 궤적 예측
US11731653B2 (en) Conditional motion predictions
US20210284161A1 (en) Traffic light estimation
US11568688B2 (en) Simulation of autonomous vehicle to improve safety and reliability of autonomous vehicle
US20220097725A1 (en) Av path planning with calibration information
US20220289198A1 (en) Automated emergency braking system
GB2616739A (en) Traffic light estimation
US11926342B2 (en) Autonomous vehicle post-action explanation system
US20220289199A1 (en) Brake arbitration

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19878622

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 202017386

Country of ref document: GB

Kind code of ref document: A

Free format text: PCT FILING DATE = 20191030

ENP Entry into the national phase

Ref document number: 20207034459

Country of ref document: KR

Kind code of ref document: A

122 Ep: pct application non-entry in european phase

Ref document number: 19878622

Country of ref document: EP

Kind code of ref document: A1