DE102022126537A1 - Coding of priority scenarios for autonomous systems - Google Patents

Abstract

In examples, autonomous vehicles are enabled to negotiate right-of-way scenarios in a safe and predictable manner. In response to detecting a right-of-way scenario, a queuing element data structure is generated that encodes the geometries of an ego path, a competitor path that includes at least one contention point with the ego path, and a contention state associated with the at least one contention point is. The geometry of the context of the right-of-way scenario can also be encoded, e.g. B. the inner surface of an intersection, entry or exit lines, etc. The data structure of the waiting elements is forwarded to a priority planner of the autonomous vehicle. The priority planner determines a priority behavior for the autonomous vehicle based at least on the queuing element data structure. A control system of the autonomous vehicle can operate the autonomous vehicle according to the right-of-way behavior so that the autonomous vehicle safely runs through the right-of-way scenario.

Description

background

Advances in machine vision techniques, neural network architectures, and computing substrates are beginning to enable autonomous vehicles such as B. Land-based autonomous vehicles (e.g. self-driving cars and trucks) and robots. For the public and government regulators to accept widespread deployment of self-driving cars and trucks on the roads, the self-driving cars and trucks must achieve a level of safety that surpasses current safety levels of the average human driver. Safe and effective driving requires that all drivers can be confident that other vehicles in the area will properly yield the right-of-way when they are required to do so. If a vehicle does not yield the right-of-way, drivers of other nearby vehicles may not be able to drive safely and efficiently because other drivers are "unpredictable", e.g. B. Drivers who have signaled by their behavior that they may not give way when required to do so. A necessary condition for the use of self-driving cars and trucks is therefore that the self-driving cars and trucks handle the right-of-way scenarios (e.g. intersections and lane junctions) “safely and politely”.

Typically, local traffic regulations and driving protocols determine which drivers (and under what conditions) have the responsibility or obligation to give way to others. These regulations include traffic laws (e.g., vehicles must give way to pedestrians at a crosswalk), situational signage (e.g., a street sign indicating which entrances to an intersection must give way to other entrances), and other real-time cues ( e.g. the almost simultaneous arrival of several vehicles in a roundabout). However, conventional autonomous vehicles are not capable of encoding and deploying such protocols. Instead, conventional systems may aim to avoid collisions without considering right-of-way protocols and thus cannot safely and predictably navigate through right-of-way scenarios.

Summary

Embodiments of the present disclosure relate to coding of give-way scenarios for autonomous systems (e.g., a manned or unmanned vehicle or a robot). Systems and methods are disclosed that provide real-time control of autonomous systems when the systems encounter a right-of-way scenario.

In contrast to conventional systems such as those described above, the disclosed embodiments enable autonomous systems to handle give-way scenarios in a safe and predictable manner. In at least one embodiment, in response to detecting a right-of-way scenario, a data structure is generated that encodes the geometries of an ego path, a competitor path that contains at least one contention point with the ego path, and a contention state that contains the at least one point of competition is connected. The geometry of the context of a right-of-way scenario can also be encoded, e.g. B. the geometry that defines an area of the inner surface of an intersection (e.g. as a polygon), entry or exit lines, etc. The data structure is passed to a priority planner of the autonomous system. The right-of-way planner determines a right-of-way behavior for the autonomous system based at least on the data structure. A control system of the autonomous system can operate the autonomous system according to the right-of-way behavior so that the autonomous system safely runs the right-of-way scenario.

In at least one embodiment, a give-way scenario (e.g., an intersection or T-junction give-way scenario) may be identified based at least on analysis of sensor data generated by at least one sensor of an autonomous vehicle. Map localization and/or perception may be used to determine various information related to the right-of-way scenario. For example, a first path for the autonomous vehicle and a second path for a competitor (eg, another vehicle or object) may be determined by the right-of-way scenario. At least one contention point may exist between the routes, which may indicate that a collision is possible when the routes are traversed. To determine a contention state (which defines how the vehicle should behave) of the at least one contention point, the system may determine one or more of the traffic rules that apply to the right-of-way scenario. The wait element data structure (also referred to as the wait element) can then encode information used by the vehicle to navigate the right-of-way scenario, such as: B. the geometry of the paths, the state of contention and other information. The waiting element can be provided for example for a control agent of the vehicle. The control agent may be enabled to use the wait element to to determine right-of-way behavior for the first vehicle.

character list

• 1 ist ein Beispiel für ein Vorfahrtsszenario gemäß einigen Ausführungsformen der vorliegenden Offenbarung, gemäß einigen Ausführungsformen der vorliegenden Offenbarung;
• 2 zeigt ein nicht einschränkendes Beispiel einer Datenstruktur eines Warteelements und ein nicht einschränkendes Beispiel einer Datenstruktur eines Konkurrenzzustands gemäß einigen Ausführungsformen der vorliegenden Offenbarung;
• 3 zeigt ein nicht einschränkendes Beispiel einer Warteelement-Engine gemäß verschiedenen Ausführungsformen, gemäß einigen Ausführungsformen der vorliegenden Offenbarung;
• 4 ist ein Flussdiagramm, das ein Verfahren zum Codieren von Vorfahrtsszenarien für autonome Fahrzeuge (z.B. ein Ego-Fahrzeug) gemäß einigen Ausführungsformen der vorliegenden Offenbarung zeigt;
• 5 ist ein Flussdiagramm, das ein Verfahren zum Codieren von Vorfahrtsszenarien für autonome Fahrzeuge (z.B. ein Ego-Fahrzeug) gemäß einigen Ausführungsformen der vorliegenden Offenbarung zeigt;
• 6 ist ein Flussdiagramm, das ein Verfahren 600 zum Auflösen eines Konkurrenzzustandes zwischen Wegen für Fahrzeuge zeigt, gemäß einigen Ausführungsformen der vorliegenden Offenbarung;
• 7A ist eine Darstellung eines beispielhaften autonomen Fahrzeugs gemäß einigen Ausführungsformen der vorliegenden Offenbarung;
• 7B ist ein Beispiel für Kamerapositionen und Sichtfelder für das autonome Fahrzeug der 7A, gemäß einigen Ausführungsformen der vorliegenden Offenbarung;
• 7C ist ein Blockdiagramm einer beispielhaften Systemarchitektur für das beispielhafte autonome Fahrzeug der 7A, gemäß einigen Ausführungsformen der vorliegenden Offenbarung;
• 7D ist ein Systemdiagramm für die Kommunikation zwischen Cloud-basierten Servern und dem beispielhaften autonomen Fahrzeug der 7A, gemäß einigen Ausführungsformen der vorliegenden Offenbarung;
• 8 ist ein Blockdiagramm einer beispielhaften Rechenvorrichtung, die zur Verwendung beim Implementieren einiger Ausführungsformen der vorliegenden Offenbarung geeignet ist; und
• 9 ist ein Blockdiagramm eines beispielhaften Datenzentrums, das zur Verwendung beim Implementieren einiger Ausführungsformen der vorliegenden Offenbarung geeignet ist.

The present systems and methods for coding give-way scenarios for autonomous systems are described in detail below with reference to the attached drawing figures, wherein:

• 1 FIG. 14 is an example of a give-way scenario, in accordance with some embodiments of the present disclosure, in accordance with some embodiments of the present disclosure;
• 2 Figure 12 shows a non-limiting example of a wait item data structure and a non-limiting example of a contention state data structure, in accordance with some embodiments of the present disclosure;
• 3 12 shows a non-limiting example of a wait item engine according to various embodiments, in accordance with some embodiments of the present disclosure;
• 4 12 is a flow chart depicting a method for encoding give-way scenarios for autonomous vehicles (eg, a ego vehicle) according to some embodiments of the present disclosure;
• 5 12 is a flow chart depicting a method for encoding give-way scenarios for autonomous vehicles (eg, a ego vehicle) according to some embodiments of the present disclosure;
• 6 6 is a flow diagram depicting a method 600 for resolving a lane contention for vehicles, according to some embodiments of the present disclosure;
• 7A 1 is an illustration of an exemplary autonomous vehicle, in accordance with some embodiments of the present disclosure;
• 7B is an example of camera positions and fields of view for the autonomous vehicle 7A , according to some embodiments of the present disclosure;
• 7C 12 is a block diagram of an example system architecture for the example autonomous vehicle of FIG 7A , according to some embodiments of the present disclosure;
• 7D 12 is a system diagram for communication between cloud-based servers and the example autonomous vehicle of FIG 7A , according to some embodiments of the present disclosure;
• 8th Figure 12 is a block diagram of an example computing device suitable for use in implementing some embodiments of the present disclosure; and
• 9 1 is a block diagram of an exemplary data center suitable for use in implementing some embodiments of the present disclosure.

Detailed description

Systems and methods for encoding give-way scenarios for autonomous vehicles are disclosed. Although the present disclosure relates to an exemplary autonomous vehicle 700 (alternatively referred to herein as “vehicle 700” or “ego vehicle 700”, an example of which relates to the 7A-7D described), this should not be construed as a limitation. For example, the systems and methods described herein can be used without limitation by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), steered and unguided robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles with a or multiple trailers, airships, boats, shuttles, ambulances, motorcycles, electric or motorized bicycles, airplanes, construction vehicles, underwater vehicles, drones and/or other types of vehicles. Although the present disclosure is described with respect to controlling an autonomous vehicle to negotiate a give-way scenario, this is not intended to be limiting and the systems and methods described herein may be used in the fields of augmented reality, virtual reality, mixed reality, robotics, security and monitoring, autonomous or semi-autonomous machine applications and/or in any other area of technology where autonomous control systems can be used.

During normal operation of an autonomous vehicle, a control agent must avoid both moving and non-moving obstacles (e.g., other vehicles, pedestrians, cyclists, lane barriers, and the like). In addition to avoidance, an agent generally has the responsibility to give way to other road users in certain scenarios (e.g. "right of way conditions"). Such right-of-way conditions may be at (controlled and uncontrolled) intersections, pedestrian crossings, T-junctions, on/off ramps on motorways, roundabouts and the like, e.g. B. when driving through multi-storey car parks and/or parking lots. In order to give another road user the opportunity to overcome the priority situation safely, confidently and efficiently, the priority behavior can include braking or even stopping the vehicle completely. At an unmarked intersection where another vehicle has already arrived, a following vehicle may give way by decelerating to allow the first vehicle to safely pass through the intersection. Such a right-of-way behavior ensures that the following vehicle only enters the intersection when the first vehicle has safely left the intersection. Under such give-way conditions, one or more road users may have a clearly defined obligation (or responsibility) to give way to other road users.

Right-of-way behavior provides a benefit beyond mere collision avoidance. Right-of-way behavior can ensure “polite and expected” driving dynamics, which are necessary for safe and efficient traffic. For example, even if a right-of-way agent avoids a potential collision (e.g., by accelerating at an intersection), the lack of right-of-way creates tense and uneasy driving conditions for all road users in the area. Even when accelerating to avoid a collision, a vehicle that does not have the right of way can create fear, a sense of danger, and anger (e.g., road rage) in other drivers, cyclists, and pedestrians. That is, even if a collision is avoided through aggressive behavior, the collision was not avoided in a "safe and courteous manner" as expected by other road users. Accordingly, in the operation of an autonomous vehicle, an agent for the autonomous vehicle may be required (e.g., either legally or normatively) to employ one or more behavioral right-of-way strategies when approaching a right-of-way scenario.

The disclosure provides in part for an autonomous vehicle (“ego vehicle”) “waiting element engine” that can actively monitor the arrival of one or more right-of-way scenarios (e.g., when the vehicle arrives at an intersection, when the vehicle has an on -/downhill or when the vehicle is preparing to change lanes). The wait element engine may generate one or more "wait element" data structures that encode a representation of the right-of-way scenario. The waiting elements can be provided as input to a "priority planner" for the ego vehicle. Examples of priority scenarios are intersections (e.g. junctions) and junctions (e.g. junctions of lanes at on- and off-ramps). Each right-of-way scenario can involve at least two actors: the ego vehicle and at least one competitor (eg, another vehicle, pedestrian, cyclist, etc.). A competitor can be another vehicle (e.g., an autonomous, semi-autonomous, and/or a conventional manually operated vehicle) and people (pedestrians and cyclists).

In various embodiments, a give-way scenario may involve more than one competitor (e.g., the ego vehicle approaching an intersection with multiple other vehicles, pedestrians and/or cyclists at an intersection, the ego vehicle entering a lane with several other vehicles, etc.). Each actor associated with a right-of-way scenario can be associated with one or more “potential paths” or lanes. For an actor of a right-of-way scenario, the potential paths or lanes may include the set of current or possible spatial paths e.g. B. are given by the current coordinates of the actor in a space-velocity-phase space. Thus, an actor's potential path may depend not only on its current space and velocity coordinates, but also on the vehicle's (or person's or entity's) limitations in terms of acceleration, deceleration (e.g. braking power) and maneuverability (e.g e.g. curve radius, traction control, etc.).

When a give way condition is detected, the wait element engine may receive and/or generate environmental data derived from various sources (e.g., onboard and/or offboard sensors and/or detectors, perception-based data, map-based data, geolocation data, etc.) and refer to the right-of-way scenario. The waiting element engine can parse and merge the various data, as well as inspect, analyze and match the data to various traffic rules related to right-of-way with respect to the merged and analyzed data to generate various "waiting geometries" and "states of contention" that the Can characterize priority scenario for the ego vehicle. For example, data encoding the vehicles' potential paths and various aspects of the geometry of the scenario can be analyzed and matched against one or more right-of-way or traffic rules to determine a state of contention of the right-of-way scenario (e.g., right-of-way, stop at the entrance , give way at the entrance and the like). The waiting geometries can be divided into one or more "Waiting groups pen", whereby a waiting group can refer to all waiting elements for a right-of-way scenario. The wait geometries and the contention states can be encoded in a "wait element" data structure. The queuing element data structure(s) may be provided to an autonomous vehicle "priority planner" for use in controlling the vehicle.

The priority planner can receive the data structures of the waiting elements and determine a suitable priority behavior. When a control agent for the ego vehicle adopts the specified right-of-way behavior (e.g., defined by a contention state), the ego vehicle can safely fulfill its required and expected right-of-way obligations while avoiding collisions.

In at least one embodiment, a queuing element engine may receive and/or collect various input data, which may include geometry-related data, signal-related data, and map-related data. Capturing (or receiving) data may be done using detection, perception, and/or recognition techniques, which may utilize geometry or visual perception, map perception (which may include localization), and signal perception. In at least one embodiment, the perception data may include lane graph data. Lane graph data may include one or more paths designated as potential paths of the ego vehicle (e.g., ego paths) and one or more paths designated as potential paths of one or more competitors (e.g., competitor paths). can be assigned. Other input data may include various raw sensor data from the ego vehicle or the competitors. Geometry input data may include various information about the geometry of the environment that is applied to the potential paths associated with the right-of-way scenario and/or the background context. Signal input data may include and/or encode traffic signals such as traffic lights, traffic signs, stop signs, give way signs, give way signs such as main road signs, speed signs, and hand signals or other postures used to signal traffic.

In various embodiments, a queue element data structure may be created for each possible pairing of a potential path of a host vehicle and a potential path of a competitor. In non-limiting examples of right-of-way scenarios, a right-of-way scenario is associated with a ego vehicle and j competitors, where j is a positive integer. The ego vehicle may be associated with i potential paths and each of the j competitors with k potential paths, where i and k are also positive integers. In such an example, the wait element engine may generate i×j×k separate wait elements. Therefore, each waiting item can be associated with a potential path for the ego vehicle and a potential path for the competitor. The wait element may encode a "wait geometry" for the ego path, a wait geometry for the competitor path, and a wait geometry for a context of the two paths. A wait element can also encode a "contention state" between the two paths.

Briefly, a queuing geometry for a claimed path (e.g., an ego path or a competitor path) may contain a set of field-value pairs or other data types or elements for the path that encode various aspects of the claimed path. Such fields for the waiting geometry of a way can include entry lines, exit lines, entry and exit areas of competitors, entry line and interior surface of an intersection, points of contention between an ego way and a competitor way (optional explicit encoding of one of the crossing or junction points between ways ) and the like. The contention state (e.g., contention state) of a wait item may be an instruction to the priority planner or define how the ego vehicle should yield or yield the right-of-way with respect to that wait item. Such states include, but are not limited to: right of way, stop at the entrance, give way at the entrance, and the like.

In at least one embodiment, input geometry data, which may be referred to as wait geometry data, may be “merged” with the lane graph data and the map data to create a wait element. The "fused geometry data" can then be classified (e.g. as left turns, right turns, U-turns, etc.) and associated with one or more paths. The signal data can be merged with the map data. A signal state (e.g. green light, red line, inactive, etc.) can be determined from the merged signal data. The merged, classified, and associated geometry data, along with the map data, signal state, and other data can be fed as input to a "state contention solver" of the waiting element engine. The contention state solver can use the geometry, signal, map, and other sensor data along with traffic rules to determine a contention state determine and resolve the data into a wait element.

With reference to 1 shows 1 an example of a right-of-way scenario 100 according to some embodiments of the present disclosure. The non-restrictive right-of-way scenario 100 der 1 is an example of a right-of-way scenario at an intersection (or junction). Other types of give-way scenarios include, at a minimum, give-way scenarios at an intersection (e.g., at a freeway slip road). In this non-limiting example of an intersection priority scenario 100, three vehicles are approaching a 4-way intersection. The three vehicles are a first vehicle 102 (e.g., an ego vehicle), a second vehicle 104 (e.g., a first competitor), and a third vehicle 106 (e.g., a second competitor). . A queuing element engine 130 is employed to generate one or more queuing element data structures (e.g., a queuing element_1 110 and a queuing element_2 120) for the right-of-way scenario 100 . In some embodiments, the queuing element engine 130 may be onboard the ego vehicle 102 . In other embodiments, the queuing element engine 130 may be at least partially remote from the ego vehicle 102 . In such embodiments, the ego vehicle 102 may access the queuing element engine 130 via one or more communication networks.

Various embodiments of wait item engines are described at least in connection with wait item engine 300 of FIG 3 discussed. It is understood that these and other arrangements described herein are to be understood as examples only. Other arrangements and elements (e.g., engines, interfaces, functions, orders, groupings of functions, etc.) may be used in addition to or in place of those shown, and some elements may be omitted altogether. In addition, many of the elements described herein are functional units that can be implemented as single or distributed components, or in conjunction with other components, and in any suitable combination and location. Various functions described herein performed by devices may be performed by hardware, firmware, and/or software. For example, various functions may be performed by a processor executing instructions stored in memory.

Wait element data structures such as e.g., wait_item_1 110 and wait_item_2 120, may be associated with a pair of claimed paths, one of the claimed paths being a claimed path of the ego vehicle 102 and another of the paired claimed paths being a claimed path for a competitor (e.g. a first competitor 104 or a second competitor 106). Each waiting element may have a waiting geometry for the ego vehicle's 102 claimed path (e.g., an ego waiting geometry 112 of waiting element_1 110 or an ego waiting geometry 122 of waiting element_2 120) and a waiting geometry for the competitor's claimed path (e.g., encode a competitor_1 queuing geometry 114 of queuing element_1 110 or a competitor_1 queuing geometry 124 of queuing element_120). Accordingly, a Waiting Geometry_1 110 may be associated with a single claimed path of the ego vehicle 102 and a single claimed path for the first competitor 104 (in other examples, multiple paths may be associated with a waiting element). Likewise, a Waiting Geometry_2 120 is associated with a single claimed path of the ego vehicle 102 and a single claimed path for the second competitor 106 . Note that the claimed path of ego vehicle 102 associated with wait item_1 110 may (but need not be) the same claimed path of ego vehicle 102 associated with wait item_2 120 . Various embodiments of waiting element geometries are discussed in connection with at least 2 discussed.

In addition to the wait geometries for a pair of claimed paths, each wait element may encode a wait geometry for a context (eg, wait geometry context 116 of wait element_1 110 and wait geometry context 126 of wait element_2 120). In addition, each wait element may encode a contention state for the paired claimed paths (e.g., contention state_1 118 of wait element_1 110 and contention state_2 of wait element_2 120). Various embodiments of wait geometry contexts and contention states are described at least in connection with 2 discussed.

More generally, a wait element may be a subset of the wait geometry of an ego path (e.g., a claimed path of the ego vehicle) and the wait geometry of a competitor path (e.g., a claimed path for a competitor of right-of-way scenario 100), contain (or encode) a wait geometry context and a contention state. Ego waiting geometry 112, competitor_1 waiting geometry 114, waiting geometry context 116, and contention state_1 118 (of waiting element_1 110) may be data objects and/or data structures. Likewise they can ego waiting geometry 122, a competitor_2 waiting geometry 124, a waiting geometry context 126, and a contention state_2 128 (of waiting element_2 120) can be data objects and/or data structures. In various embodiments, the data encodings for these missing items may be set to "invalid" and/or "not applicable" if data values (or items) of one or more of these data objects are not readily available (or not applicable to a particular right-of-way scenario).

In one or more embodiments, the waiting elements (e.g., waiting element_1 110 and waiting element_2 120) form the "atoms" for how information (or data) about waiting conditions (e.g., right-of-way scenario 100) may be encoded. The waiting elements may be provided as input to the ego vehicle's priority planner 102 (in 1 not shown). At least based on the coding of the waiting elements, the right-of-way planner can determine a suitable right-of-way behavior in order to drive through the right-of-way scenario 100 safely. As discussed herein, a wait item may be determined and/or created using at least one of two methods (and/or a combination thereof). One method involves applying a set of right-of-way traffic rules (e.g., right-of-way heuristics) to the right-of-way scenario 100. Another method involves applying a mapping and real-time perception of geometry-related and/or signal state data to the right-of-way scenario 100. As previously mentioned, can the two methods can be combined in various embodiments to employ mapping data, real-time perception of geometry/signal data, and right-of-way heuristics.

When we turn our attention to the 2 turn, shows 2 a non-limiting example of a wait item data structure 200; and a non-limiting example of a contention state data structure 210 (also referred to as a contention state) according to some embodiments of the present disclosure. In various embodiments, a wait geometry 200 and/or a contention state 210 may be a data object or other such structured data. In general, the waiting geometry 200 may represent geometry and associated metadata that result when additional information (e.g., information and/or data related to a right-of-way scenario, e.g., the right-of-way scenario 100 of 1 ) are applied to the lane graphs (e.g., a path claimed by a ego vehicle and/or a path claimed by the competitor). That is, the wait geometry 200 may be applied to a ego path (e.g., a ego vehicle's claimed path), a competitor path (e.g., a competitor's claimed path), or to a background context. For example, if the waiting geometry 200 relates to an ego path, the waiting geometry 200 may be the ego waiting geometry 112 of the waiting element_1 110 and/or the ego waiting geometry 122 of the waiting element_2 120 of 1 be similar to. If the queuing geometry 200 is for a competitor path, the queuing geometry 200 may be similar to the queuing_1 queuing geometry 114 of queuing_item_1 110 and/or the queuing_2 queuing geometry 124 of queuing_item_2 120 . If the wait geometry 200 relates to a background context, then the wait geometry 200 may be similar to the wait geometry context 116 of the wait element_1 110 and/or the wait geometry context 126 of the wait element_2 120 . Such contextual waiting geometries may relate to the boundaries and/or the area of the interior surface of an intersection or the presence of an intersection entry line.

In various non-limiting embodiments, the queuing geometry may encode at least some of its data in field-value pairs. Accordingly, a set of fields (e.g., wait geometry fields 202) may be associated with (or encoded in) wait geometry 200. FIG. Each field of the waiting geometry fields 202 can be assigned one or more values to encode a set of field-value pairs. Note that a value can be a data structure or data. In some embodiments, a value for a particular field may be another field, such that the cue geometry 200 may encode one or more data trees. As in 2 1, such fields 202 may include, but are not limited to: entry line (e.g., of the corresponding ego way or competitor way), exit line (e.g., of the corresponding ego way or competitor way), entry line competitor area (e.g., of the corresponding ego way or competitor way), exit competitor area (e.g., of the corresponding ego way or competitor way), and intersection entry line (e.g., of the corresponding ego way or competitor way). In some embodiments, fields 202 may contain coordinates or other information defining the boundaries and/or location of an area of the interior surface (as part of the general context of a queuing group), as well as one or more points of contention between an ego path and a competitor path. Path (optional explicit coding of one of the crossing or junction points between paths). The fields 202 can additionally contain a speed limit applicable in the general context (to be considered as applying between the entry and exit lines). The values for each of these fields can be coded as invalid to allow coding of wait conditions where the fields do not apply to the given right-of-way scenario (e.g. a traffic light at an on-ramp has only one exit and one entrance, but none exit, no oncoming lane and no interior surface). Another example would be a queue group encoding a new speed limit that only contains an entry line and a speed limit in the overall context, with everything else set to invalid. The exit line is then to be interpreted as infinity or until further notice, and the situation is similar with other properties.

Entry lines for an ego path can encode a stopping point for several of the right-of-way behaviors. Entry lines can also signal the beginning of the general contention area, terminated by the exit line, which can indicate which segment of the ego path needs to be cleared in order to resolve that waiting set of waiting conditions. A region of an interior surface may represent the interior surface of an intersection as a polygonal surface that can be related to a coordinate space. A portion of an interior surface may include the segment between the entry line and the corresponding exit line (sometimes the exit line may be moved outward, e.g., past a pedestrian crossing, even though the interior surface does not). The driveways and interior grounds provide a context for analysis by other actors. This can be done by assigning actors to pathways and areas (in a non-mutually exclusive way). The geometry of the ego and competitor paths, as well as the competition point, can be used by the right-of-way planner to implement the right-of-way behavior as needed. Geometry can also be used to determine which rules apply.

In some examples, a contention point may indicate or represent an explicit geometric point. In other examples, a contention point may refer to an abstract concept that is the particular contention that a wait element refers to and/or whose state it encodes. In such examples, the contention state at a contention point may be the payload of the overall contention resolution process. For each contention point, it can provide a determination of the manner in which the ego vehicle should or should not give way with respect to that contention point. In this sense, the competition point can also indicate the choice of the ego path, access to a competitor path and via this competitor path, the actual competitors and how the ego vehicle is to behave towards them .

The waiting geometry can be grouped together (e.g. organized logically), with the semantic meaning of a waiting group being that all waiting conditions in this group can be considered together and, in particular, clarified together, so that the ego vehicle does not stand in between (e.g. the ego vehicle should not have to wait for pedestrians at the end of a left turn while the ego vehicle is still in the lane of oncoming traffic, so that oncoming traffic can be accounted for along with the pedestrian crossing in the same queue).

Referring to the contention state 210, a queuing engine, such as B. the wait element engine 300 of the 3 , contain a contention state solver (e.g., a contention state solver 340 of the queuing element engine 300). Such a contention solver can perform a contention solving process. The goal of a contention state resolution process may be to resolve a contention state (e.g., the contention state 210) for each of the wait items (e.g., wait item_1 110 and wait item_2 120 of the 1 ) to provide. The contention state 210 of a waiting item may be an instruction to the traffic planner as to how the ego vehicle should yield or give way to that waiting item, as a rule, expectation, formal or informal convention, or norm.

In some non-limiting embodiments, the contention state 210 may not indicate what is actually happening in the right-of-way scenario, what is physically possible in the right-of-way scenario, or whether the ego vehicle can be forced to give way even though it is in the right-of-way scenario (e.g., a crossing priority scenario or a merging priority scenario) has the right of way. Rather, the contention state 210 may indicate what should happen according to the conventions. It may then be the responsibility of the right-of-way planner to actually implement the right-of-way, in the sense that it takes into account what should happen (e.g., as encoded in the contention state 210), whether the ego vehicle is actually in is able to stop and follow this instruction and whether the other actors (e.g. the Konkur yields of the right-of-way scenario) appear to be fulfilling their expected right-of-way obligations and takes appropriate action. For example, the priority planner can For example, discovering that a competitor is not yielding the right of way when the Right of Way state is (essentially capturing "honk appropriately") and decide to yield the right of way when it should not be. The priority planner can implement a priority behavior that analyzes all contentions in a queue group until all queue items in the group can be resolved collectively. All contentions in the queue can be obeyed together, ie the most restrictive contention can determine the expected right-of-way behavior of the ego vehicle. For example, if one state of the queue is Right of Way and another state is Stop at Entry, the ego vehicle may remain at the entry line.

As in 2 As illustrated, contention state 210 (e.g., as encoded in a wait element) may include one or more of the seventeen states listed in contention state 210 . Note that this list of possible contention states is not exhaustive and that in other embodiments contention state 210 may include additional and/or alternative contention states. The Right of Way state may indicate that the competitors are expected to yield the right-of-way. Accordingly, the Right-of-Way state cannot imply any formal restriction on the corresponding queue element (other than that the planner observes the competitors associated with that queue element and ensures that they yield the right-of-way as expected). For the various contention states, the keyword may be used temporarily to indicate that the corresponding contention state may be updated and/or evolve in the near future. For example, a Temporary Right-of-Way state may indicate that a Right-of-Way state is currently in effect but is likely to change to a more restrictive state soon. The canonical example is the "amber" state of a traffic light, which can be coded as "Temporary Right-of-Way". A “stop at entry” may mean that there is an instruction to stop at the entry line, await further instructions and not proceed until the state of the competition changes. A yield at entry state may indicate that the ego vehicle should remain at the entry line until the competition is expected to be resolved. In such entry stop conditions, a prior stop is not mandatory, but a ego vehicle control agent should ensure that the competition is eliminated before the ego vehicle passes the entry line, which often results in a prior stop. In practice, since the queuing conditions in the queuing group can be considered together, this usually means that the control agent should ensure that all contentions in the queuing group are resolved before the ego vehicle passes the entry line. In other words, if a competitor in the queuing group has the state "Give Way at the Gateway", all other competitors in the queuing group can inherit the same contention state when analyzed by the priority planner, and if a competitor has the state "Stop at the gateway". , all waiting elements in the group may inherit the Stop Before state The Give way at the entrance temporarily state may be the transient version of the Give way at the entrance state.

The Yield at Competitive Point state may indicate that prior stopping is not mandatory, or that the ego vehicle must explicitly remain at the line of entry while awaiting competitor clarification (although there is nothing wrong with that in principle). Tax Agent may need to ensure that the ego vehicle properly yields to competitors associated with that competition, that the ego vehicle does not block the competition, and that the ego vehicle behaves in a manner that is clear to the competitors associated with that competition that the ego vehicle yields the right of way. This could mean that the ego vehicle is moving forward to make a left turn at an intersection, but slowly enough and with enough distance that oncoming traffic understands that the ego vehicle's apparent intent is to to pull ahead and that it obviously does not get in the way of oncoming traffic. The "Temporarily yield at the point of contention" state may be the temporary version of the "Give way at the point of contention" state. The stop at the entrance and then give way at the entrance condition may be equivalent (or at least similar) to the give way at the entrance condition, but with the additional stipulation that prior stopping is required at the line of entry. The stop at entry then give way at contention point condition may be equivalent (or at least similar) to the give way at contention point condition, but with the additional stipulation that prior stopping is required at the line of entry.

The Temporarily stop at the entry then give way at the point of contention condition may be the temporary version of the condition Stop at the entrance then give way at the competition point. The first-stop-priority condition may be the canonical case of the "US multiway stop". Right-of-way can be defined as a first-in-first-out queue, where "in" is defined as arriving at the intersection (probably in the relevant competitor's area at the entry line pointing into the interior surface) as the first actor out of it Competitor Walk and Stopping. In other words, this contention state may imply further processing of "who stopped first" to actually result in a Yield or Yield state for each actor connected to that contention path.

The Negotiating state may indicate that there is no known basis for determining the right of way, such as B. at a motorway junction, for which there are no clues based on traffic rules, map statistics, the geometry or the size of the road (motorways of the same size that merge and have a similar straight shape). The state of stopping at the entry and then negotiating may be equivalent (or at least similar) to the state of negotiating, but with the additional stipulation that a prior stop is required at the line of entry. This state can be used when there is no convention to be followed but there is a clear line of entry. The Not Allowed state can be used to enable coding that something is not allowed. For example, the waiting element could include a path to turn left across traffic into a parking lot, and there could be signals indicating that the turn is prohibited (e.g., via a solid double yellow line). In this case, the ego vehicle can act by not only signaling (to other road users) that the condition is to stop now, but also that it will never change, as it is simply not allowed. This state can be useful for the right-of-way planner as it considers multiple ego path choices (e.g., multiple ego paths may be considered simultaneously). The Stop and Request Takeover state may indicate that the ego vehicle has encountered something outside of the operational design area (e.g. signals indicating a roadworks, or a warning post may have been detected and the ego vehicle has handling of such a condition is not yet implemented). In such states, the control agent (or the precedent planner) may request the ego vehicle to significantly slow down or stop and request vehicle takeover. The Unknown state can be used to encode predictions for future contentions, where it may be useful to encode that there is no knowledge or prediction.

3 FIG. 3 shows a non-limiting example of the wait item engine 300) according to some embodiments of the present disclosure. As previously mentioned, the queuing element engine 300 may reside onboard an autonomous vehicle (e.g., the ego vehicle 102 of FIG 1 ). In other embodiments, an autonomous vehicle may access a remote queuing element engine via one or more communication networks. As previously mentioned, the wait item engine 300 may have one or more wait items (e.g., the wait item_1 110 of the 1 , the wait element_2 120 the 1 and a waiting element 310 of 3 ) generate as an output when an ego vehicle approaches an intersection or junction give-way scenario (e.g., an intersection give-way scenario 100 of 1 ). Various inputs to the wait item engine 300 are discussed below.

The waiting element engine 300 may include a waiting geometry perceiver 302, a mapper 304, a signal perceiver 306, a lane recorder 350, and/or other sensor data receiver 308. The waiting geometry perceiver 302, the mapper 204, the signal perceiver 306, the lane recorder 250 and the other sensor data receiver 308 receive various inputs as described below. The wait element engine 300 may further include a wait geometry blender 322 , a geometry classifier 324 , and a geometry associator 326 . The queuing engine 300 may also include a signal fuser 328 , a signal state estimator 330 , and the contention state solver 340 . The contention state solver 340 may include a constraint checker 342 , a base rule parser 344 , a mapping rule checker 346 , and a wait element fuser 348 . The output wait element 310 may include an ego wait geometry 312 , a competitor wait geometry 314 , a context wait geometry 316 , and a contention state 318 .

The lane recorder 350 is generally responsible for receiving one or more lane graphs as input to the queu engine 300 . The lane graphs may be received in response to approaching and/or detecting a give-way scenario. A lane graph may contain a set of claimed paths from the same lane cluster and may include a set of ego paths 352 and a set of competitor paths 354 . The ego paths 352 may be received or generated from the output of one or more neural networks or other machine learning models, a map, and/or vehicle tracks. For example and without limitation, the machine learning models described herein may include any type of machine learning model, such as one or more machine learning models using linear regression, logistic regression, decision trees, support vector machines (SVM), naive Bayes, k-nearest neighbor (Knn), K-means clustering, random forest, dimensionality reduction algorithms, gradient boosting algorithms, Markov random fields, neural networks (e.g., autoencoder, convolution algorithms, recurrent algorithms, perceptrons, long/short term memory (LSTM) , Hopfield, Boltzmann, Deep Belief, Deconvolutional, Generative Adversarial, Liquid State Machine, etc.) and/or other types of machine learning models.

The ego paths 352 can be merged in the ensemble into a new lane graph. The lane graphs may be structured in an array of lane graphs, providing the flexibility to use any combination of lane graphs. One or more of the competitor paths 354 may intersect with one or more of the ego paths 352 at one or more contention points. A particular contention point can be classified as one of two primary contention point types: intersection contention points or junction contention points. For example, in intersection competition, an ego path may intersect with a competitor path at a single point (or in a small neighborhood of points), but is not limited thereto. At junction contention points, an ego path may meet and join a competitor path for at least one segment of the competitor path (or vice versa). The competitor paths 354 may be received or generated from outputs of one or more neural networks or other machine learning models, a map, or an intersection analysis. While the ego paths 352 are claimed paths for the ego vehicle, the competitor paths are claimed paths by vehicles, bicycles, pedestrians, trams and trains, or any other actor in the right-of-way scenario. Like the ego paths 352, the competitor paths 354 can be assembled from their sources (which may be similar or the same as the sources of the ego paths 352) into an ensemble.

The waiting geometry perceiver 302 and the signal perceiver 306 may generally perform a "waiting perception". The waiting geometry perceiver 302 receives various geometry-related inputs (in response to identifying and/or recognizing the ego vehicle approaching a right-of-way scenario) and generates a waiting geometry for the output (e.g., the waiting geometry 200 of FIG 2 ). Thus, the waiting geometry perceiver 302 may generate a waiting geometry (eg, ego waiting geometry 312 ) for each ego path of the set of ego paths 352 . Likewise, the queuing geometry perceiver 302 may generate a queuing geometry (e.g., competitor queuing geometry 314 ) for each competitor path of the set of competitor paths 354 . Likewise, the signal perceiver 306 receives various signal-related inputs (in response to identifying and/or detecting the ego vehicle approaching a right-of-way scenario) and generates a signal for output.

A queuing geometry (e.g., ego queuing geometry 312 or competitor queuing geometry 314) may include geometry and metadata that result when additional information about queuing conditions is applied to the lane graph. The waiting geometry may refer to an ego path (e.g., an entry line), a competitor path (e.g., a competitor area), or to a background context (e.g., the interior surface of an intersection or the presence of an entry line). . As related to at least Waiting Geometry 200 of the 2 discussed, a particular waiting geometry may contain field-value pairs for: entry and exit lines (for both ego and competitor paths), entry and exit areas for competitors (for both ego and competitor paths). ), intersection entry line and interior surface (as part of the general context of a queuing group), points of contention between an ego path and a competitor path (optional explicit coding of one of the crossing or junction points between paths). A waiting geometry may contain a field-value pair for a speed limit applied in the general context (considered applicable between the entry and exit lines). The value for each of these fields can be coded as invalid to allow coding of wait conditions where the fields do not apply to the right of way scenario (e.g., a traffic light at a freeway entrance has only an ego lane and an entry line, but no exit line, no oncoming lane and no interior surface). Another example is the coding of a new speed limit by a queuing group that only contains an entry line and a speed limit in the overall context, while everything else is set to invalid. The exit line can then be interpreted as infinite or until further notice, and similarly with other properties.

A value for the line-of-entry field for an ego path may encode the stopping point of the path claimed by the ego vehicle for several possible right-of-way behaviors. Such entry lines may also encode the beginning of the general contention area, terminated by the exit line, which may indicate which segment of the ego path must be cleared in order to clear that queuing group of queuing conditions. The area of the inner surface can represent the inner surface of an intersection as a polygonal surface. The area of the interior surface can cover the path segment between the entry and exit lines (sometimes the exit line is moved outwards, e.g. via a pedestrian crossing, although the interior surface is unaffected). The driveway areas and interior surface can provide context for analysis by other actors. This can be made possible through an Obstacle-in-the-Path Analysis (OIPA) that maps actors to pathways and areas (in a non-mutually exclusive manner). The geometry of ego and competitor paths, as well as the competition point, can be used by the right-of-way planner to implement the right-of-way as needed. The wait geometry can also be used to determine which rules apply.

A competition point may represent an explicit geometric point (e.g., an intersection of an ego path and a competitor path). In some embodiments, a contention point can be viewed as an abstract concept that represents the particular contention that a wait element refers to and whose state it encodes. In this latter sense, the contention state at a contention point may be the payload of the contention state resolution process (as performed by the contention state solver 340). The contention state can provide, for each contention point, the determination of "how the ego vehicle should yield or not yield the right-of-way with respect to that contention point. In this sense, when choosing the ego path, by accessing a competitor path and through that competitor path, the contention point may also indicate the actual competitors and how the ego vehicle should act in relation to them.

The waiting geometry can be summarized in one or more waiting groups, with the semantic meaning of a waiting group being that all waiting conditions in this group have to be taken into account together and, in particular, can be clarified together, so that the ego vehicle does not stand in the way (e.g The ego vehicle must not wait for pedestrians at the end of a left turn while the ego vehicle is still in the lane of oncoming traffic, so the oncoming traffic must be considered together with the pedestrian crossing in the same waiting group).

The mapper 304 may receive map data (e.g., based at least on the map localization) in response to identifying and/or detecting the ego vehicle approaching a right-of-way scenario. The map data may include one or more 2D or 3D maps surrounding the upcoming right-of-way scenario. The signal perceiver 306 receives signal-related data in response to identifying and/or detecting the ego vehicle approaching a right-of-way scenario. The signal-related data may include signal data, traffic light displays and traffic signs, e.g. B. stop signs, give way signs, such as main road signs, and code speed signs. The signal data can also encode the outputs generated by one or more neural networks or other machine learning models, e.g. For example, whether an intersection is an intersection with traffic signs, an intersection with stop signs, an unmarked intersection, a roundabout, a freeway ramp, a toll booth, or some other type. Signal data may include indications of police officers, warning posts or road workers directing traffic, barriers blocking the way, and traffic lights at a pedestrian crossing. The signal data may include indications of traffic cones, merge arrows, and any temporary elements placed on the road to divert traffic. In various embodiments, the signal perceiver 306 can distinguish the presence of a signal and its state. The signal perceiver 306 can perform live signal perception that provides both the presence and state of a signal (e.g., there is a traffic light at those 2D or 3D coordinates and its current state is amber). In some embodiments, a map (e.g., a map received from mapper 304) may include information about the likely presence and/or location of a signal, such as: B. the presence of a traffic light or a stop sign, while the condition of the signal, such. e.g. the status of a traffic light, can be provided by live perception (or infrastructure-to-vehicle communication) via the signal perceiver 306. Signal detection (via the signal perceiver 306) can provide the presence, nature and spatial characteristics of a signal, e.g. For example, the 3D position and bounding box of a traffic light or road sign, or a police officer surveying traffic.

The wait geometry blender 322 may "fuse" or combine the map data from the mapmaker 304 with the geometry data (e.g., wait elements) from the wait geometry perceiver 302 via a process referred to as wait geometry blending. The wait geometry blending may optionally be performed to obtain enhanced geometry information from a combination of live perception (e.g., real-time geometry perception as performed by the wait geometry perceiver 302) and the map information (as received from the mapper 304 and/or or to be provided). For example, the waiting geometry perceiver 302 may detect the presence of an intersection through live (e.g., real-time) perception. On a map, the intersection may also be noted in the map data. In some embodiments, the map and geometry data may be merged to instantiate an intersection when present in one of the sources and to associate the detections when present in both sources. Similarly, entry lines may be detected in real-time (via wait geometry perceiver 302) and/or provided in a map (via mapper 304) based at least on past map streams containing live detections or statistics of actual stops. Competitor areas, interior surfaces, path geometries and competition points can be recognized live and inserted into the map stream, which benefits future trips. The actual routes (the “de facto lane graph”) can also be obtained from multiple trips. Therefore, wait geometry merging can match and combine multiple sources to provide clear wait geometries for further processing stages.

The signal blender 328 may “fuse” or combine the map data from the mapper 304 with the signal data from the signal perceiver 306 via a process referred to as signal fusion. Signal merging may provide an option to match information from the live perception (via the real-time perception of the signal perceiver 306) with information from a map (received and/or provided by the mapmaker 304), such as live perception support with a confirmation of the presence of a traffic light, sign or type of intersection. Similar to geometry merging, signal merging may be optional in some embodiments. Such signal merging offers the possibility to make a state estimation for a hard-to-see traffic light, and/or real-time signal perception results as well.

The signal state estimator 330 is generally responsible for a process referred to as signal state estimation. The signal state estimation can determine and/or provide the states of traffic lights, the gestures of a traffic police officer directing traffic, the "stop" or "slow down" sign of a warning post, the state of the traffic light at a pedestrian crossing, or the position of a barrier blocking traffic. The result is usually a selection from an enumeration class (e.g. green/yellow/red) or a combination of several.

The geometry classifier 324 is generally responsible for geometry classification. Geometry classification categorizes waiting geometries into discrete classes. For example, geometry classifier 324 may classify both an ego path (e.g., as encoded in ego waiting geometry 312) and a competitor path (e.g., as encoded in competitor waiting geometry 314), such as B: "turn left", "straight ahead", "turn right", "turn around" and the like. This classification may be performed to standardize the path so that common rules such as "right turns on red are not allowed in Manhattan" can be applied to the path in the contention resolution process performed by the contention solver 340 . To apply such a rule, the trail can be placed in a classification that contains a common language (e.g., "turn right"). Note that such a classification can benefit from the context of other paths (e.g. a relatively straight shape might be a right turn if it is the rightmost path, while it might not be if there is also a very sharp right turn ). The geometry classification can also determine for a waiting element whether the ego or the competitor comes from the right.

Geometry classification can also be applied to path pairs. For example, the geometry classifier 324 can determine whether two paths (e.g., an ego path and a competitor path) intersect or converge (if not explicitly stated in the lane graph), where the contention points are, and which paths Coming from the right (to support the right-before-left right-of-way rules common in Europe and in some cases the US). To do this, at the contention point, one can check whether the direction vectors of the two paths are clearly not parallel, and if so, the sign of the 2D vector cross product between them (the sign of the determinant applied to the 2x2 matrix obtained from the Ego way direction vector stacked as the top row and the concurrency renten-path direction vector is formed as the bottom line). If the sign is positive, the competitor comes from the right. Note that this definition can mean that when the ego makes a left turn through oncoming traffic, the oncoming traffic is assumed to be coming from the right (since this is the case at the point of competition). If the directional vectors are nearly parallel at the point of collision (typically because it is an intersection), the vectors at the corresponding entry lines can be used.)

Geometry classifier 324 may also determine or consolidate from live perception sources (e.g., waiting geometry perceiver 302) whether a competitor's path originates from a "cul-de-sac" (e.g., a driveway, gas station, or parking lot). originates or is "significantly larger" or "significantly smaller" than a "cul-de-sac". That is, a road can be classified (e.g., as a "cul-de-sac") by the geometry classification process. The geometry classifier 324 can provide predicate-like classifications that allow the application of logical rules to a coherent collection of entry variables. Rather, some of this information may come directly from outputs of one or more neural networks or other machine learning models via a wait geometry data structure, rather than being a geometric determination.

The geometry classifier 324 can also determine whether a path has a property such as B., but not limited to: "crosses a line", which can be an explicit determination of whether the path crosses a line (e.g., a path can be classified as "crosses a line"). Such a classification can be used as a guide for some rules when the priority is unclear how to decide. if e.g. For example, where two competing and otherwise equal routes are competing but one crosses a line and the other does not, the route that does not cross any lines may have priority. A lane turning left into oncoming traffic can be classified with the properties "crosses a dashed line", "crosses a solid line", or "crosses a solid double yellow line" so that country-specific rules can be applied to determine whether this is allowed or not. The geometry classifier 324 can also be a variable such as B. "intersection", set to true or false for a waiting group (and thus for each waiting element). This can also be used to trigger certain rules (e.g. to distinguish how a pedestrian crossing is treated when it is adjacent to an intersection and when it is not).

Geometry associator 326 may associate signals with paths through a process referred to as geometry association. Geometry mapping determines which signals apply to a path. This can answer questions like: "Is this traffic light close enough to this path to apply to it?" Is this traffic light the closest/most relevant of this type for this way? Is this sign intended for this way?” The light-way association can be difficult to separate from the rules, for example, it can be difficult to know if a light is for a left turn and then it makes sense green arrow that resolves the ambiguity. Also note that this analysis often benefits from considering all signals and paths together. For example, a traffic light offset to the right may apply to an ego path in one case where there is no other path, but not in another case where there is a path further to the right. Likewise, a traffic light offset to the right may apply to an ego path in one case where there is no other traffic light, but not in another case where there is a traffic light directly over the path. Therefore, the geometry association can take into account the entire scene, as well as signal states (or even original sensor data) if needed. For the same reason, the architecture may allow the geometry classification performed by geometry classifier 324 and the geometry association performed by geometry associator 326 to coexist and have access to waiting geometry, signals, and even original sensor data. In this sense, the process can map connections between signals such as traffic lights and signs and paths. In at least one embodiment, geometry mapping may first check whether a traffic light or sign is at all within a distance that allows connection to a path, and then, if so, the nearest (in a sense) traffic light or that find the closest (in a sense) sign of each applicable type and establish a priority order (e.g. a left-turn light has the highest priority for a left-turn, but the closest regular light also applies, although it has second priority). Note that the mapping can change instantaneously when a traffic light changes state and resolves some form of ambiguity.

The outputs of the waiting geometry fuser 322, the mapper 304, the signal state estimator 330, the geometry classifier 324, the geometry associator 326 and the other sensor data receiver 308 can be fed (as inputs) to the contention state solver 340. The contention resolver 340 may perform a process referred to as a contention resolution process. The goal of the contention state resolution process may be to provide a contention state (e.g., contention state 318) for the wait element 310 (and/or other wait elements). The contention state 318 of the wait element 310 may be an instruction to the traffic planner as to how the ego vehicle should yield or yield the right-of-way with respect to that wait element, as a matter of rule, expectation, formal, or informal convention or norm. In some non-limiting embodiments, the contention state 318 may not indicate what is actually happening in the right-of-way scenario, what is physically possible in the right-of-way scenario, or whether the ego vehicle can be forced to yield even though it is in the right-of-way scenario (e.g., B. an intersection priority scenario or a T-junction priority scenario) has the right of way. Rather, the contention state 318 may indicate what should happen according to the conventions. It may be the responsibility of the priority planner to actually implement the priority, in the sense that it checks what should happen (e.g., as encoded in contention state 318) whether the ego vehicle is actually in the able to stop and obey this instruction, and whether the other actors (e.g., the competitors in the give-way scenario) appear to be fulfilling their expected right-of-way obligations and taking appropriate action. In other words, the planner may determine that, even though the state is Right of Way, a competitor is not yielding the right-of-way (essentially capturing "honk appropriately") and decide to yield the right-of-way when it shouldn't. The priority planner can implement a priority behavior that analyzes all contentions in a queue group until all queue items in the group can be resolved collectively. All contentions in the waiting group can be satisfied together, ie the most restrictive contention can determine the expected right-of-way behavior of the ego vehicle. For example, if one state of the queue is "yield" and another is "stop at the entrance," the ego vehicle can stay at the entry line. The contention state 318 can be similar to the contention state 210 in 2 be. Various possible state values for contention state 318 are discussed in connection with contention state 210 .

The contention state resolver 340 process for resolving the contention state may be based on at least basic “traffic rules” (or “base rules”) that may vary from country to country, state to state, region to region, and the like. The base rules may be basic logical rules that apply to the waiting geometries (e.g., ego waiting geometry 312 and competitor waiting geometry 314) and signals after they are passed through a geometry classification process (e.g., via the geometry classifier 324) and the geometry association process (e.g. via geometry associator 326) have been reduced to the state of basic enumeration variables. After these reductions, each waiting element can have a well-defined set of geometry classes and signal states that apply to the waiting geometry. In addition to the base rules, the contention resolution process may also use "mapping rules". An association rule may include a wait element (e.g., wait element 310) and an array of suggestion pairs associated with the contention states.

Each proposition can be made dependent on any number of signal states (no signal state dependency is allowed either). The contention state and/or the associated proposition can be paired with the specified contention state if the proposition evaluates to true. The semantics of the mapping rule can be that each of the propositions is evaluated in turn and the first proposition that evaluates to true defines the contention state. The state already contained in the wait element can be regarded as the default contention state, which is selected if none of the propositions evaluate to true. In general, the constraint checker 342 of the contention state solver 340 may be responsible for performing such evaluations and linking propositions.

In at least one embodiment, the condition checker 342 may generally determine which signals are valid or invalid (or active or inactive) based at least on one or more sensed and/or determined environmental conditions. For example, some signs may change depending on weather conditions (e.g. rain, snow, fog, wind), time of day, day of week, presence of other signs (e.g. construction site signs may displace other signs), autonomous vehicle vehicle type (e.g. car vs. truck) etc. apply or not. For example, if there is a card rule that contains a constraint that only applies from time X to time Y, the constraint checker 342 can be used to determine and flag whether the card rule currently applies.

A card rule matcher 346 is generally responsible for performing a card rule check responsible for the matching process. The card rule matching process can take the card rules and match their components (or data components) to the items that have been determined to actually exist and create wait items in the process. For example, card rule matcher 346 may resolve signals and other inputs determined to be true by constraint checker 342 into one or more card rules. For example, the map rule matcher 346 may determine that one or more map rules apply when an associated traffic light is green, while the map rule matcher 346 may not determine that the map rule(s) apply when the traffic light is red (but determine that another rule applies). At a minimum, these determinations may be based on the map rule matcher 346 knowing the condition(s) that the condition checker 342 has determined actually apply (e.g., the traffic light state).

To increase the ability to benefit from mutual exclusion conditions and generally make a holistic decision, the map rule matching process performed by the map rule matcher 346 may begin with a process that establishes a match between paths, queuing geometry, and signals in the map and the paths, waiting geometry and signals established by the constraint checker 342 as actually present. Note that in some cases these entities may initially come from a map (e.g., the wait element engine 300 may be configured to receive the lane graph from a map and consider a map rule with one of the same paths and checks if it matches) so that it can perform the matching "by identifiers" already specified during the merging of lane graphs, waiting geometry or signals. However, to increase flexibility and generality, the card rule matcher 346 may perform the matching without using information from a card. For example, the map rule matcher 346 can take a lane graph from live perception (e.g., via the waiting geometry perceiver 302) and apply a map rule from the map that associates a traffic light with the ego path, thus avoiding architectural complications , which result from card identification numbers having to be passed all the way through the lane graph, waiting geometry and signals. If a trail, piece of wait geometry, or signal is actually derived from a map, its geometry should be identical (or nearly identical if successfully merged/blended) so that matches should be recovered correctly. The match can also match items that are not exactly identical. For example, roughly similar shapes and placements of left turns can be expected to match (again, note that if there are two parallel left turns, the match will benefit from being viewed together). So this process can essentially perform a match that matches the “map scene” to the “actual scene” determined by any combination of live perception and map localization. The result can be a 1:1 correspondence between a subset of the actual features and a subset of the map features.

In various embodiments, each map rule may result in an exit waiting element (e.g., waiting element 310 or a precursor thereof) by matching all entities and resolving all signals. Many map rules may contain a valid ego path (since many competitive states depend on it and make no sense without it). If this ego path does not find a match, there can be several reasons. If the localization fails in a known manner, the mapping rules cannot be used and the failed localization can be handled in other ways. However, if the localization is inaccurate, it could cause the ego path to mismatch. Another possibility is that the ego path is inaccurate either in the map or in the actual scene. Another possibility is that the trail is too far away or obscured by live perception. In this case, a conservative approach can force the ego path into the scene. Because of this, ego paths from map rules that did not find a match can be added along with their wait geometry. The same procedure can also be applied to competitor paths. In fact, the entire set of queuing elements from the card rules could be considered separately, with the exception of the signals, although the correspondence may be important when considering the merging of the card rules with the base rules performed by the queuing element fuse 348. The signals, on the other hand, may need to be reconciled to resolve their state. Any mismatched signal state can be set to the Unknown state, and the proposal in the mapping rules can take this possibility into account and assign the appropriate contention state. This can e.g. B. mean that the contention state is set to stop at the entrance when the state of an individual traffic light is unknown. This can also mean that one of several synchronized traffic lights is used to resolve the same waiting element, switching to stopping at the entrance only when all traffic lights are unknown. In still other cases, this could mean that when turning left into oncoming traffic, when the traffic light with the green turn arrow is not visible, but a green circle can be seen and it is known that the turn is in, the Yield at Competing Point approach is resorted to case is always allowed (although it is not known whether it is protected or not). This example is aggressive, but the design offers high flexibility without high complexity (another, less flexible option would be to list possible states and number all possible combinations and look for the most restrictive contention state of all possibilities).

The base rule parser 344 is generally responsible for performing a base rule parsing process, which can use the suggestion rules derived in two ways. First, the suggestion rules can be derived from the base rules about base state variable estimation, geometry classification, and geometry mapping. Second, the suggestion rules can be derived from map rules. For example, base rule parser 344 may resolve signals and other input determined to be true by constraint checker 342 into one or more base rules. In at least one embodiment, base rule parser 344 may operate similarly to map rule matcher 346, but applies general or universal rules and conditions for driving that are independent of historical or observed behavior of vehicles at the location of the right-of-way scenario.

While the basic rules may vary from country to country, state to state, region to region, etc., they can apply uniformly to any give-way scenario as long as the appropriate conditions are met. In contrast, map rules may apply rules and conditions to driving based at least on the historical or observed behavior of vehicles at a right-of-way scenario location or similar right-of-way scenario locations. In at least one embodiment, the map rules may be encoded in the map data and applied at least based on the location of the autonomous vehicle on a map. However, the basic rules can apply regardless of the location and location of the right-of-way scenario. By providing the base rule parser 344, wait items can be generated even when no map data is available or cannot be applied or located to the current right-of-way scenario. For example, if the map rule matcher 346 is unable to determine one or more components and/or elements of a wait element, the base rule parser 344 may fill in any gaps or vice versa. Thus, waiting elements can be generated entirely from map data, entirely from perception data, or from a combination of both types of data.

The queuing element merger 348 is generally responsible for merging or combining data corresponding to the parsed base rules and the matched card rules to resolve the contention state 318 . For example, in at least one embodiment, base rule parser 344 and map rule matcher 346 may each generate corresponding wait elements and/or elements and/or components thereof. The wait element merger 348 can merge any of these various aspects to form the wait element 310 . In at least one embodiment, one or more aspects of base rule parser 344 and map rule matcher 346 may compete with one another. For example, the same field or data item can have different values. The wait element merger 348 may identify and/or detect such contentions to determine one or more resolved values for the wait element 310 . In at least one embodiment, the determinations of the card rule matcher 346 may generally take precedence since they are based on locality and perception data is not always reliable. This may be useful, for example, when the base rule parser 344 is unable to derive a relevant rule that should be applied. For example, if there is no sign or other visual indicator that no left turns are allowed at an intersection, base rule parser 344 may not apply an appropriate base rule, even if that rule is followed by convention. However, the map rule matcher 346 may at least apply the rule based on the observation that the rule applies to historical autonomous vehicle trips through the intersection. However, it is possible that there may be transient or new signals that do not exist or have a sufficiently high level of confidence (e.g. due to inconsistent or too few observations, outdated observations, etc.) to include in the map data. The base rule parser 344 may be used to account for such scenarios (e.g., for construction signs, electronic signs, or other temporary or transient signals, corresponding determinations by the base rule parser 344 that are at least based on being associated or associated with these signal types) when resolving data to wait element 310.

Although in 3 not shown, a queuing element engine may optionally include an obstacle-in-the-path analyzer that performs obstacle-in-the-path analysis (OIPA). An OIPA can associate actors with paths and waiting geometries using the lane graph, waiting geometry, and Semantic Motion Segmentation (SMS) output for obstacle perception. The OIPA can be performed by rendering the pathways and areas into an index image and then projecting the polygonal shapes of the actors into the image and integrating the degree of overlap.

Additionally, the wait element engine may include an occlusion analyzer. The occlusion analysis can add an occlusion understanding to the OIPA results using a lane graph and obstacle perception outputs such as SMS and a depth map, and detect segments in the lane graph that could be occluding an unseen actor. As a result, the right-of-way planner can take into account not only visible but also invisible actors. A competing path can be provided with an expected speed limit, and obscured parts of that path can then be used to insert an unseen actor with that maximum speed at the closest obscured location where it can currently be, with appropriate caveats for expected conventions ( e.g., where the first stop, first priority condition of competition is present, an unseen vehicle well beyond its entry zone and entry line should not be expected to enter at top speed if that would prevent it from stopping at its entry line, while at a yield point it can be assumed that it could enter at top speed.) With this information the yield planner can then take into account non-visible vehicles and generate correct behavior e.g. B. Slowing down, waiting or slowly rolling forward until hidden areas can be revealed or until the traffic light turns green in the case of a right turn on red.

The wait element engine 300 may additionally perform a who stopped first analysis. The who-stopped-first analysis can use the motion analysis and OIPA results to determine which actor stopped first to support the first-stop-first-priority contention state. The OIPA results can be used to determine if an actor has entered the inner surface, is within its entry zone, and has reached the line of entry. Motion can also be used to determine if the actor is in motion, has been stopped but has recently moved, or may be a parked vehicle. Further analysis of the wait item engine 300 may include a who goes first analysis. A who-goes-first analysis can be a machine-learned analysis that estimates a competitive state for each competitor. Such an analysis can be trained by many examples of future trajectories, in which it can be determined whether an actor has passed the competition point in front of the ego vehicle or vice versa. Knowing the probability of who comes first can be an indication of expected returns.

As already mentioned, the output of the waiting element 310 of the waiting element engine 300 can be forwarded to a priority planner of the ego vehicle. The priority planner can implement a priority behavior for the ego vehicle given the waiting elements with resolved contention, the OIPA results with occlusion, the outputs "who stopped first", "who drives first", and obstacle perception. If necessary and possible, the right-of-way planner can cause the ego vehicle to give way and monitors the granting of the right-of-way to the other actors when they give way. The analysis of the right-of-way planner can be performed to predict what would happen if the ego vehicle continues on the required path. If the ego vehicle is moving forward and is to yield the right-of-way, the ego vehicle can resolve the competition before competitors who have the right-of-way are disrupted (e.g., forced to change their behavior). When giving way, care can also be taken to ensure that the ego vehicle does not cause the other road users to behave in a way that differs from their preferred or intended behavior.

Referring now to Figures 4-6, each block of methods 400-600 and other methods described herein comprises a computational process that may be performed using any combination of hardware, firmware, and/or software. For example, various functions may be performed by a processor executing instructions stored in memory. The methods may also be stored on computer storage media as computer-usable instructions. The methods can be implemented by a standalone application, service, or hosted service (standalone or in combination with another hosted service) or a plugin for another product, to name a few. In addition, the methods 400-600 are exemplified with respect to the wait item engine 300 of FIG 3 described. However, these methods may additionally or alternatively be performed by any system or combination of systems, including but not limited to the systems described herein.

4 FIG. 4 is a flow chart depicting a method 400 for encoding give-way scenarios for autonomous vehicles (e.g., a ego vehicle) according to some embodiments of the present disclosure. The method may be performed via a wait item engine such as, but not limited to, the wait item engine 300 of FIG 3 . In block B402, the method 400 includes recognizing and/or identifying an upcoming priority scenario. The give way scenario may correspond to a give way scenario at an intersection (or an intersection) or a give way scenario at an intersection. The right-of-way scenario may involve an autonomous vehicle (e.g., an ego vehicle) and one or more competitors.

Geometry data can be received in block B404. In some embodiments, the geometry data may be received in response to detecting the give-way scenario. The geometry data may include geometry recognition data. For example, the wait geometry perceiver 302 of the wait element engine 300 may receive real-time geometry data generated by one or more sensors of the autonomous vehicle.

In block B406 signal data can be received. In some embodiments, the signaling data may be received in response to detecting the give-way scenario. The signal data may include signal perception data. For example, the signal perceiver 306 of the queuing element engine 300 may receive real-time signal data generated by the one or more sensors of the autonomous vehicle.

Map data can be received in block B408. In some embodiments, the map data may be received in response to detecting the give-way scenario. For example, the mapper 304 of the queuing element engine 300 may receive map data.

Lane graph data may be received in block B410. In some embodiments, the lane graph data may be received in response to detecting the give-way scenario. The lane graph data may include one or more ego paths and one or more competitor paths for one or more competitors of a right-of-way scenario. For example, the lane recorder 350 of the waiting element engine 300 may receive the ego paths 352 for a ego vehicle (e.g., the autonomous vehicle) and the competitor paths 354 for the competitor(s) associated with the right-of-way scenario.

In block B412, a geometry for an ego path of the one or more ego paths may be determined. The geometry for the ego path can be determined using the geometry data, the map data, the signal data and/or the lane graph data. As such, the waiting geometry perceiver 302, the mapper 304, the signal perceiver 306, the lane recorder 350, or any combination thereof may be generally responsible for determining the geometry of the ego path. In some embodiments, the waiting element engine 300's waiting geometry fuser 322, waiting element engine 300's geometry classifier 324, waiting element's geometry associator 326, waiting element's signal fuser 328, waiting element engine 300's signal state estimator 330 of the waiting element engine 300 or any combination thereof contribute to the determination of the geometry for the ego path.

Also in block B412, a geometry for a competitor path of the one or more competitor paths of the one or more competitors may be determined. Similar to the ego path, the geometry for the competitor path may be determined from the geometry data, the map data, the signal data, and/or the lane graph data. As such, the waiting geometry observer 302, the mapper 304, the signal observer 306, the lane recorder 350, or any combination thereof may be generally responsible for determining the geometry of the competitor's path. In some embodiments, the waiting element engine 300's waiting geometry fuser 322, waiting element engine 300's geometry classifier 324, waiting element's geometry associator 326, waiting element's signal fuser 328, waiting element engine 300's signal state estimator 330 of the queuing engine 300, or any combination thereof, contribute to determining the geometry for the competitor path. In some embodiments, in block 414, a geometry is determined for a context of the paths and/or the right-of-way scenario.

In block B414 the geometries for the ego path and the competitor path can be encoded. The geometry of the ego path can be encoded in an ego waiting geometry (e.g. the ego waiting geometry 312 of 3 ). The geometry of the competitor path may be encoded in a competitor queuing geometry (e.g., the competitor queuing geometry 314 in 3 ). In at least one embodiment, the geometry for the context of the ways and/or the right of way geometry is encoded in a context queuing geometry (e.g., the context queuing geometry 316 of the 3 ).

In block B416, a contention state between the ego path and the competitor path may be determined based at least on the determined geometries. Various embodiments for determining a contention state are provided in connection with at least the wait item engine 300, the method 500 of FIG 5 and/or the method 600 of 6 discussed. However, it is briefly mentioned here that the contention state solver 340 of the wait element engine 300 may be generally responsible for determining and/or resolving a contention state for the ego-path and the competitor-path. The contention state may be encoded (e.g., the contention state 318 of the 3 ).

In block B418 a wait element data structure (or data object) may be created, e.g. B. the waiting element 310 of 3 . The wait element data structure may contain at least one of the following elements: the geometry for the ego path, the geometry for the competitor path, and the contention state. The wait element can also contain a context wait geometry.

At B420, the wait item may be communicated to a priority planner for the autonomous vehicle.

5 FIG. 5 is a flow diagram depicting a method 500 for encoding give-way scenarios for autonomous vehicles (eg, a ego vehicle) according to some embodiments of the present disclosure. The method may be performed via a wait item engine such as, but not limited to, the wait item engine 300 of FIG 3 . The method 500 includes, at block B502, perceiving a wait geometry for an ego path and a competitor path. A geometry perceiver (e.g., geometry perceiver 302 of wait element engine 300) may perceive the wait geometries. The waiting geometries may be perceived in response to the detection and/or identification of a right-of-way scenario for an ego vehicle (an autonomous vehicle). The ego path and the competitor path may be perceived by a lane recorder (e.g., lane recorder 350 of queuing element engine 300).

In block B504, one or more signals for the ego/competitor pathways may be perceived. A signal perceiver (e.g., signal perceiver 306) may perceive the signals for the paths.

In block B506 the waiting geometries can be merged with map data. A wait geometry blender (e.g., the wait geometry blender 322 of the wait element engine 300) may merge the wait geometries with the map data.

In block B508, the merged wait geometries can be classified. A geometry classifier (e.g. geometry classifier 324 of wait element engine 300) may classify the wait geometries.

In block B510 the merged waiting geometries can be assigned. A geometry associator (e.g., geometry associator 326 of wait element engine 300) may associate the wait geometries.

In block B512 the signals can be merged with the map data. A signal merger (e.g., the signal merger 328 of the cue engine 300) may merge the signals with the map data.

In block B514, the states of the merged signals can be estimated. A signal state estimator (e.g., signal state estimator 330 of wait element engine 300) may estimate the state of the merged signals.

In block B516, a contention condition between the ego path and the competitor path may be resolved. Various embodiments of resolving a contention state are provided at least in connection with the method 600 of FIG 6 discussed. However, it should be briefly noted that a contention state solver (e.g., contention state solver 340 of wait element engine 300) may resolve the contention state between the ego path and the competitor path.

A wait element may be created in block B518. For example, the wait item engine 300 may generate the wait item 310 .

In block B520, the queuing element may be provided to a system (e.g., a priority planner) that provides guidance services to an autonomous vehicle (e.g., the ego vehicle).

6 FIG. 6 is a flow diagram depicting a method 600 for resolving a lane contention for vehicles, in accordance with some embodiments of the present disclosure. The method can be performed via a contention solver such as e.g. B., but not limited to the contention state solver 340 of FIG 3 . Various inputs can be provided to the contention state solver to resolve the contention state between the two paths (e.g., an ego path and a competitor path). As in 3 For example, as shown, map data as well as merged, classified, and/or associated geometries (e.g., waiting geometries) for the two paths may be communicated to the contention solver 340 . In addition, fused signals (including estimated signal states) can be submitted to the contention state solver. Various other sensor data (from autonomous vehicle sensors) may be provided to the contention solver. The various blocks of the method 600 may use any of this input data.

The method 600 includes checking a state of the contention state at block B602. A constraint checker (e.g., the constraint checker 342 of the contention state solver 340) may be used to check the constraint of the contention state.

In block B604, one or more base rules may be parsed. A base rule parser (e.g., base rule parser 344 of contention state solver 340) may parse the base rules.

In block B606, one or more card rules may be compared to the contention state. A card rule matcher (e.g., card rule matcher 346 of contention state solver 340) may match contention state to one or more card rules.

In block B608, the various data structures (e.g., wait geometries and contention state) may be merged into one wait element. A wait element merger (e.g., wait element merger 348 of contention solver 340) may merge the data structures.

In block B610, the merged data structures (e.g., wait geometries and a resolved contention state) may be merged into a wait element (e.g., wait element 310 of the 3 ) to be packed.

Exemplary autonomous vehicle

7A FIG. 7 is an illustration of an example of an autonomous vehicle 700, in accordance with some embodiments of the present disclosure. The autonomous vehicle 700 (alternatively referred to herein as “vehicle 700”) may be, without limitation, a passenger vehicle, such as an automobile. B. Car, truck, bus, first responder vehicle, shuttle, electric or motorized bicycle, motorcycle, fire engine, police vehicle, ambulance, boat, construction vehicle, underwater vehicle, drone , a vehicle coupled to a trailer, and/or another type of vehicle (e.g., an unmanned vehicle and/or a vehicle that accommodates one or more passengers). Autonomous vehicles are generally described in terms of automation levels defined by the National Highway Traffic Safety Administration (NHTSA), a division of the US Department of Transportation, and the Society of Automotive Engineers (SAE) "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806, published June 15, 2018, Standard No. J3016-201609, published September 30, 2016, and previous and future versions of this standard). The vehicle 700 may be capable of performing functions according to one or more of levels 3 through 5 of the autonomous driving levels. The vehicle 700 may have functionality according to one or more of the Level 1-Level 5 autonomous driving tiers. For example, depending on the embodiment, the vehicle 700 can offer driver assistance (level 1), partial automation (level 2), conditional automation (level 3), high automation (level 4) and/or full automation (level 5). The term “autonomous” as used herein may include any and/or all types of autonomy for the vehicle 700 or other machine, such as a vehicle. e.g. fully autonomous, highly autonomous, conditionally autonomous, semi-autonomous, supportive autonomy, semi-autonomous, primarily autonomous, or some other designation.

Vehicle 700 may include components such as a chassis, vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. The vehicle 700 may include a propulsion system 750, such as. B. an internal combustion engine, a hybrid electric power plant, a pure electric motor and / or another type of drive system. The propulsion system 750 may be connected to a powertrain of the vehicle 700, which may include a transmission, to enable the vehicle 700 to be propelled. The powertrain 750 may be controlled in response to receiving signals from the throttle/accelerator pedal 752 .

A steering system 754, which may include a steering wheel, may be used to steer the vehicle 700 (e.g., along a desired path or route) when the propulsion system 750 is operational (e.g., when the vehicle is in movement is). The steering system 754 can receive signals from a steering actuator 756 . The steering wheel can optionally be used for full automation (level 5).

The brake sensor system 746 may be used to actuate the vehicle brakes in response to receiving signals from the brake actuators 748 and/or brake sensors.

The controller(s) 736, which include one or more System-on-Chips (SoCs) 704 ( 7C ) and/or GPU(s) may send signals (e.g., representative of commands) to one or more vehicle 700 components and/or systems. For example, the controller(s) may send signals to actuate the vehicle brakes via one or more brake actuators 748, to actuate the steering system 754 via one or more steering actuators 756, and to actuate the propulsion system 750 via one or more throttle/acceleration controllers 752. The controller 736 may include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and issue operational commands (e.g., signals representing commands) to enable autonomous driving and/or or to assist a human driver in driving the vehicle 700 . The controller 736 may include a first controller 736 for autonomous driving functions, a second controller 736 for functional safety functions, a third controller 736 for artificial intelligence (e.g. computer vision) functions, a fourth controller 736 for infotainment functions, a fifth controller 736 for redundancy under emergency conditions and/or other controllers. In some examples, a single controller 736 may perform two or more of the above functions, two or more controllers 736 may perform a single function, and/or any combination thereof.

The controller 736 may provide the signals to control one or more components and/or systems of the vehicle 700 in response to sensor data received from one or more sensors (e.g., sensor inputs). The sensor data may include, for example and without limitation, global navigation satellite system sensor(s) 758 (e.g., global positioning system sensor(s)), RADAR sensor(s) 760, ultrasonic sensor(s) 762, 764 LIDAR Sensor(s), Inertial Measurement Unit (IMU) Sensor(s) 766 (e.g., Accelerometer, Gyroscope(s), Magnetic Compass(s), Magnetometer, etc.), Microphone(s) 796, Stereo Camera(s). ) 768, wide-angle camera(s) 770 (e.g., fisheye cameras), infrared camera(s) 772, surround camera(s) 774 (e.g., 360-degree cameras), long-range and/or mid-range camera(s). ) 798, speed sensor(s) 744 (e.g. for measuring the speed of the vehicle 700), vibration sensor(s) 742, steering sensor(s) 740, brake sensor(s) (e.g. as part of the brake sensor system 746) and /or other sensor types.

One or more controllers 736 may receive inputs (e.g., in the form of input data) from an instrument cluster 732 of the vehicle 700 and outputs (e.g., in the form of output data, display data, etc.) via a human-machine interface (HMI ) display 734, an audible indicator, a speaker, and/or via other vehicle 700 components. The outputs can include information such as vehicle speed, RPM, time, map data (e.g. the HD map 722 of the 7C ), location data (e.g., the location of the vehicle 700, e.g., on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and the status of objects, such as from/ perceived by the controller(s) 736, etc. For example, the HMI display 734 may display information about the presence of one or more objects (e.g., a road sign, a warning sign, a changing traffic light, etc.) and/or information about maneuvers that the vehicle has performed, is performing, or is performing will perform (e.g., change lanes now, exit 34B in two miles, etc.).

The vehicle 700 further includes a network interface 724 that may use one or more wireless antenna(s) 726 and/or modem(s) to communicate over one or more networks. For example, network interface 724 may be capable of communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, and so on. The wireless antenna(s) 726 can also enable communication between objects in the vicinity (e.g. vehicles, mobile devices, etc.) via local area networks such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc. and /or Enable Low Power Wide Area Networks (LPWANs) such as LoRaWAN, SigFox, etc.

7B 12 is an example of camera positions and fields of view for the autonomous vehicle 700 of FIG 7A , according to some embodiments of the present disclosure. The cameras and respective fields of view represent an example and are not to be considered limiting. For example, additional and/or alternative cameras may be included and/or the cameras may NEN be arranged at different locations of the vehicle 700.

The camera types for the cameras may include, but are not limited to, digital cameras, which may be adapted for use with the vehicle 700 components and/or systems. The camera(s) can be operated with security level B (ASIL) and/or another ASIL. Depending on the embodiment, the camera types can have any image recording rate, e.g. 60 frames per second (fps), 120 fps, 240 fps, etc. The cameras can use rolling shutter, global shutter, another type of shutter, or a combination of these. In some examples, the color filter array may be a red-clear-clear-clear color filter array (RCCC), a red-clear-clear-blue color filter array (RCCB), a red-blue-green-clear color filter array (RBGC), a foveon X3 color filter array, a Bayer sensor color filter array (RGGB), a monochrome sensor color filter array, and/or another type of color filter array. In some embodiments, clear pixel cameras, such as B. cameras with an RCCC, an RCCB and / or an RBGC color filter array can be used to increase the light sensitivity.

In some examples, one or more of the cameras may be used to run advanced driver assistance systems (ADAS) (e.g., as part of a redundant or failsafe design). For example, a multifunction mono camera can be installed to provide functions such as Lane Departure Warning, Traffic Sign Assist and Intelligent Headlight Control. One or more of the cameras (e.g. all cameras) can simultaneously record and deliver image data (e.g. video).

One or more of the cameras can be mounted in a mounting assembly, e.g. a custom (3D printed) assembly, to eliminate stray light and reflections from inside the vehicle (e.g. reflections from the dashboard reflected in the windshield mirrors) that could affect the camera’s ability to capture image data . When assembling outside mirrors, the outside mirrors can be custom-printed in three dimensions so that the camera mounting plate corresponds to the shape of the outside mirror. In some examples, the cameras may be integrated into the outside mirror. With side cameras, the cameras can also be integrated into the four pillars at each corner of the cabin.

Cameras with a field of view that includes portions of the environment in front of the vehicle 700 (e.g., forward-looking cameras) can be used for the environment view to help identify forward-facing paths and obstacles, as well as using a or multiple controllers 736 and/or control SoCs to provide information critical to constructing an occupancy grid and/or determining preferred vehicle paths. Forward-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. Forward-facing cameras can also be used for ADAS functions and systems such as Lane Departure Warning (LDW), Autonomous Cruise Control (ACC), and/or other functions such as Traffic Sign Recognition.

A variety of cameras can be used in a forward-facing configuration, e.g. B. a monocular camera platform with a color CMOS (complementary metal oxide semiconductor) imager. Another example is the wide-angle camera(s) 770, which can be used to perceive objects coming into view from the periphery (e.g., pedestrians, crossing vehicles, or bicycles). Although in 7B only one wide-angle camera is shown, the vehicle 700 may be equipped with any number of wide-angle cameras 770 . Additionally, the wide-angle camera(s) 798 (e.g., a wide-angle stereo camera pair) can be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. The long-range camera(s) 798 can also be used for object detection and classification, as well as basic object tracking.

One or more stereo cameras 768 may also be included in a forward-facing configuration. The stereo camera(s) 768 may include an integrated control unit that includes a scalable processing unit that can provide programmable logic (FPGA) and a multi-core microprocessor with integrated CAN or Ethernet interface on a single chip. With such a unit, a 3-D map of the vehicle's surroundings can be created, including a distance estimate for all points in the image. An alternative stereo camera(s) 768 may include a compact stereo vision sensor(s) that includes two camera lenses (one each on the left and right sides) and an image processing chip that measures the distance between the vehicle and the Measure target object and use the generated information (e.g. metadata) to activate autonomous emergency braking and lane departure warning functions. In addition or as an alternative to those described here Stereo cameras, other types of stereo cameras (768) can also be used.

Cameras with a field of view that includes portions of the environment to the side of the vehicle 700 (e.g., side cameras) may be used for the environment view and provide information used to create and update the occupancy grid and generate side impact collision alerts. For example, the surround camera(s) 774 (e.g. four surround cameras 774 as in 7B shown) are positioned on the vehicle 700 . Surround camera(s) 774 may include wide angle camera(s) 770, fisheye camera(s), 360 degree camera(s), and/or the like. For example, four fisheye cameras can be mounted on the front, rear and sides of the vehicle. In an alternative arrangement, the vehicle may use three surround cameras 774 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., a front-facing camera) as the fourth surround camera .

Cameras with a field of view that includes portions of the environment behind the vehicle 700 (e.g., backup cameras) may be used for parking assist, surround view, rear collision alerts, and creating and updating the occupancy grid. A variety of cameras may be used, including but not limited to cameras that are also suitable as front-facing camera(s) (e.g., long-range and/or mid-range camera(s) 798, stereo camera(s) 768 , infrared camera(s) 772 etc.) as described here.

7C FIG. 7 is a block diagram of an example system architecture for the example autonomous vehicle 700 of FIG 7A , according to some embodiments of the present disclosure. It should be understood that these and other arrangements described herein are presented as examples only. Other arrangements and elements (e.g., engines, interfaces, functions, arrangements, groupings of functions, etc.) may be used in addition to or in place of those illustrated, and some elements may be omitted entirely. In addition, many of the elements described herein are functional units that can be implemented as single or distributed components, or in conjunction with other components, and in any suitable combination and location. Various functions described herein performed by devices may be performed by hardware, firmware, and/or software. For example, various functions may be performed by a processor executing instructions stored in memory.

All components, features and systems of the vehicle 700 in 7C are shown connected via bus 702 . Bus 702 may include a controller area network (CAN) data interface (alternatively referred to herein as “CAN bus”). A CAN bus can be a network within the vehicle 700 that is used to control various features and functions of the vehicle 700, e.g. to operate brakes, acceleration, brakes, steering, windshield wipers, etc. A CAN bus can be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g. a CAN ID). The CAN bus can be read to determine steering wheel angle, vehicle speed, engine speed (RPM), button positions and/or other vehicle status indicators. The CAN bus can be ASIL B compliant.

Although bus 702 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and/or Ethernet can also be used in addition or as an alternative to the CAN bus. Although bus 702 is represented by a single line, this is not intended to be limiting. For example, there may be any number of buses 702, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and/or one or more other types of buses with a different protocol. In some examples, two or more buses 702 can be used to perform different functions and/or they can be used for redundancy. For example, a first bus 702 may be used for collision avoidance functionality and a second bus 702 for actuation control. In each example, each bus 702 can communicate with any component of the vehicle 700, and two or more buses 702 can communicate with the same components. In some examples, each SoC 704, each controller 736, and/or each computer within the vehicle may have access to the same input data (e.g., inputs from sensors of the vehicle 700) and be connected to a common bus, such as the CAN bus .

The vehicle 700 may include one or more controllers 736 as described herein with respect to FIG 7A are described. The 736 controller can be used for a variety of functions. The controller 736 may be coupled to the various other components and systems of the vehicle 700 and may be used to control tion of the vehicle 700, the artificial intelligence of the vehicle 700, the infotainment for the vehicle 700 and/or the like.

The vehicle 700 may include one or more systems on a chip (SoC) 704 . SoC 704 may include CPU(s) 706, GPU(s) 708, processor(s) 710, cache(s) 712, accelerators 714, memory 716, and/or other components and features not shown. The SoC(s) 704 can be used to control the vehicle 700 in a variety of platforms and systems. For example, the SoC(s) 704 may be combined in a system (e.g., the vehicle's 700 system) with an HD card 722 that may receive map updates and/or updates over a network interface 724 from one or more servers (e.g., B. the server(s) 778 the 7D ) can get.

The CPU(s) 706 may comprise a CPU cluster or CPU complex (also referred to herein as "CCPLEX"). The CPU(s) 706 may include multiple cores and/or L2 caches. For example, in some embodiments, CPU(s) 706 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU(s) 706 may include four dual-core clusters, with each cluster having a dedicated L2 cache (e.g., a 2MB L2 cache). The CPU(s) 706 (e.g., the CCPLEX) can be configured to support concurrent operation of clusters such that any combination of CPU(s) 706 clusters are active at any given time can be.

The CPU(s) 706 may implement power management functions that include one or more of the following features: individual hardware blocks may be automatically clocked when idle to conserve dynamic power; each core clock can be controlled when the core is not actively executing commands due to the execution of WFI/WFE commands; each core can be independently current controlled; each core cluster can be independently clocked if all cores are clocked or power-driven; and/or each core cluster can be independently current controlled if all cores are current controlled. The CPU(s) 706 may also implement an advanced power state management algorithm where allowable power states and expected wake up times are set and the hardware/microcode determines the best power state for the core, cluster and CCPLEX. The processor cores may support simplified power state entry sequences in software, with the work offloaded to microcode.

The GPUs 708 may be an integrated GPU (alternatively referred to herein as “iGPU”). The GPU(s) 708 may be programmable and efficient for parallel workloads. The GPU(s) 708 may use an extended tensor instruction set in some examples. The GPU(s) 708 may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96 KB of storage capacity) and two or more of the streaming Microprocessors can share an L2 cache (e.g. a 512 KB L2 cache). In some embodiments, the GPU (or GPUs) 708 may include at least eight streaming microprocessors. The GPU(s) 708 may use one or more application programming interfaces (API(s)) for computations. Additionally, the GPU(s) 708 may use one or more parallel computing platforms and/or programming models (e.g., NVIDIA's CUDA).

The graphics processor(s) 708 can be optimized for best performance in automotive and embedded use cases. The GPU(s) 708 may be fabricated on a fin field effect transistor (FinFET), for example. This is not meant to be limiting, however, and the GPU(s) 708 may be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor may contain a number of mixed-precision cores divided into multiple blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can have 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit and/or a 64 KB register file. In addition, the streaming microprocessors can include independent parallel integer and floating point data paths to enable efficient execution of workloads with a mix of computation and addressing computation. The streaming microprocessors may include an independent thread scheduling facility to allow finer granular synchronization and collaboration between parallel threads. The streaming microprocessors may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

Graphics processor(s) 708 may include high bandwidth memory (HBM) and/or a 16GB HBM2 memory subsystem to provide peak memory bandwidth of about 900GB/second in some examples. In some examples, a synchronous graphics random access memory (SGRAM), such as a double data rate synchronous graphics random access memory type 5 (GDDR5), may be used in addition to or as an alternative to HBM memory.

The GPU(s) 708 may include unified memory technology with access counters to allow more accurate allocation of memory pages to the processor that accesses them most often, thereby improving the efficiency of the memory areas shared between the processors. In some examples, address translation services (ATS) support may be used to allow GPU(s) 708 to access CPU(s) 706 page tables directly. In such examples, if the memory management unit (MMU) of the GPU(s) 708 encounters an error, an address translation request may be communicated to the CPU(s) 706. In response, the CPU(s) 706 may look up the virtual-physical mapping for the address in its page tables and transmit the translation back to the GPU(s) 708. Thus, unified memory technology can enable a single, unified virtual address space for the memory of both the CPU(s) 706 and the GPU(s) 708, thereby enabling the programming of the GPU(s) 708 and the porting of applications to the Simplify GPU(s) 708.

In addition, the GPU(s) 708 may include an access counter that tracks the number of times the GPU(s) 708 have accessed the memory of other processors. The hit counter can help move memory pages to the physical memory of the processor that is accessing the pages most frequently.

The SoC(s) 704 may include any number of cache(s) 712, including those described herein. The cache(s) 712 may include, for example, an L3 cache available to both the CPU(s) 706 and the GPU(s) 708 (e.g., shared with both the /the CPU(s) 706 as well as the GPU(s) 708). The cache(s) 712 may include a write-back cache that may track the states of lines, e.g. B. by using a cache coherency protocol (e.g. MEI, MESI, MSI, etc.). The L3 cache can be 4MB or larger depending on the embodiment, although smaller cache sizes can also be used.

The SoC(s) 704 may include an arithmetic logic unit(s) (ALU(s)) that are used in performing processing related to one of the various tasks or operations of the vehicle 700 - such as processing DNNs - can be used. In addition, the SoC(s) 704 may include a floating point unit(s) (FPU(s)) - or other math coprocessor or numeric coprocessor type - for performing math operations within the system. For example, the SoC(s) 104 may include one or more FPUs that are integrated into a CPU(s) 706 and/or GPU(s) 708 as execution units.

The SoC(s) 704 may include one or more accelerators 714 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC(s) 704 may include a hardware acceleration cluster, which may include optimized hardware accelerators and/or large on-chip memory. The large on-chip memory (e.g. 4MB SRAM) can enable the hardware acceleration cluster to accelerate neural networks and other calculations. The hardware acceleration cluster may be used to complement the GPU(s) 708 and offload some tasks from the GPU(s) 708 (e.g., to free up more cycles of the GPU(s) 708 to perform other tasks). For example, the accelerator(s) 714 may be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to be accelerated. As used herein, the term "CNN" can encompass all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (eg, for object detection).

The accelerator(s) 714 (e.g., hardware acceleration cluster) may include a deep learning accelerator (DLA). The DLA(s) may include one or more Tensor Processing Units (TPUs) that may be configured to provide an additional tens of trillion operations per second for deep learning applications and inferencing. The TPUs can be accelerators configured and optimized to perform image processing functions (e.g. for CNNs, RCNNs, etc.). The DLA(s) may also be optimized for a particular set of neural network types and floating point operations, and for inference. The design of the DLA(s) can offer more performance per millimeter than a general purpose GPU and far exceeds the performance of a CPU. The TPU(s) can perform multiple functions, including a single instance Convolution function z. B. INT8, INT16 and FP16 data types are supported for both features and weights as well as post-processor functions.

The DLA(s) can quickly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for a variety of functions including, without limitation: a CNN for object identification and detection using data from camera sensors; a CNN for distance estimation using data from camera sensors; a CNN for emergency vehicle detection and identification and detection using data from microphones; a CNN for facial recognition and vehicle owner identification using data from camera sensors; and/or a CNN for security-related events.

The DLA(s) can perform any function of the GPU(s) 708, and through use of an inference accelerator, for example, a developer can use either the DLA(s) or the GPU(s) 708 for any function. For example, the developer may concentrate the processing of CNNs and floating point operations on the DLA(s) and leave other functions to the GPU(s) 708 and/or other accelerators 714.

The accelerator(s) 714 (e.g., hardware acceleration cluster) may include a programmable image processing accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. The PVA(s) can be developed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and/or augmented reality (AR) and/or virtual reality (VR) applications . The PVA(s) can provide a balance between performance and flexibility. For example, and without limitation, each PVA may include any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and/or any number of vector processors.

The RISC cores can interact with image sensors (e.g. the image sensors of one of the cameras described here), image signal processors and/or the like. Each of the RISC cores can contain any amount of memory. The RISC cores can use any number of protocols depending on the embodiment. In some examples, the RISC cores can run a real-time operating system (RTOS). The RISC cores can be implemented with one or more integrated circuits, application specific integrated circuits (ASICs), and/or memory devices. The RISC cores may include, for example, an instruction cache and/or tightly coupled RAM.

DMA may allow components of the PVA(s) to access system memory independently of the CPU(s) 706. The DMA can support any number of functions used to optimize the PVA, including but not limited to supporting multi-dimensional addressing and/or circular addressing. In some examples, the DMA may support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block gradation, vertical block gradation, and/or depth gradation.

The vector processors can be programmable processors that can be designed to efficiently and flexibly perform computer vision algorithm programming and provide signal processing functions. In some examples, the PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, DMA engine(s) (e.g., two DMA engines), and/or other peripherals. The vector processing subsystem may function as the primary processing unit of the PVA and may include a vector processing unit (VPU), an instruction cache, and/or vector memory (e.g., VMEM). A VPU core may include a digital signal processor such as a SIMD (single instruction, multiple data) digital processor with very long instruction words (VLIW). The combination of SIMD and VLIW can increase throughput and speed.

Each of the vector processors may include an instruction cache and may be associated with dedicated memory. Therefore, in some examples, each of the vector processors can be configured to operate independently of the other vector processors. In other examples, the vector processors included in a particular PVA may be configured to use data parallelism. For example, in some embodiments, the plurality of vector processors included in a single PVA may execute the same computer vision algorithm, but for different regions of an image. In other examples, the vector processors included in a particular PVA may simultaneously run different computer vision algorithms on the same image, or even different algorithms on consecutive images or portions of an image. Among other things, any number of PVAs in the hardware acceleration cluster and any number of vector processors in each of the PVAs. In addition, the PVA(s) may include additional error correction code (ECC) memory to increase overall system security.

The accelerator(s) 714 (e.g., hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator(s) 714 . In some examples, the on-chip memory may include at least 4MB of SRAM consisting of, for example and without limitation, eight field-configurable memory blocks accessible to both the PVA and the DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and the DLA can access the storage over a backbone that allows the PVA and the DLA to access the storage at high speed. The backbone may include an on-chip computer vision network that connects the PVA and DLA to memory (e.g., using the APB).

The on-chip computer vision network may include an interface that determines that both the PVA and DLA are providing operational and valid signals before transferring control signals/address/data. Such an interface may provide separate phases and separate channels for control signal/address/data transmission and burst communication for continuous data transmission. This type of interface can meet the standards ISO 26262 or IEC 61508, but other standards and protocols can also be used.

In some examples, the SoC(s) 704 may include a real-time ray tracing hardware accelerator as described in FIG U.S. Patent Application No. 16/101,232 , filed August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the positions and extents of objects (e.g. within a world model), to generate real-time visualization simulations, for RADAR signal interpretation, for sound propagation synthesis and/ or analysis, for simulating SONAR systems, for general wave propagation simulation, for comparison with LIDAR data for localization purposes and/or for other functions and/or for other purposes. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing operations.

The accelerator(s) 714 (e.g., hardware accelerator cluster) can be used for autonomous driving in a variety of ways. The PVA can be a programmable image processing accelerator that can be used for key processing steps in ADAS and autonomous vehicles. The capabilities of the PVA are well suited to algorithmic domains that require predictable processing with low power consumption and low latency. In other words, the PVA is well suited for semi-dense or dense regular computations, even with small data sets that require predictable runtimes with low latency and low power consumption. In the context of autonomous vehicle platforms, the PVAs are therefore designed to run classic computer vision algorithms, as they are efficient in object recognition and processing of integer mathematical data.

For example, in one embodiment of the technology, the PVA is used to perform computer stereo vision. In some examples, an algorithm based on semi-global adaptation may be used, although this is not intended as a limitation. Many Level 3-5 autonomous driving applications require motion estimation/stereo matching while driving (e.g. texture from motion, pedestrian detection, lane detection, etc.). The PVA can perform a computer stereo vision function with inputs from two monocular cameras.

In some examples, the PVA can be used to perform dense optical flow. Corresponding to the processing of raw RADAR data (e.g. using a 4D Fast Fourier Transform) to obtain processed RADAR data. In other examples, the PVA is used for processing flight time depth data, e.g. B. by processing raw time-of-flight data to obtain processed time-of-flight data.

Any type of network can be operated with the DLA to improve control and driving safety, e.g. B. a neural network that outputs a confidence value for each object detection. Such a confidence value can be interpreted as a probability or a relative "weight" of each detection compared to other detections. This confidence value allows the system to make further decisions about which detections are considered true positives and which which should be considered as false positive detections. For example, the system can set a confidence threshold and only consider detections that exceed this threshold as true positive detections. In an Automatic Emergency Braking (AEB) system, false positive detections would result in the vehicle automatically performing emergency braking, which is clearly undesirable. Therefore, only the safest detections should be considered as triggers for AEB. The DLA can use a neural network to regress the confidence value. The neural network can take as input at least a subset of parameters such as: B. the dimensions of the bounding box, the estimate of the ground plane obtained (e.g. from another subsystem), the output of the IMU sensor 766 correlating with the orientation of the vehicle 700, the distance, the estimates of the 3D position of the object obtained from the neural network and/or other sensors (e.g., LIDAR sensor(s) 764 or RADAR sensor(s) 760), and others.

The SoC(s) 704 may include data storage 716 (e.g., memory). The data store(s) 716 may be on-chip memory of the SoC(s) 704 that may store neural networks to be executed on the GPU and/or the DLA. In some examples, the capacity of the data store(s) 716 may be large enough to store multiple instances of neural networks for redundancy and security. The data store(s) 712 may include L2 or L3 cache(s) 712. The reference to the data store(s) 716 may include a reference to the memory associated with the PVA, DLA, and/or other accelerators 714, as described herein.

The SoC(s) 704 may include one or more processor(s) 710 (e.g., embedded processors). The processor(s) 710 may include a boot and power management processor, which may be a dedicated processor and subsystem to handle boot power and management functions and associated security enforcement. The boot and power management processor may be part of the boot sequence of the SoC(s) 704 and may provide runtime power management services. The boot power and management processor may provide clock and voltage programming, support for system transitions to a low power state, management of SoC(s) 704 temperatures and temperature sensors, and/or management of the SoC(s) 704 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC(s) 704 may use the ring oscillators to measure the temperatures of the CPU(s) 706, GPU(s) 708, and/or of the accelerator(s) 714 to detect. If temperatures are determined to exceed a threshold, the boot and power management processor may enter a temperature fault routine and place the SoC(s) 704 in a lower power state and/or the vehicle 700 in a chauffeur-to-safe state - Set stop mode (e.g. bring vehicle 700 to a safe stop).

The processor(s) 710 may also include a number of embedded processors that may serve as an audio processing module. The audio processing engine can be an audio subsystem that enables full hardware support for multi-channel audio across multiple interfaces and a wide and flexible range of audio I/O interfaces. In some examples, the audio processing engine is a dedicated processing core with a digital signal processor with dedicated RAM.

The processor(s) 710 may also include an always-on processor engine that can provide the necessary hardware functions to support low-power sensor management and wake-up use cases. The always-on processor engine may include a processor core, tightly coupled memory, supporting peripherals (e.g., timers and interrupt controllers), various I/O controller peripherals, and routing logic.

The processor(s) 710 may also include a security cluster engine that includes a dedicated processor subsystem for security management of automotive applications. The security cluster engine may include two or more processor cores, tightly coupled memory, supporting peripherals (e.g., timers, an interrupt controller, etc.), and/or routing logic. In a security mode, the two or more cores can operate in lockstep mode and act as a single core with comparison logic to detect differences between their operations.

The processor(s) 710 may also include a real-time camera engine, which may include a dedicated processor subsystem for managing the real-time camera.

The processor(s) 710 may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

The processor(s) 710 may include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to produce the final image for the to create a player window. The video image compositor may apply lens distortion correction to the wide-angle camera(s) 770, to the surround camera(s) 774, and/or to the cabin surveillance camera sensors. The in-cabin security camera sensor is preferably monitored by a neural network running on another instance of the Advanced SoC, configured to detect events in the cabin and respond accordingly. A system in the cabin can perform lip-reading to activate cellular service and make a call, dictate email, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or voice-controlled web browsing to allow. Certain features are only available to the driver when the vehicle is operating in an autonomous mode and are otherwise disabled.

The video image compositor can include enhanced temporal denoising for both spatial and temporal denoising. For example, if there is movement in a video, noise reduction will weight the spatial information appropriately and reduce the weight of the information coming from neighboring frames. When there is no motion in a frame or part of a frame, the temporal noise reduction performed by the video compositor can use information from the previous frame to reduce noise in the current frame.

The video image compositor can also be configured to perform stereo de-emphasis on the input stereo lens images. The video image compositor can also be used for user interface rendering when the operating system desktop is in use and the GPU(s) 708 does not need to constantly render new interfaces. Even when the graphics processor(s) 708 is on and actively engaged in 3D rendering, the video image compositor can be used to offload the graphics processor(s) 708 and improve performance and responsiveness.

The SoC(s) 704 may also include a MIPI serial camera interface for receiving video and input from cameras, a high-speed interface, and/or a video input block that can be used for camera and related pixel input functions. The SoC(s) 704 may also include an input/output controller(s) that may be controlled by software and used to receive I/O signals that are not on are tied to a specific role.

The SoC(s) 704 may also include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management, and/or other devices. The SoC(s) 704 can be used to process data from cameras (e.g., connected via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LIDAR sensor(s) 764, RADAR sensor( en) 760, etc., which may be connected via Ethernet), data from bus 702 (e.g., vehicle 700 speed, steering wheel position, etc.), data from GNSS sensor(s) 758 (e.g., connected via Ethernet or CAN bus) can be used. The SoC(s) 704 may also include dedicated high-performance mass storage controllers, which may include their own DMA engines, and which may be used to offload the CPU(s) 706 from routine data management tasks.

The SoC(s) 704 can be an end-to-end platform with a flexible architecture that covers automation levels 3 to 5 and thereby offers a comprehensive functional security architecture that uses computer vision and ADAS techniques for Leverages diversity and redundancy, providing a platform for a flexible, reliable driving software stack along with deep learning tools. The SoC(s) 704 can be faster, more reliable, and even more power and space efficient than traditional systems. For example, the accelerator(s) 714 in combination with the CPU(s) 706, the GPU(s) 708, and the data storage(s) 716 can form a fast, efficient platform for level 3-5 autonomous vehicles .

The technology thus offers capabilities and functions that cannot be achieved with conventional systems. For example, computer vision algorithms can run on CPUs that can be configured to run a variety of processing algorithms using high-level languages such as the C programming language for a variety of image data. However, CPUs are often unable to meet the performance requirements of many image processing applications, e.g. B. in terms of execution time and power consumption. In particular, many CPUs are unable to execute complex object detection algorithms in real time, which is a requirement for in-vehicle ADAS applications and a requirement for practical Level 3-5 autonomous vehicles.

In contrast to traditional systems, the technology described here, by providing a CPU complex, a GPU complex and a hardware acceleration cluster, allows multiple neural networks to run simultaneously and/or sequentially and the results to be combined with each other to create autonomous Enable level 3-5 driving functions. For example, a CNN running on the DLA or dGPU (e.g., the GPU(s) 720) may include text and word recognition that enables the supercomputer to read and understand traffic signs, including road signs, for which the neural network has not been specially trained. The DLA may further include a neural network capable of identifying, interpreting and semantically understanding the traffic sign and passing this semantic understanding to the path planning modules running on the CPU complex.

Another example is that multiple neural networks can be operated simultaneously as required for level 3, 4 or 5 driving. For example, a warning sign reading “Caution: Flashing lights indicate black ice” along with an electric light can be interpreted by multiple neural networks independently or jointly. The sign itself can be identified as a road sign by a first deployed neural network (e.g., a trained neural network), the text "Blinking lights indicate black ice" can be interpreted by a second deployed neural network that runs the vehicle's path planning software (which preferably runs on the CPU complex) that there is black ice when detecting blinking lights. The blinker can be identified through the operation of a third neural network over several frames that informs the vehicle's path planning software of the presence (or absence) of blinkers. All three neural networks can run simultaneously, e.g. B. within the DLA and/or on the GPU(s) 708.

In some examples, a CNN may use data from camera sensors for facial recognition and vehicle owner identification to detect the presence of an authorized driver and/or owner of the vehicle 700 . The "always on" sensor processing engine can be used to unlock the vehicle when the owner approaches the driver's door and turns on the lights, and in security mode to disarm the vehicle when the owner exits the vehicle. In this way, the SoC(s) 704 provide security against theft and/or carjacking.

In another example, an emergency vehicle detection and identification CNN may use data from microphones 796 to detect and identify emergency vehicle sirens. Unlike traditional systems that use general classifiers to detect sirens and manually extract features, the SoC(s) 704 use CNN to classify environmental and city sounds, as well as classify visual data. In a preferred embodiment, the CNN running on the DLA is trained to recognize the relative closing speed of the emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the local area in which the vehicle is operating, as determined by GNSS sensor(s) 758 . For example, CNN will attempt to detect European sirens when deployed in Europe and only North American sirens when deployed in the United States. Once an emergency vehicle is detected, a control program can be used to run an emergency vehicle safety routine, slow the vehicle, pull to the curb, park the vehicle, and/or allow the vehicle to coast using the ultrasonic sensors 762. until the emergency vehicle(s) drive by.

The vehicle may include one or more CPU(s) 718 (e.g., discrete CPU(s) or dCPU(s)) connected to the SoC(s) 704 via a high-speed interconnect (e.g., PCIe). could be. The CPU(s) 718 may include an X86 processor, for example. The CPU(s) 718 may be used to perform a variety of functions, including arbitrating potentially conflicting results between ADAS sensors and the SoC(s) 704 and/or monitoring the health and status of the/ the controller(s) 736 and/or the infotainment SoC 730, for example.

The vehicle 700 may include one or more GPU(s) 720 (e.g., discrete GPU(s) or dGPU(s)) connected to the SoC(s) via a high-speed interconnect (e.g., NVIDIA's NVLINK). (s) 704 may be coupled. The graphics processor(s) 720 may provide additional artificial intelligence functions, e.g. by executing redundant and/or different neural networks, and may be used to train and/or update neural networks based on inputs (e.g., sensor data) from vehicle 700 sensors.

The vehicle 700 may further include the network interface 724, which may include one or more wireless antennas 726 (e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface 724 can be used to establish a wireless connection over the Internet to the cloud (e.g., to the server(s) 778 and/or other network devices), to other vehicles, and/or to computing devices (e.g., . Passenger client devices). To communicate with other vehicles, a direct connection can be established between the two vehicles and/or an indirect connection (e.g. via networks and the Internet). Direct connections can be made via a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide the vehicle 700 with information about vehicles in the vicinity of the vehicle 700 (e.g., vehicles in front of, beside, and/or behind the vehicle 700). This function may be part of a cooperative vehicle 700 adaptive cruise control function.

Network interface 724 may include an SoC that provides modulation and demodulation functions and enables controller(s) 736 to communicate over wireless networks. The network interface 724 may include a radio frequency front end for baseband to radio frequency upconversion and radio frequency to baseband downconversion. The frequency conversions can be carried out using known methods and/or using superheterodyne methods. In some examples, the radio frequency front end functionality may be provided by a separate chip. The network interface can contain wireless functions for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN and/or other wireless protocols.

The vehicle 700 may further include one or more data stores 728 that may be stored off-chip (e.g., off the SoC(s) 704). Data storage(s) 728 may include one or more storage elements such as RAM, SRAM, DRAM, VRAM, Flash, hard drives, and/or other components and/or devices capable of storing at least one bit of data.

Vehicle 700 may also include GNSS sensor(s) 758 . The GNSS sensor(s) 758 (e.g., GPS, supported GPS sensors, differential GPS sensors (DGPS), etc.) assist in mapping, perception, mapping grid creation, and/or path planning functions. Any number of GNSS sensors 758 can be used, e.g. B. and without limitation, a GPS using a USB port with an Ethernet-to-Serial (RS-232) bridge.

Vehicle 700 may also include RADAR sensor(s) 760 . The RADAR sensor(s) 760 can be used by the vehicle 700 for long-range vehicle detection, even in darkness and/or poor weather conditions. RADAR sensor(s) 760 may use the CAN bus and/or bus 702 (e.g., to transmit data generated by RADAR sensor(s) 760) for control and use to access object tracking data, in some examples raw data being accessed over Ethernet. A variety of RADAR sensor types can be used. For example and without limitation, the RADAR sensor(s) 760 may be suitable for front, rear, and side RADAR deployment. In some examples, pulse Doppler RADAR sensors are used.

The RADAR sensor(s) 760 may include various configurations such as: e.g., long range with narrow field of view, short range with wide field of view, short range side coverage, etc. In some examples, long range RADAR can be used for adaptive cruise control. The long-range RADAR systems can provide a wide field of view realized by two or more independent scans, e.g. B. within a range of 250 m. The RADAR sensor(s) 760 can help to distinguish between static and moving objects and can be used by ADAS systems for emergency brake assist and forward collision warnings. Long-range RADAR sensors may include monostatic multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and a high-speed CAN and FlexRay interface. In a six-antenna example, the center four antennas can produce a focused beam pattern designed to view the vehicle's surroundings at higher speeds with minimal noise Detect disruptions caused by traffic in the adjacent lanes. The other two antennas can expand the field of view so that vehicles entering or leaving the vehicle's lane can be quickly detected.

For example, medium-range RADAR systems can have a range of up to 760 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 750 degrees (rear). Short-range RADAR systems may include, but are not limited to, RADAR sensors placed at both ends of the rear bumper. When such a RADAR sensor system is installed at both ends of the rear bumper, it can generate two beams that constantly monitor the blind spot behind and beside the vehicle.

Short-range RADAR systems can be used in an ADAS system for blind spot detection and/or as a lane change assistant.

The vehicle 700 may also include one or more ultrasonic sensors 762 . The ultrasonic sensor(s) 762, which may be attached to the front, rear, and/or sides of the vehicle 700, may be used for parking assistance and/or to create and update a population grid. A variety of ultrasonic sensors 762 can be used, and different ultrasonic sensors 762 can be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensor(s) 762 can operate at ASIL B functional safety levels.

Vehicle 700 may include LIDAR sensor(s) 764 . The LIDAR sensor(s) 764 may be used for object and pedestrian detection, emergency braking, collision avoidance, and/or other functions. The LIDAR sensor(s) 764 may be ASIL B functional safety level. In some examples, the vehicle 700 may include multiple LIDAR sensors 764 (e.g., two, four, six, etc.) that may use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

In some examples, the LIDAR sensor(s) 764 may be able to provide a list of objects and their distances for a 360 degree field of view. Commercially available LIDAR sensors 764 can have a range of about 700 m, with an accuracy of 2 cm to 3 cm and with support for a 700 Mbit/s Ethernet connection, for example. In some examples, one or more non-protruding LIDAR sensors 764 may be used. In such examples, the LIDAR sensor(s) 764 may be implemented as a small device that may be embedded in the front, rear, sides, and/or corners of the vehicle 700 . The LIDAR sensor(s) 764 in such examples can provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, with a range of 200 m, even on objects with low reflection. The front-mount LIDAR sensor(s) 764 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

In some examples, lidar technologies such as 3D flash lidar may also be employed. 3D Flash LIDAR uses a laser flash as a transmission source to illuminate the vehicle's surroundings up to a distance of about 200 m. A flash LIDAR unit includes a receptor that records the travel time of the laser pulse and the reflected light on each pixel, which in turn corresponds to the distance between the vehicle and the objects. With Flash-LIDAR, high-precision and distortion-free images of the environment can be generated with each laser flash. In some examples, four flash LIDAR sensors may be deployed, one on each side of vehicle 700. Available 3D flash LIDAR systems include a solid state 3D star array LIDAR camera that has no moving parts other than a fan (e.g. a non-scanning LIDAR device). The flash LIDAR device can use a 5 nanosecond Class I (eye safe) laser pulse per image and collect the reflected laser light in the form of 3D range point clouds and jointly registered intensity data. By using flash LIDAR, and because flash LIDAR is a solid state device with no moving parts, the LIDAR sensor(s) 764 may be less susceptible to motion blur, vibration, and/or shock.

The vehicle may also include IMU sensor(s) 766 . The IMU sensor(s) 766 may be located at the center of the rear axle of the vehicle 700 in some examples. The IMU sensor(s) 766 may include, for example and without limitation, an accelerometer(s), magnetometer(s), gyroscope(s), magnetic compass(es), and/or or other types of sensors. In some examples, such as For example, in six-axis applications, IMU sensor(s) 766 may include accelerometers and gyroscopes, while in nine-axis applications, IMU sensor(s) 766 may include accelerometers, gyroscopes, and magnetometers.

In some embodiments, the IMU sensor(s) 766 may be implemented as a miniaturized, high-performance GPS-based inertial navigation system (GPS/INS) that includes microelectromechanical inertial sensors (MEMS), a high-sensitivity GPS receiver, and advanced Kalman filter algorithms combined to provide estimates of position, velocity and attitude. In some examples, the IMU sensors 766 can enable the vehicle 700 to estimate heading without requiring input from a magnetic sensor by directly observing and correlating the speed changes from GPS with the IMU sensors 766 . In some examples, IMU sensor(s) 766 and GNSS sensor(s) 758 may be combined into a single integrated unit.

The vehicle may include one or more microphones 796 mounted in and/or around the vehicle 700 . The microphone(s) 796 may include: used to detect and identify emergency vehicles.

The vehicle may further include any number of camera types, including stereo camera(s) 768, wide-angle camera(s) 770, infrared camera(s) 772, surround camera(s) 774, far and/or mid-range camera(s) 798, and/or or other camera types. The cameras can be used to capture image data around the entire perimeter of the vehicle 700 . The types of cameras used depend on the embodiments and requirements for the vehicle 700 and any combination of camera types can be used to provide the required coverage around the vehicle 700 . In addition, the number of cameras can vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and/or a different number of cameras. For example and without limitation, the cameras may support Gigabit Multimedia Serial Link (GMSL) and/or Gigabit Ethernet. Each of the cameras is referred to herein with respect to 7A and 7B described in more detail.

The vehicle 700 may also include one or more vibration sensors 742 . The vibration sensor(s) 742 may detect vibrations from components of the vehicle, such as B. the axis (s), measure. Changes in vibration can indicate a change in road surface, for example. In another example, when using two or more vibration sensors 742, the differences between the vibrations can be used to determine the friction or slippage of the road surface (e.g., when the difference in vibration between a driven axle and a freely rotating axis exists).

The vehicle 700 may include an ADAS system 738 . ADAS system 738 may include an SoC in some examples. The 738 ADAS system can include Autonomous/Adaptive/Automatic Cruise Control (ACC), Cooperative Adaptive Cruise Control (CACC), Forward Collision Warning (FCW), Automatic Emergency Braking (AEB), Lane Departure Warning (LDW), Lane Departure Warning (LKA), blind spot warning (BSW), cross traffic warning (RCTW), collision avoidance system (CWS), lane centering (LC), and/or other features and functions.

The ACC systems may use RADAR sensor(s) 760, LIDAR sensor(s) 764, and/or camera(s). The ACC systems may include a longitudinal ACC and/or a transverse ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately ahead of vehicle 700 and automatically adjusts vehicle speed to maintain a safe distance from vehicles in front. The side ACC maintains the distance and advises the vehicle 700 to change lanes if necessary. Lateral ACC is related to other ADAS applications such as LCA and CWS.

CACC uses information from other vehicles, which may be received via network interface 724 and/or radio antenna(s) 726 from other vehicles over a wireless connection or indirectly over a network connection (e.g., over the Internet). Direct connections can be established via a vehicle-to-vehicle (V2V) communication link, while indirect connections can be an infrastructure-to-vehicle (I2V) communication link. In general, the V2V communication concept provides information about the vehicles immediately ahead (e.g., vehicles that are immediately ahead and in the same lane as the vehicle 700), while the I2V communication concept provides information about the traffic further ahead. CACC systems can contain either or both I2V and V2V information sources. Given the information about the vehicles in front of the vehicle 700, CACC can be more reliable and has the potential to improve traffic flow and reduce congestion on the road.

FCW systems are designed to warn the driver of a hazard so that corrective action can be taken. Use FCW systems a forward-looking camera and/or RADAR sensor(s) 760 coupled to a dedicated processor, DSP, FPGA and/or ASIC electrically connected to feedback to the driver, e.g. B. via a display, a speaker and / or a vibrating component. FCW systems can issue a warning, e.g. B. in the form of a tone, a visual warning, a vibration and / or a quick brake impulse.

AEB systems detect an impending collision with another vehicle or object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. AEB systems may use forward-looking camera(s) and/or RADAR sensor(s) 760 connected to a dedicated processor, DSP, FPGA, and/or ASIC. When the AEB system detects a hazard, it typically first warns the driver so that corrective action can be taken to avoid the collision. If the driver does not take corrective action, the AEB system can automatically apply the brakes to prevent, or at least mitigate, the effects of the predicted collision. AEB systems can include techniques such as Dynamic Brake Assist and/or Crash Imminent Braking.

Lane departure warning systems warn the driver with visual, acoustic and/or tactile signals, e.g. B. by vibrations in the steering wheel or the seat when the vehicle 700 drives over the lane markings. A lane departure warning system will not activate if the driver indicates intentional departure from the lane by using the turn signal. LDW systems may use forward-facing cameras connected to a dedicated processor, DSP, FPGA, and/or ASIC that is electrically coupled to driver feedback, e.g. B. with a display, a speaker and / or a vibrating component.

LKA systems are a variant of LDW systems. LKA systems provide steering intervention or braking to correct the vehicle 700 when the vehicle 700 begins to drift out of lane.

BSW systems detect and warn the driver of vehicles in the vehicle's blind spot. BSW systems can provide a visual, audible and/or tactile warning signal to indicate that it is unsafe to merge or change lanes. The system can issue an additional warning if the driver activates a turn signal. BSW systems may use rear-facing camera(s) and/or RADAR sensor(s) 760 coupled to a dedicated processor, DSP, FPGA, and/or ASIC that is electrically connected to feedback to the driver, e.g. B. with a display, a speaker and / or a vibrating component.

RCTW systems may provide a visual, audible, and/or tactile notification when an object is detected out of range of the rear view camera while the vehicle 700 is backing up. Some RCTW systems include AEB to ensure vehicle brakes are applied to avoid an accident. RCTW systems may use one or more rear-facing RADAR sensors 760 coupled to a dedicated processor, DSP, FPGA, and/or ASIC that is electrically coupled to driver feedback, e.g. B. with a display, a speaker and / or a vibrating component.

Traditional ADAS systems can produce false positives, which can be annoying and distracting for the driver, but are usually not catastrophic because the ADAS systems warn the driver and give them the opportunity to decide if there really is a state of security and to act accordingly. In an autonomous vehicle 700, however, the vehicle 700 must decide for itself whether to consider the result of a primary computer or a secondary computer (e.g., a first controller 736 or a second controller 736) in the event of conflicting results. For example, in some embodiments, ADAS system 738 may be a backup and/or secondary computer that provides perceptual information to a rationality module of the backup computer. The backup computer's rationality monitor can run redundant software on hardware components to detect errors in perception and dynamic driving tasks. The ADAS system 738 outputs can be forwarded to a higher-level MCU. When the outputs of the primary and secondary computers collide, the monitoring MCU must determine how to resolve the contention to ensure safe operation.

In some examples, the primary computer may be configured to provide the high-level MCU with a confidence value indicative of the primary computer's confidence in the selected outcome. If the confidence level exceeds a threshold, the monitoring MCU can follow the primary computer's instruction, regardless of whether the secondary computer provides a conflicting or inconsistent result. If the confidence value does not meet the threshold and the primary and secondary computers report different outcomes (e.g., the competition), the master MCU can mediate between the computers to determine the appropriate outcome.

The monitoring MCU may be configured to execute a neural network or networks that are trained and configured to determine the conditions based on the outputs of the primary computer and the secondary computer .determine under which the secondary computer generates false alarms. This allows the neural network(s) in the monitoring MCU to learn when the output of the secondary computer can and cannot be trusted. For example, if the secondary computer is a RADAR-based FCW system, a neural network in the higher-level MCU can learn when the FCW system identifies metallic objects that do not actually pose a threat, such as a drain grate or a manhole cover , which triggers an alarm. If the secondary computer is a camera-based lane departure warning system, a neural network in the monitoring MCU can learn to override the lane departure warning system when cyclists or pedestrians are present and lane departure is indeed the safest manoeuvre. In embodiments where a neural network(s) is/are running on the monitoring MCU, the monitoring MCU may include at least one DLA or graphics processor capable of operating the neural network(s) with associated memory. In preferred embodiments, the monitoring MCU may comprise and/or be included as a component of the SoC(s) 704 .

In other examples, the ADAS system 738 may include a secondary computer that performs the ADAS functions using conventional computer vision rules. So, the secondary computer can use classic computer vision rules (if-then), and having a neural network(s) in the parent MCU can improve reliability, security, and performance. For example, the different implementation and intentional non-identity makes the overall system more fault-tolerant, especially to faults caused by software functions (or software-hardware interfaces). For example, if a software error occurs in the software running on the primary computer and the non-identical software code on the secondary computer gives the same overall result, the monitoring MCU can be more confident that the overall result is correct and the error is in the software or hardware of the primary computer did not cause any significant error.

In some examples, the output of the ADAS system 738 may be fed into the perception block of the host computer and/or the dynamic driving task block of the host computer. For example, if the ADAS system 738 displays a warning of an impact due to an object immediately ahead, the perception block can use this information in identifying objects. In other examples, the secondary computer may have its own neural network that is trained and thus reduces the risk of false positives, as described herein.

The vehicle 700 may also include the infotainment SoC 730 (eg, an in-vehicle infotainment system (IVI)). Although the infotainment system is illustrated and described as an SoC, it does not necessarily have to be one SoC and may also include two or more discrete components. The infotainment SoC 730 may include a combination of hardware and software used to provide audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., television, movies, streaming, etc.), telephone (e.g. hands-free), network connectivity (e.g., LTE, Wi-Fi, etc.) and/or information services (e.g. navigation systems, rear parking assistance, a radio data system, vehicle-related information such as fuel level , total distance traveled, brake fuel level, oil level, door open/close, air filter information, etc.) to the vehicle 700. For example, the infotainment SoC 730 can provide radios, turntables, navigation systems, video players, USB and Bluetooth connectivity, carputers, in-car Entertainment, Wi-Fi, steering wheel audio controls, speakerphone, a heads-up display (HUD), an HMI display 734, a telematics device, a control panel (e.g., for controlling and/or interacting with various components, functions, and /or systems) and/or other components. The infotainment SoC 730 may further be used to provide information (e.g., visually and/or audibly) to one or more users of the vehicle, e.g. B. information from the ADAS system 738, autonomous driving information such as planned vehicle maneuvers, trajectories, environmental information (e.g., intersection information, vehicle information, road information, etc.), and/or other information.

The infotainment SoC 730 may include GPU functionality. The infotainment SoC 730 can communicate with other devices, systems, and/or components of the vehicle 700 via the bus 702 (eg, CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 730 may be coupled to a supervisory MCU such that the infotainment system's GPU may perform some self-driving functions if the primary controller(s) 736 (e.g., primary and/or or vehicle backup computer 700) fail. In such an example, the infotainment SoC 730 may place the vehicle 700 in a chauffeured mode until a safe stop, as described herein.

The vehicle 700 may further include an instrument cluster 732 (eg, a digital dashboard, an electronic instrument cluster, a digital instrument panel, etc.). The cluster 732 may include a controller and/or a supercomputer (eg, a discrete controller or a supercomputer). The instrument cluster 732 may include a number of instruments, e.g. B. Speedometer, fuel level, oil pressure, tachometer, odometer, turn signals, shift position indicator, seat belt warning light(s), parking brake warning light(s), engine malfunction light(s), airbag system (SRS) information, lighting controls, security system controls , navigation information, etc. In some examples, the information may be displayed and/or shared between the infotainment SoC 730 and the instrument cluster 732 . In other words, the instrument cluster 732 can be part of the infotainment SoC 730 or vice versa.

7D 7 is a system diagram for communication between one or more cloud-based servers and the example autonomous vehicle 700. FIG 7A , according to some embodiments of the present disclosure. System 776 may include server(s) 778, network(s) 790, and vehicles, including vehicle 700. The server(s) 778 may include a variety of GPUs 784(A)-784(H) (collectively referred to herein as GPUs 784), PCIe switches 782(A)-782(H) (collectively referred to herein as PCIe switches 782) and/or CPUs 780(A)-780(B) (collectively referred to herein as CPUs 780). The 784 GPUs, 780 CPUs, and PCIe switches can be used with high-speed connections such as g. NVLink interfaces 788 and/or PCIe connections 786 developed by NVIDIA. In some examples, the GPUs 784 are connected via NVLink and/or NVSwitch SoC, and the GPUs 784 and PCIe switches 782 are connected via PCIe connections. Although eight 784 GPUs, two 780 CPUs, and two PCIe switches are shown, this should not be taken as a limitation. Depending on the embodiment, each of the servers 778 may include any number of GPUs 784, CPUs 780, and/or PCIe switches. For example, each of the servers 778 may include eight, sixteen, thirty-two, and/or more GPUs 784.

The server(s) 778 may receive, via the network(s) 790 and from the vehicles, image data representative of images showing unexpected or changing road conditions, e.g. B. Recently started road works. The server(s) 778 may transmit neural networks 792, updated neural networks 792, and/or map information 794, including information about traffic and road conditions, over the network(s) 790 and to the vehicles. Map information 794 updates may include HD map 722 updates, e.g. B. Information about roadworks, potholes, detours, flooding and/or other obstacles. In some examples, the neural networks 792, the updated neural networks 792, and/or the map information 794 may result from new training and/or experience represented in data received from any number of vehicles in the area and/or or based on training performed in a data center (e.g., using server(s) 778 and/or other servers).

The server(s) 778 can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated by the vehicles and/or in a simulation (e.g. with a game engine). In some examples, the training data is tagged (e.g., where the neural network benefits from supervised learning) and/or undergoes other preprocessing, while in other examples, the training data is untagged and/or preprocessed (e.g., . if the neural network does not require supervised learning). Training may be performed following one or more classes of machine learning techniques, including but not limited to classes such as: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analyses) , multilinear subspace learning, diverse learning, representational learning (including learning with replacement dictionaries), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning models are trained, the machine learning models can be used by the vehicles (e.g. by transmission to the vehicles via the network(s) 790 and/or the machine learning models can be used by the server(s) 778 to remotely monitor the vehicles.

In some examples, the servers 778 may receive data from the vehicles and apply the data to real-time up-to-date neural networks for real-time intelligent reasoning. Server(s) 778 may include deep learning supercomputers and/or dedicated AI computers powered by GPUs 784, such as B. the DGX and DGX Station machines developed by NVIDIA. However, in some examples, the servers 778 may also include deep learning infrastructures using only CPU-powered data centers.

The deep learning infrastructure of the server(s) 778 may be capable of high-speed, real-time inferencing and may use this capability to assess and verify the state of the processors, software, and/or associated hardware in the vehicle 700 . For example, the deep learning infrastructure may receive periodic updates from the vehicle 700, such as a sequence of images and/or objects that the vehicle 700 located in that sequence of images (e.g., via computer vision and/or other machine tools). object classification techniques). The deep learning infrastructure can run its own neural network to identify the objects and compare them to the objects identified by the vehicle 700. If the results do not match and the infrastructure concludes that the AI in the vehicle 700 is not working properly, the server (or servers) 778 can send a signal to the vehicle 700 instructing a failsafe vehicle 700 computer to do the Take control, notify passengers and perform a safe parking manoeuvre.

For inferencing, the server(s) 778 may include the GPU(s) 784 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-driven servers and inference acceleration can enable real-time responsiveness. In other examples where performance is less critical, servers with CPUs, FPGAs, and other processors can be used for inference.

Exemplary computing device

8th 8 is a block diagram of an example computing device(s) 800 suitable for use in implementing some embodiments of the present disclosure. The computing device 800 may include an interconnection system 802 that directly or indirectly connects the following devices: memory 804, one or more central processing units (CPUs) 806, one or more graphics processing units (GPUs) 808, a communications interface 810, input/output (I/O) ports O) 812, input/output components 814, a power supply 816, one or more presentation components 818 (e.g. display(s)) and one or more logic units 820. In at least one embodiment, the computing unit(s) 800 can have one or more include virtual machines (VMs) and/or each of their components may include virtual components (e.g., virtual hardware components). As non-limiting examples, one or more of the GPUs 808 may include one or more vGPUs, one or more of the CPUs 806 may include one or more vCPUs, and/or one or more of the logical units 820 may include one or more virtual logical units. As such, a computing device(s) 800 may have discrete components (e.g., a full GPU dedicated to computing device 800), virtual components (e.g., a portion of a GPU dedicated to computing device 800 dedicated) or a combination thereof.

Although the various blocks in 8th shown connected to leads via connection system 802, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 818, such as a a display device, may be considered an I/O component 814 (e.g., if the display is a touch screen). Another example is that CPUs 806 and/or GPUs 808 may include memory (e.g., memory 804 may represent a storage device in addition to the memory of GPUs 808, CPUs 806, and/or other components). In other words, the computing device of the 8th is purely illustrative. It does not differentiate between categories such as "workstation", "server", "laptop", "desktop", "tablet", "client device", "mobile device", "handheld device", "game console", "electronic control unit". (ECU)", "Virtual Reality System" and/or other types of devices or systems, as they all fall within the scope of the computing device 8th fall.

Connection system 802 may represent one or more connections or buses, such as B. an address bus, a data bus, a control bus or a combination thereof. Connection system 802 may include one or more bus or connection types, e.g. an ISA (Industry Standard Architecture) bus, an EISA (Extended Industry Standard Architecture) bus, a VESA (Video Electronics Standards Associa tion), a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and/or other type of bus or interconnect. In some embodiments, there are direct connections between the components. For example, the CPU 806 can be directly connected to the memory 804. In addition, the CPU 806 can be connected to the GPU 808 directly. For direct or point-to-point connections between components, connection system 802 may include a PCIe connection to establish the connection. In these examples, a PCI bus need not be included in computing device 800 .

Memory 804 may consist of a variety of computer-readable media. The computer-readable media can be any available media that the computing device 800 can access. The computer-readable media can include both volatile and non-volatile media, as well as removable and non-removable media. By way of example and without limitation, the computer-readable media may include computer storage media and communications media.

Computer storage media may include both volatile and non-volatile media and/or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and/or other data types. For example, memory 804 may store computer-readable instructions (e.g., those representing a program(s) and/or program element(s), such as an operating system. Computer storage media may include, but is not limited to, RAM, ROM, EEPROM , flash memory or other storage technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other medium that can be used to store the desired information and onto which the computing device 800. As used herein, the term "computer storage medium" does not encompass signals per se.

The computer storage media may contain computer-readable instructions, data structures, program modules, and/or other types of data in a modulated data signal such as a B. embody a carrier wave or other transport mechanism and include any information transmission media. The term "modulated data signal" may refer to a signal that has one or more of its properties adjusted or altered to encode information in the signal. The computer storage media may include, for example, wired media, such as a wired network or direct cable connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of the above media should also fall within the scope of computer-readable media.

The CPU(s) 806 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 800 and perform one or more of the methods and/or processes described herein. The CPU(s) 806 may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing multiple software threads concurrently . The CPU(s) 806 may include any type of processor and may include different types of processors (e.g., processors with fewer cores for mobile devices and processors with more cores for servers) depending on the type of computing device 800 implemented. Depending on the nature of the computing device 800, the processor may be, for example, an Advanced RISC Machines (ARM) processor that uses Reduced Instruction Set Computing (RISC) or an x86 processor that uses Complex Instruction Set Computing (CISC). The computing unit 800 can include one or more CPUs 806 in addition to one or more microprocessors or additional co-processors, such as e.g. B. mathematical co-processors included.

In addition to or in the alternative to the CPU(s) 806, the GPU(s) 808 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 800 to perform one or more to carry out the methods and/or processes described herein. One or more of the GPU(s) 808 may be an integrated GPU (eg, with one or more of the CPU(s) 806 and/or one or more of the GPU(s) 808 may be a discrete GPU. In embodiments, one or more of the GPU(s) 808 may be a co-processor of one or more of the CPU(s) 806. The graphics processor(s) 808 may be used by the computing device 800 to render graphics (e.g., 3D graphics). or perform general purpose computations. The GPU(s) 808 may be used, for example, for General-Purpose Computing on GPUs (GPGPU).The GPU(s) 808 may include hundreds or thousands of cores capable of Process hundreds or thousands of software threads concurrently The GPU(s) 808 may, in response to rendering Commands (e.g., rendering commands from the CPU(s) 806 received over a host interface) generate pixel data for output images. The GPU(s) 808 may include graphics memory, e.g. a display memory, for storing pixel data or other suitable data, e.g. B. GPGPU data included. Display memory may be included as part of memory 804 . The GPU(s) 808 may include two or more GPUs operating in parallel (e.g., via an interconnect). The connection can connect the GPUs directly (e.g. using NVLINK) or connect the GPUs through a switch (e.g. using NVSwitch). When combined together, each GPU 808 can generate pixel data or GPGPU data for different parts of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can have its own memory or share memory with other GPUs.

In addition to or in the alternative to CPU(s) 806 and/or GPU(s) 808, logic unit(s) 820 may be configured to execute at least some of the computer-readable instructions ) to control one or more components of computing device 800 to perform one or more of the methods and/or processes described herein. In embodiments, CPU(s) 806, GPU(s) 808, and/or logic unit(s) 820 may execute any combination of the methods, processes, and/or portions thereof, discretely or collectively. One or more of the logic units 820 may be part of one or more of the CPU(s) 806 and/or the GPU(s) 808 and/or one or more of the logic units 820 may be discrete components or otherwise external to the CPU(s) 806 and/or or the GPU(s) 808. In embodiments, one or more of the logic units 820 may be a co-processor of one or more of the CPU(s) 806 and/or one or more of the GPU(s) 808 .

Examples of the logic unit(s) 820 include one or more processing cores and/or components thereof, such as data processing units (DPUs), tensor cores (TCs), tensor processing units (TPUs), pixel visual cores (PVCs), image processing units (VPUs), graphics processing clusters (GPCs). ), texture processing clusters (TPCs), streaming multiprocessors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic-logic units (ALUs), application-specific integrated circuits (ASICs ), Floating Point Units (FPUs), Input/Output (I/O) elements, Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) elements and/or the like.

Communications interface 810 may include one or more receivers, transmitters, and/or transceivers that enable computing device 800 to communicate with other computing devices over an electronic communications network, including wired and/or wireless communications. Communications interface 810 may include components and functionality that enable communication over a variety of networks, such as B. wireless networks (e.g. Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g. communication via Ethernet or InfiniBand), low-power wide area networks (e.g. LoRaWAN, SigFox, etc.) and/or the Internet. In one or more embodiments, logical unit(s) 820 and/or communication interface 810 may include one or more data processing units (DPUs) to transmit data received over a network and/or over connection system 802 directly to (e.g., a memory) one or more GPU(s) 808 to transmit.

Through I/O ports 812, computing device 800 may be logically coupled to other devices, including I/O components 814, presentation component(s) 818, and/or other components, some of which are built into computing device 800 (e.g., B. integrated) can be. Illustrative I/O components 814 include a microphone, mouse, keyboard, joystick, gamepad, game controller, satellite dish, scanner, printer, wireless device, etc. I/O components 814 can be a natural Provide a user interface (NUI) that processes a user's air gestures, speech, or other physiological input. In some cases, the inputs may be transmitted to an appropriate network element for further processing. A NUI may use any combination of speech recognition, pen recognition, facial recognition, biometric recognition, both on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) in conjunction with a display of the computing device implement 800. Computing device 800 may include depth cameras such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touch screen technology, and combinations thereof for gesture recognition and capture. Additionally, computing device 800 may include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit (IMU)) that enable motion detection. In some examples, the output of the accelerometers or gyroscopes from the computing device 800 can be used to display immersive augmented reality or virtual reality.

The power supply 816 can be a hardwired power supply, a battery power supply, or a combination thereof. The power supply 816 may provide power to the computing device 800 to enable the components of the computing device 800 to operate.

The presentation component(s) 818 may include a display (e.g., a monitor, touch screen, television screen, heads-up display (HUD), other display types, or a combination thereof), speakers, and/or other presentation components . The presentation component(s) 818 may receive data from other components (e.g., the GPU(s) 808, the CPU(s) 806, DPUs, etc.) and output the data (e.g. as image, video, sound, etc.).

Exemplary data center

9 FIG. 9 shows an exemplary data center 900 that may be used in at least one embodiment of the present disclosure. The data center 900 may include a data center infrastructure layer 910, a framework layer 920, a software layer 930, and/or an application layer 940.

As in 9 As illustrated, the data center infrastructure layer 910 may include a resource orchestrator 912, clustered compute resources 914, and node compute resources ("node CRs") 916(1)-916(N), where "N" represents any positive integer. In at least one embodiment, node CRs 916(1)-916(N) may include any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc. ), storage devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input/output devices (NW I/O), network switches, virtual machines (VMs), power supply modules, and /or cooling modules, etc. In some embodiments, one or more node CRs among node CRs 916(1)-916(N) may correspond to a server having one or more of the computing resources mentioned above. Additionally, in some embodiments, node CRs 916(1)-9161(N) may include one or more virtual components, such as: , vGPUs, vCPUs, and/or the like, and/or one or more of node CRs 916(1)-916(N) may correspond to a virtual machine (VM).

In at least one embodiment, the clustered computing resources 914 may include separate clusters of node CRs 916 housed in one or more racks (not shown) or in many racks in data centers in different geographic locations (also not shown). Separate groupings of node C.R.s 916 within the grouped compute resources 914 may include grouped compute, network, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs 916 containing CPUs, GPUs, DPUs, and/or other processors may be grouped in one or more racks to provide compute resources to support one or more workloads. The one or more racks may also contain any number of power modules, cooling modules, and/or network switches in any combination.

The resource orchestrator 912 may configure or otherwise control one or more node CRs 916(1)-916(N) and/or clustered computing resources 914. In at least one embodiment, resource orchestrator 912 may comprise a data center software design infrastructure (SDI) manager 900 . The resource orchestrator 912 may include hardware, software, or a combination thereof.

In at least one embodiment, as in 9 As shown, the framework layer 920 may include a job scheduler 933, a configuration manager 934, a resource manager 936, and/or a distributed file system 938. Framework layer 920 may include a framework to support software 932 of software layer 930 and/or one or more applications 942 of application layer 940 . The software 932 or application(s) 942 may each include web-based service software or applications such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 920 can be some type of free and open source software web application framework such as Apache Spark™ (hereafter "Spark"), which provides a distributed file system 938 for processing large amounts of data (e.g. " Big Data"), but is not limited to it. In at least one embodiment, job scheduler 933 may include a Spark driver to facilitate scheduling of workloads supported by different data center 900 tiers. The configuration manager 934 may be able to configure different layers, e.g. B. the software layer 930 and the framework layer 920 including Spark and the distributed file system 938 to support processing of large amounts of data. Resource manager 936 may be able to manage clustered or grouped computing resources allocated to support distributed file system 938 and job scheduler 933 . In at least one embodiment, clustered or grouped computing resources may include clustered computing resources 914 at the data center infrastructure layer 910 . The resource manager 936 may coordinate with the resource orchestrator 912 to manage these allocated or assigned computing resources.

In at least one embodiment, the software 932 included in the software layer 930 may include software used by at least portions of the node C.R.s 916(1)-916(N), the clustered computing resources 914, and/or the distributed file system 938 of the framework layer 920 becomes. One or more types of software may include: Internet search software, email virus scanning software, database software and streaming video content software.

In at least one embodiment, the application(s) 942 contained in the application layer 940 may include one or more types of applications used by at least portions of the node C.R.s 916(1)-916(N), the clustered computing resources 914 and/or the distributed file system 938 of the framework layer 920. One or more types of applications may include any number of genomic, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and/or but are not limited to other machine learning applications used in connection with one or more embodiments.

In at least one embodiment, configuration manager 934, resource manager 936, and resource orchestrator 912 may implement any number and type of self-modifying actions based on any amount and type of data collected in any technically possible manner. Self-modifying actions can relieve a data center operator of the data center 900 from potentially making bad configuration decisions and avoiding potentially underutilized and/or malfunctioning parts of a data center.

Data center 900 may include tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model(s) may be trained by computing weight parameters according to a neural network architecture using software and/or computing resources described above with respect to data center 900 . In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks may be used to derive or predict information using the resources described above in relation to the data center 900 using weighting parameters determined by a or several training techniques are calculated, e.g. B., but not only, those described here.

In at least one embodiment, data center 900 may use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and/or other hardware (or equivalent virtual computing resources) to perform training and/or inference using the resources described above. Additionally, one or more of the software and/or hardware resources described above may be configured as a service to enable users to train or infer information such as: B. image recognition, speech recognition or other artificial intelligence services.

Sample Network Environments

Network environments suitable for use in implementing embodiments of the disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and/or other types of devices. The client devices, servers, and/or other device types (e.g., any device) may reside on one or more instances of the computing device(s) 800 of the 8th be implemented - e.g. e.g., each device may include similar components, features, and/or functionality of the computing device(s) 800. Additionally, where backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may be part of a data center 900, an example of which is provided here with reference to FIG 9 is described in more detail.

The components of a network environment can communicate with one another over one or more networks, which may be wired, wireless, or both. The network can include multiple networks or a network of networks. For example, the network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet and/or a public switched telephone network (PSTN), and/or one or more private networks . When the network includes a wireless telecommunications network, components such as a base station, a communications tower, or even access points (among other components) can enable a wireless connection.

Compatible network environments include one or more peer-to-peer network environments - in which case a server cannot be included in one network environment - and one or more client-server network environments - in which case one or more servers can be included in one network environment to be involved. In peer-to-peer network environments, the functionality described herein may be implemented in relation to one or more servers on any number of client devices.

In at least one embodiment, a network environment may include one or more cloud-based network environments, a distributed computing environment, a combination thereof, and so forth. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and/or edge servers. A framework layer may include a framework to support software of a software layer and/or one or more applications of an application layer. The software or application(s) may each include web-based service software or applications. In embodiments, one or more client devices may utilize the web-based service software or applications (e.g., by accessing the service software and/or applications through one or more application programming interfaces (APIs)). The framework layer can be some type of free and open source software web application framework, e.g. B. a distributed file system used for processing large amounts of data (e.g. "Big Data"), but not limited to it.

A cloud-based network environment may provide cloud computing and/or cloud storage that performs any combination of the computing and/or data storage functions (or one or more portions thereof) described herein. Each of these various functions may be distributed across multiple central or core server locations (e.g., from one or more data centers that may be spread across a state, region, country, globe, etc.). When a connection to a user (e.g., a client device) is relatively close to one or more edge servers, a core server may assign at least some functionality to the edge server or servers. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and/or a combination thereof (e.g., a hybrid cloud environment).

The Client Device(s) may have at least some of the components, features and functionality of the(s) referred to herein 8th described computing device(s) 800 included. By way of example and without limitation, a client device may include a personal computer (PC), laptop, mobile device, smartphone, tablet computer, smart watch, wearable computer, personal digital assistant (PDA), MP3 player , a virtual reality headset, a global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, boat, flying ship, virtual machine, computer installed in capable of processing and storing the data, a boat, a flying ship, a virtual machine, a drone, a robot, a portable communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded System controller, remote control, device, consumer electronic device, workstation, edge device, any combination of the described devices, or any other suitable device.

The disclosure may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program modules, executed by a computer or other machine, e.g. a personal data assistant or other handheld device. In general, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs specific tasks or implements specific abstract data types. The disclosure can be applied in a variety of system configurations, including portable devices, consumer electronics, general purpose computing, more specialized computing devices, etc. The disclosure may also be applied in distributed computing environments where tasks are performed by remotely controlled devices that are linked through a communications network.

When reference is made herein to "and/or" in relation to two or more elements, it should be understood to mean only one element or a combination of elements. For example, "Item A, Item B, and/or Item C" can include only Item A, only Item B, only Item C, Item A and Item B, Item A and Item C, Item B and Item C, or Items A, B and C include. Additionally, "at least one of A or B" may include at least one of A, at least one of B, or at least one of A and at least one of B.

The subject matter of the present disclosure is described herein with sufficient detail to meet the requirements of the law. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors contemplated that the claimed subject matter could be embodied in other ways to include various steps or combinations of steps similar to those described in this document in conjunction with other present or future technologies. Although the terms "step" and/or "block" may be used herein to denote various elements of the methods employed, the terms should not be construed to imply any particular order among or between the various steps disclosed herein, es unless the order of each step is specifically described.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• US16101232 [0137]

Non-patent Literature Cited

• ISO 26262 [0136]

Claims (20)

Procedure that includes: identifying a scenario for a first vehicle based at least on analyzing perception data generated by at least one sensor of the first vehicle in an environment; determining, based at least on the perception data, a first path for the first vehicle in the scenario and a second path for a second vehicle in the scenario, the second path including at least one contention point with the first path; determine, based at least on the perception data and a set of traffic rules corresponding to the environment, one or more traffic rules from the set of traffic rules that apply to the scenario; assigning a contention state to the at least one contention point based on the at least one or more traffic rules applicable to the scenario; generating a wait element associated with the scenario, the wait element encoding a representation of the first geometry associated with the first path, the second geometry associated with the second path, and the state of the contention; and providing the queuing element to a control agent of the first vehicle, wherein the control agent is configured to determine a right-of-way behavior for the first vehicle using the queuing element. procedure after claim 1 , further comprising detecting, using the perception data, a current state of one or more traffic signals contained in the environment, wherein determining the one or more traffic rules of the set of traffic rules applicable to the scenario at least on the current State of the one or more traffic signals is based. procedure after claim 1 wherein the perceptual data includes first geometry information associated with at least the first pathway, and the method further comprises: receiving map data representative of second geometry information associated with at least the first pathway; and generating a representation of the first geometry of the first path based at least on the merging of the first geometry information with the second geometry information. procedure after claim 1 or 2 wherein the perception data includes first signal information associated with one or more detected traffic signals in the area, and the method further comprises: receiving map data including second signal information associated with one or more localized traffic signals from at least one map; and generating merged signal information associated with at least one traffic signal based at least on the merging of the first signal information with the second signal information, wherein determining the one or more traffic rules of the set of traffic rules applicable to the scenario based at least on the merged Signal information based. The method of any preceding claim, wherein the perceptual data includes perceptual geometry data, and the method further comprises classifying the first path as belonging to at least one class of a set of predetermined classes based at least on the perceptual geometry data, wherein determining the one or more traffic rules of the set of traffic rules applicable to the scenario is based at least on the first path belonging to the at least one class. The method of any preceding claim, wherein the perception data includes signal perception data associated with one or more traffic signals and geometry data associated with the first path, and the method further at least based on associating the one or more traffic signals with the first path on evaluating a distance between the one or more traffic signals and the first path using the signal perception data and the geometry data. The method of any preceding claim, wherein the set of traffic rules includes at least one of one or more map rules associated with locating the first vehicle on at least one map, or one or more base rules associated with a geographic region of the first vehicle are. A processor comprising: one or more circuits to: identify a scenario for a first vehicle based at least on analysis of sensor data generated by at least one sensor of the first vehicle in an environment; based at least on locating the first vehicle on one or more maps determine a first path for the first vehicle in the scenario and a second path for a second vehicle in the scenario, the second path including at least one contention point with the first path; determine one or more traffic rules applicable to the scenario based at least on the locating of the first vehicle; assign a state of contention to the at least one contention point based at least on one or more of the traffic rules applicable to the scenario; generate a wait element associated with the scenario, the wait element encoding a representation of the first geometry associated with the first path, the second geometry associated with the second path, and the state of the contention; and provide the wait element to a control agent of the first vehicle, wherein the control agent is configured to determine a give-way behavior for the first vehicle using the wait element. processor after claim 8 wherein the one or more circuits are further operable to sense a current state of one or more traffic signals included in the one or more maps using perception data generated by at least one sensor of the first vehicle in the vicinity wherein determining the one or more traffic rules applicable to the scenario is based at least on the current state of the one or more traffic signals. processor after claim 8 or 9 wherein the one or more circuits are further to: receive map data of the one or more maps based at least on the locating, the map data including first geometry information associated with at least the first path; determine perception data generated by the at least one sensor of the first vehicle in the environment, the perception data including second geometry information associated with at least the first pathway; and generate a representation of at least the first geometry of the first path based at least on the merging of the first geometry information with the second geometry information. Processor after one of Claims 8 until 10 wherein the one or more circuits are further to: receive map data from the one or more maps based at least on the locating, the map data including first signal information associated with one or more localized traffic signals from the one or more maps ; determine perception data generated by the at least one sensor of the first vehicle in the environment, the perception data including second signal information associated with one or more detected traffic signals in the environment; and generate fused signal information associated with at least one traffic signal based at least on the merging of the first signal information with the second signal information, wherein determining the one or more traffic rules applicable to the scenario is based at least on the fused signal information . Processor after one of Claims 8 until 11 , wherein the map data includes geometry data and the method further comprises classifying the first path as belonging to at least one class of a set of predetermined classes based on at least the geometry data, wherein determining the one or more traffic rules that apply to the scenario at least based on the first way belonging to the at least one class. Processor after one of Claims 8 until 12 , wherein the map data includes signal data associated with one or more traffic signals, and the one or more circuits are further operable to associate the one or more traffic signals with the first path based at least on evaluating a distance between the one or more multiple traffic signals and the first path using the signal data and the geometry data associated with the first path. Processor after one of Claims 8 until 13 , wherein the traffic rules include one or more map rules associated with locating the first vehicle on at least one map or one or more base rules associated with a geographic region of the first vehicle. A system comprising: one or more processing units to determine one or more traffic rules applicable to a scenario based at least on analyzing sensor data generated using at least one sensor of a vehicle in an environment, the scenario having a generated path of the vehicle has in the area; assign a state of contention based on the at least one or more traffic rules applicable to the scenario; generate a wait element associated with the scenario, the wait element encoding a first geometry associated at least with the generated path and the state of contention; and provide the wait item to a control agent of the vehicle, wherein the control agent is configured to determine a right-of-way behavior for the vehicle using the wait item. system after claim 15 , wherein the one or more processing units are further operative to detect a current state of one or more traffic signals contained in the environment, wherein the determining of the one or more traffic rules applicable to the scenario is at least based on the current state of the one or more traffic signals. system after claim 15 or 16 , wherein the one or more processing units are further operable to: receive perception data including first geometry information associated at least with the path; receive map data representative of second geometry information associated at least with the path; and generate at least the first geometry of the path based at least on the merging of the first geometry information with the second geometry information. system according to one of the Claims 15 until 17 wherein the one or more processing units are further operable to: receive perception data including first signal information associated with one or more detected traffic signals in the environment; receive map data including second signal information associated with one or more localized traffic signals from at least one map; and generate fused signal information associated with at least one traffic signal based at least on the merging of the first signal information with the second signal information, wherein determining the one or more traffic rules applicable to the scenario is based at least on the fused signal information . system according to one of the Claims 15 until 18 , wherein the one or more processing units are further operative to classify the path as belonging to at least one class of a set of predetermined classes based at least on geometry perception data, wherein determining the one or more traffic rules applicable to the scenario is based at least on the path that belongs to the at least one class. system according to one of the Claims 15 until 19 wherein the one or more processing units are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system containing one or more virtual machines (VMs); a system implemented at least in part in a data center; or a system implemented at least in part using cloud computing resources.

Images