DE102022114521A1 - PARALLEL PROCESSING OF A VEHICLE PATH PLANNING SUITABLE FOR PARKING - Google Patents

Abstract

In order to determine a path through a pose configuration space, trajectories of poses can be evaluated at least on the basis of a parallel translation of the trajectories along at least one axis of the pose configuration space (e.g. an alignment axis). A trajectory may include at least a portion of a turn that has a fixed turn radius. Twists or twist portions that have the same twist radius and initial orientation can be translated along the θ-axis and processed in parallel since they are translated copies of each other, but with different starting points (x,y). At a minimum, trajectories can be evaluated based on processing variables used to evaluate reachability as bit vectors with threads effectively performing large vector operations synchronously. A concurrency reduction pattern can be used to account for dependencies that may exist between sections of a trajectory to assess reachability, and to allow parallel processing of the sections.

Description

BACKGROUND

To control a vehicle or other maneuverable object, a recommended path may be determined from a current pose (e.g., position and orientation) of the vehicle to a target pose for the vehicle (e.g., for autonomous parking). Determining a recommended path for a vehicle may involve exploring potential paths in configuration space under non-holonomic constraints dictated by vehicle kinematics. Traditional approaches to solving this problem can use graph search (e.g. A*) with a heuristic that reduces the search space by encouraging early exploration of promising paths. However, the best paths may not be immediately apparent and therefore may not be identified in scenarios. Accordingly, there are environments where the heuristic does not exactly apply, since a heuristic may not be applicable in all situations.

In addition, the algorithms used by conventional systems can only be slightly parallelized. For example, parallel implementations of A* using eight processing threads have been developed to achieve four times speedup. However, modern parallel processors can provide thousands of processor cores, allowing for a much higher degree of parallelism. Due to the limited parallelism offered by conventional approaches, the density of the configuration space explored is often relatively low to enable path determination in a computationally efficient manner. That is, if the configuration space is too dense, identifying a path using conventional approaches may take too long or consume too many resources for practical application. The number of potential paths that can be determined can therefore be limited by the density of the configuration space.

SUMMARY

Embodiments of the present invention relate to mass-parallel vehicle path planning suitable for parking. Systems and methods are disclosed that can determine a vehicle's path through a pose configuration space in a highly parallelized manner.

In contrast to traditional approaches such as those described above, the disclosed approaches can be used to determine a path through the pose configuration space in a dense pose configuration space by quickly exploring paths (e.g. all paths) in parallel. Trajectories of poses in pose configuration space may be evaluated in parallel based at least on a parallel translation of the trajectories along at least one axis of the pose configuration space (e.g., a θ-axis representing vehicle orientation). In at least one embodiment, a trajectory may include at least a portion of a turn that has a fixed turning radius. Twists that have the same turning radius and initial orientation can be translated along the θ-axis and processed in parallel since they are translated copies of each other, but with different starting points (x,y). Furthermore, trajectories can be evaluated at least on the basis of processing variables used to evaluate reachability as bit vectors. Bitwise logical operations can be performed by each processing thread to control reachability propagation and avoid conditional processing or branches, allowing the threads to effectively perform large vector operations synchronously. The disclosed approaches may use a parallelism reduction pattern to account for dependencies that may exist between sections of a trajectory when assessing reachability and to allow parallel processing of the sections.

character list

• 1A eine Darstellung eines beispielhaften Wegplaners gemäß Ausführungsformen der vorliegenden Erfindung ist;
• 1B eine Darstellung ist, die zusätzliche Elemente zeigt, die gemäß Ausführungsformen der vorliegenden Erfindung im Wegplaner von 1A enthalten sein können;
• 2 eine Darstellung ist, die einen beispielhaften Posenkonfigurationsraum zeigt, der gemäß Ausführungsformen der vorliegenden Erfindung verwendet werden kann, um Fahrzeugposen zu modellieren;
• 3 ein beispielhaftes Bewegungsmodell zeigt, das gemäß Ausführungsformen der vorliegenden Erfindung verwendet werden kann, um Trajektorien des Fahrzeugs durch den Posenkonfigurationsraum zu definieren;
• 4 eine Darstellung eines beispielhaften Belegungsraums ist, der gemäß Ausführungsformen der vorliegenden Erfindung die Belegung von Posen in einem Posenkonfigurationsraum erfasst;
• 5 beispielhafte Wendungen verschiedener Wendungstypen zeigt, die gemäß Ausführungsformen der vorliegenden Erfindung in einem Posenkonfigurationsraum bewertet werden können;
• 6A einen beispielhaften „Korkenzieher“ einer Wendung zeigt, der gemäß Ausführungsformen der vorliegenden Erfindung in einem Posenkonfigurationsraum beschrieben wird;
• 6B den beispielhaften Korkenzieher der Wendung von 6A zusammen mit einen im Posenkonfigurationsraum beschriebenen Korkenzieher einer anderen Wendung zeigt, die gemäß Ausführungsformen der vorliegenden Erfindung eine parallelverschobene Kopie der ersten Wendung ist;
• 6C beispielhafte verschobene Trajektorien zeigt, die gemäß Ausführungsformen der vorliegenden Erfindung durch Parallelverschiebung der Korkenzieher von 6B gebildet wurden;
• 7 veranschaulicht, wie geradlinige Fahrwege konzeptionell auf einen Posenkonfigurationsraum gemäß Ausführungsformen der vorliegenden Erfindung abgebildet werden können;
• 8 einen beispielhaften Berechnungsflussgraphen zeigt, der gemäß Ausführungsformen der vorliegenden Erfindung zur Verarbeitung von Wendungen verwendet werden kann, wobei die Erreichbarkeit mit einem Binärwert kodiert ist;
• 9 einen beispielhaften Berechnungsflussgraphen zeigt, der gemäß Ausführungsformen der vorliegenden Erfindung zur Verarbeitung von Wendungen verwendet werden kann, wobei die Erreichbarkeit mit einem nicht binären Wert kodiert ist;
• 10 ein Ablaufplan ist, der ein Verfahren gemäß Ausführungsformen der vorliegenden Erfindung zur Bestimmung verschobener Posen eines Posenkonfigurationsraums zeigt, um einen Weg durch den Posenkonfigurationsraum zu bestimmen;
• 11 ein Ablaufplan ist, der ein Verfahren 1100 gemäß Ausführungsformen der vorliegenden Erfindung zur Parallelverschiebung der Trajektorien von Posen in einem Posenkonfigurationsraum zu verschobenen Trajektorien zeigt, um einen Weg durch den Posenkonfigurationsraum zu bestimmen;
• 12 ein Ablaufplan ist, der ein Verfahren 1200 gemäß Ausführungsformen der vorliegenden Erfindung zur parallelen Bewertung der Erreichbarkeit von Abschnitten einer Trajektorie zeigt.
• 13A eine Darstellung eines beispielhaften autonomen Fahrzeugs gemäß Ausführungsformen der vorliegenden Erfindung ist;
• 13B ein Beispiel für Kamerapositionen und Sichtfelder für das beispielhafte autonome Fahrzeug von 13A gemäß Ausführungsformen der vorliegenden Erfindung ist;
• 13C ein Blockdiagramm einer beispielhaften Systemarchitektur für das beispielhafte autonome Fahrzeug von 13A gemäß Ausführungsformen der vorliegenden Erfindung ist;
• 13D ein Systemdiagramm für die Kommunikation zwischen Cloud-basierten Servern und dem beispielhaften autonomen Fahrzeug von 13A gemäß Ausführungsformen der vorliegenden Erfindung ist;
• 14 ein Blockdiagramm einer beispielhaften Computereinheit ist, die zur Implementierung von Ausführungsformen der vorliegenden Erfindung geeignet ist; und
• 15 ein Blockdiagramm eines beispielhaften Datenzentrums ist, das zur Implementierung von Ausführungsformen der vorliegenden Erfindung geeignet ist.

The present systems and methods for parallel processing of a parking-ready vehicle path plan are described in detail below with reference to the accompanying drawings, wherein:

• 1A Figure 12 is an illustration of an exemplary path planner in accordance with embodiments of the present invention;
• 1B 12 is a diagram showing additional elements included in the path planner of FIG 1A may be included;
• 2 Figure 12 is a diagram showing an example pose configuration space that may be used to model vehicle poses in accordance with embodiments of the present invention;
• 3 Figure 12 shows an exemplary motion model that may be used to define trajectories of the vehicle through pose configuration space, in accordance with embodiments of the present invention;
• 4 Figure 12 is an illustration of an exemplary occupancy space that captures the occupancy of poses in a pose configuration space, in accordance with embodiments of the present invention;
• 5 FIG. 12 shows exemplary turns of different types of turns that can be evaluated in a pose configuration space according to embodiments of the present invention;
• 6A Figure 12 shows an exemplary "corkscrew" of a twist described in a pose configuration space in accordance with embodiments of the present invention;
• 6B the exemplary corkscrew of the phrase from 6A Figure 1 together with Figure 1 shows a corkscrew described in the pose configuration space of another twist that is a translated copy of the first twist in accordance with embodiments of the present invention;
• 6C shows exemplary shifted trajectories obtained by parallel shifting the corkscrews of FIG 6B were formed;
• 7 Figure 12 illustrates how straight-line travel paths can be conceptually mapped to a pose configuration space in accordance with embodiments of the present invention;
• 8th Figure 12 shows an exemplary computation flow graph that may be used to process idioms where reachability is encoded with a binary value, according to embodiments of the present invention;
• 9 Figure 12 shows an exemplary computation flow graph that may be used to process idioms where reachability is encoded with a non-binary value, according to embodiments of the present invention;
• 10 Figure 12 is a flow chart showing a method according to embodiments of the present invention for determining translated poses of a pose configuration space to determine a path through the pose configuration space;
• 11 12 is a flowchart showing a method 1100 according to embodiments of the present invention for translating the trajectories of poses in a pose configuration space to translated trajectories to determine a path through the pose configuration space;
• 12 12 is a flowchart showing a method 1200 according to embodiments of the present invention for evaluating the reachability of sections of a trajectory in parallel.
• 13A Figure 12 is an illustration of an exemplary autonomous vehicle in accordance with embodiments of the present invention;
• 13B an example of camera positions and fields of view for the example autonomous vehicle of FIG 13A according to embodiments of the present invention;
• 13C FIG. 14 is a block diagram of an example system architecture for the example autonomous vehicle of FIG 13A according to embodiments of the present invention;
• 13D a system diagram for communication between cloud-based servers and the exemplary autonomous vehicle of FIG 13A according to embodiments of the present invention;
• 14 Figure 12 is a block diagram of an exemplary computing device suitable for implementing embodiments of the present invention; and
• 15 Figure 12 is a block diagram of an exemplary data center suitable for implementing embodiments of the present invention.

DETAILED DESCRIPTION

Systems and methods are disclosed relating to parallel processing of parking-ready vehicle path planning. Although the present invention relates to an exemplary autonomous vehicle 1300 (also referred to herein as "vehicle 1300" or "ego vehicle 1300"), that up 13A-13D Described with reference to an example, this is not to be construed as limiting. The systems and methods described here are, for example, but not limited to, also in non-autonomous vehicles, semi-autonomous vehicles (e.g. in adaptive driver assistance systems (ADAS)), guided and unguided robots or robotic platforms, storage vehicles , ATVs, flying watercraft, boats, shuttles, emergency vehicles, motorcycles, electric or motorized two-wheelers, airplanes, construction vehicles, underwater vehicles, drones and/or other types of vehicles. Although the present invention is described in relation to path or route planning for autonomous or semi-autonomous driving, this is not to be understood in a limiting sense and the systems and methods described herein can be used for path planning in augmented reality, virtual reality -, mixed reality, robotics, security and surveillance applications, autonomous or semi-autonomous machine applications and/or in other technology areas in which path or route planning is used.

In a parking application, a goal may be to maneuver a vehicle into a parking space under non-holonomic constraints (e.g., approximately from line-of-sight distances) while avoiding collisions with obstacles (e.g., other vehicles, pillars, barriers, walls, car park structures, pedestrians, etc.). Systems and methods are disclosed that can determine a path (e.g., a multipoint turn suitable for parking or maneuvering in tight spaces) of a vehicle from a current pose to a target pose in a pose configuration space in which open space poses reside or blocked by an obstacle (eg, an obstacle detected based on vehicle sensor data). That is, there can be multiple paths from the current pose to the target pose, and each path can consist of various combinations of turns (e.g., sharp left, slightly left, straight, slightly right, and sharp right) and directions ( e.g. forwards and backwards). In addition, one or more obstacles can be positioned along one or more paths. The disclosed approaches can be used to evaluate the paths to identify a recommended path around obstacles and to the target pose based on one or more criteria (e.g., shortest path, fastest path, fewest number of turns, etc.). .

The disclosed approaches can be used to quickly explore paths (e.g., all paths) in a dense pose configuration space in parallel when determining a recommended path from a current pose to a target pose (e.g., in a target set of poses). An iterative approach can be used, where in one iteration the reachability of a set of trajectories can be evaluated in parallel and in a subsequent iteration the results serve as inputs for evaluating the reachability of the set of trajectories (or another set of trajectories). (thereby effectively adding additional trajectories to the achievable trajectories). Poor route recommendations can be avoided by thoroughly investigating the route options. Although this amount of work has so far been too great to be practical for dense pose configuration spaces, the disclosed approaches enable mass parallelism that can take advantage of modern parallel processing architectures (e.g., with thousands of cores and/or threads). This allows dense pose configuration spaces to be explored much faster, more efficiently, and/or with much lower granularity (e.g. higher spatial density and angular density) than previously possible.

In at least one embodiment, a pose configuration space may represent poses of a vehicle in an environment using at least x- and y-axes for the vehicle's position and a θ-axis for an orientation (e.g., heading angle) of the vehicle. Mass parallelism may arise, at least in part, by evaluating trajectories of poses in a pose configuration space in parallel based at least on a parallel translation of the trajectories along at least one axis of the pose configuration space (e.g., a θ-axis). Such an arrangement allows trajectories and/or portions thereof to be processed independently for parallel processing. For example, at least some of the poses of the trajectories can be shifted (e.g. during travel or in advance by a common parallel shift function) to form parallel lines, one or more sections of which can be independently traversed in parallel to improve accessibility to rate.

In one or more embodiments, a trajectory may include at least a portion of a turn that has a fixed turning radius. If a trajectory is a turn, it can form a cyclic trajectory or a straight trajectory (with an infinite turning radius). The disclosed approaches can exploit the property that each turn (or part of it) among the turns genes that have the same twist radius and initial orientation (same twist type) can be a translated copy of each of the other twists (or corresponding parts thereof), but with a different starting point (x,y), resulting from the fact that the vehicle behaves the same regardless of its initial position. Therefore, the twist-type twists (or twist portions) corresponding to different trajectories can be translated along the θ-axis and processed in parallel.

Further, the variables used to assess reachability may include bits representing binary values (e.g., reachable or not, space or not), with each pose corresponding to a respective bit of a variable. This allows trajectories to be evaluated at least on the basis of processing the variables as bit vectors. For example, a trajectory per bit of a bit vector can be evaluated in parallel. Bitwise logical operations can be performed by each thread to control reachability propagation and avoid conditional processing or branching that is detrimental to parallelism, allowing the threads to effectively perform large vector operations synchronously. Therefore, this approach may be appropriate for a modern parallel processor that benefits from large vector operations that are well matched in memory in a Simultaneous Instruction Multiple Thread (SIMT) fashion.

The disclosure also provides approaches that can be used to split the processing overhead used to evaluate the trajectories into sections of the trajectories, allowing for parallel evaluation of such sections rather than entire trajectories. Sections of a trajectory cannot be independent of each other, as reachability can affect through the entire trajectory (e.g. both forward and backward in a cyclic trajectory). This dependency can be an obstacle to parallel processing of the sections. The disclosed approaches may handle this dependency using a concurrency reduction scheme that hierarchically collects the results (e.g., computed in parallel) of sections, performs a small amount of processing on the collected results (e.g., in parallel), and then the results backscatters back to individual sections as inputs for further (e.g., parallel) processing.

This path planner can offer additional parallelization options. For example, when processing the reachability of each phrasal of each phrasal type, the phrasal may be divided into phrasal subsections, and each phrasal subsection may be processed semi-independently by a separate thread. One of the benefits of being able to edit turns subsections independently is to spread the work across multiple threads (if any). Because portions of a turn are expected to be shorter than full turns, memory latencies can be reduced since one thread is less likely to have to wait (that long) for another thread to finish.

Now on 1A referring, is 1A 12 is an illustration of an exemplary path planner 110 in accordance with embodiments of the present invention. It should be understood that these and other arrangements described herein are exemplary only. Other arrangements and elements (e.g., engines, interfaces, functions, sequences, groups of functions, etc.) may be used in addition to or in place of those shown, and some elements may be omitted altogether. Furthermore, many of the elements described herein are functional units that can be implemented as single or distributed components, or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by devices may be performed by hardware, firmware, and/or software. For example, various functions may be performed by a processor executing instructions stored in memory.

In at least one embodiment, the path planner 110 may include a configuration space manager 136 , a free space manager 120 , a reachability manager 122 , and a path assessor 130 . Configuration space manager 136 may manage a pose configuration space 112 representing poses (e.g., poses 114, 116, and 117) representing positions and orientations of a vehicle (or other object) in an environment (e.g., on a parking lot).

The clearance manager 120 and the reachability manager 122 may process the pose configuration space 112 to determine one or more ways to maneuver from a current pose C to a target pose T (or generally between any two poses) in the pose configuration space 112 . For example, the clearance manager 120 can collision check the vehicle with objects 124 in the environment to determine one or more occupancy spaces 118 that can detect which poses are blocked or occupied (e.g., pose 116 and pose 118) and which poses are free or unoccupied (e.g., pose 114 ). The reachability manager 122 may analyze at least portions of trajectories (eg, trajectories 126 and 128) of poses that the vehicle may traverse in the pose configuration space 112 to determine one or more reachability spaces 102 (also referred to as a cost space) that may capture , which poses in the pose configuration space 112 are reachable by the trajectories, and/or the cost of reaching those poses. The path estimator 130 may identify one or more recommended or potential paths for the vehicle based at least on the evaluation by the accessibility manager 122 . For example, the path estimator 130 may identify the paths or multi-segment trajectories (e.g., paths 132 and 134) based on one or more criteria (e.g., travel distance, number of turns, number of gear changes, cost, accessibility, etc.). and/or evaluate.

Now on 1B referring, is 1B 12 is a diagram showing additional elements included in path planner 110 of FIG 1A can be included. The configuration space manager 136 may include a parameter controller 138, the reachability manager 122 may include a pose translator 146 and a reachability evaluator 148, the free space manager 120 may include an object detector 140 and an occupancy evaluator 142, and the path evaluator 130 may include a trajectory estimator 150 and a backtracker 152 .

In at least one embodiment, configuration space manager 136 may facilitate operations related to pose configuration space 112 by using parameter controller 138 to configure pose configuration space 112 parameters. The pose configuration space 112 may pose vehicle poses using a space such as B. a multi-dimensional space (e.g. a 3D space). For example, the vehicle may be in a driving environment (eg, in a parking lot). Each pose in the pose configuration space 112 may include at least a vehicle position in the driving environment and an orientation of the vehicle to the vehicle position. In at least one embodiment, the positions in the environment in pose configuration space 112 may be represented by an XY grid (e.g., representing a ground plane), as shown in FIG 1A shown. The orientations in the environment may be represented in pose configuration space 112 by an angular orientation component θ (e.g., rotation relative to the x-axis). In at least one embodiment, the pose configuration space 112 may be parameterized as the pose P=(x,y,θ).

2 12 is a diagram showing an example pose configuration space that may be used to model vehicle poses in accordance with embodiments of the present invention. 2 illustrates an example of how a grid of poses 210—each pose P=(x,y,θ)—stored in pose configuration space 112 may represent positions 212 of a vehicle in the driving environment. For example shows 2 that each θ-disk 214 and 216 of the pose configuration space 112 can represent a set of (x,y) values in combination with a respective angular orientation θ.

3 12 shows an exemplary motion model 300 that may be used to define trajectories of the vehicle through the pose configuration space 112 in accordance with embodiments of the present invention. The poses captured in pose configuration space 112 may correspond to points on trajectories captured by motion model 300 . In the example shown, the motion model 300 includes an Ackermann motion model. According to the motion model 300, a turn may proceed as a circular body motion 310 with a center 312 of the turn axially aligned with a rear wheel axle 314 of the vehicle at a distance 316 from the center 318 of a rear wheel axle. As in 3 As shown, the rear wheel axis center 318 may correspond to the x,y coordinates of a pose in pose configuration space 112 . The distance 316 corresponds to the radius of the turn, which can be determined by an angle of the front wheels 320 and 322 of the vehicle. As in 3 As shown, the angle of the vehicle's front wheels 320 and 322 may correspond to the θ coordinate of a pose in pose configuration space 112 .

Various parameters can affect the pose configuration space 112, such as: B. the size of the environment represented by the pose configuration space 112 and the spatial size and angular size of the cells. Therefore, the configuration space manager 136 can use the parameter controller 138 to define these settings. For example, the parameter controller 138 may receive dimensions of the environment to be evaluated (e.g. 30m x 30m) and a cell size (e.g. 0.3m) relating to the spatial density of the pose configuration space 112 ( eg the number of cells in the x-axis and in the y-axis). In addition, the parameter controller 138 may also be given the angular size of the cells (e.g., in radians) or the number of angular cells, which affects the angular density of the pose configuration space 112 (e.g., the number of layers in the θ- axis) can affect. A pose configuration space with more cells in a given space (e.g., due to a smaller spatial cell size and/or angular cell size in a given region) may have a larger number of poses and paths eligible for evaluation, and a higher one Allow for accuracy when reaching a pose.

In at least one embodiment, to determine paths and/or multipoint turns specific to an environment, the occupancy of the pose configuration space 112 may be calculated, e.g. e.g., using clearance manager 120. That is, clearance manager 120 can determine which poses in pose configuration space 112 are recognized as at least partially occupied by an obstacle and which poses are recognized as open or vacant. Clearance manager 120 may use object detector 140 to detect and/or identify objects that may occupy or obstruct locations in the environment. The free space manager 120 may additionally or alternatively receive data representing objects from an object detector 140 operating external to the path planner 110 . The clearance manager 120 may use the occupancy evaluator 142 to perform a collision check of the body of the vehicle with the objects. Some expansion of the body of the vehicle and/or objects may be used to provide a margin.

In collision testing, the obstacle and vehicle body representations used as input to occupancy evaluator 142 may be polygonal or rasterized, and the output may be rasterized. The collision check may use the center of the vehicle's rear axle at (x,y) with a rotation θ relative to the x-axis. Now on 4 referring, is 4 Figure 4 shows an example occupancy space 412 that captures the occupancy of poses in a pose configuration space, in accordance with embodiments of the present invention. In particular shows 4 at least a portion of the occupancy space 412, which may be parameterized like the pose configuration space 112 (e.g., using at least x, y, θ). Also shows 4 an example obstacle 408 that may be detected, and represents the position of the obstacle 408 with respect to a set of (x,y) values 414, which may be the same for all θ's. In addition, 4 12 depict exemplary data values for a θ-plane 416 and for a θ-plane 418 that may correspond to a respective angular orientation of the vehicle.

By way of example and not limitation, the occupancy space 412 may store one bit per cell, where a one may represent that the corresponding pose is vacant and a zero may represent that the corresponding pose is not vacant (e.g., is occupied). As in 4 As shown, some of the poses 420 for the θ plane 416 are set to one and are free due to the angular orientation of the vehicle. However, the corresponding poses 422 in the θ plane 418 are set to zero and are potentially obstructed because the angle of the vehicle is different and potentially intersecting the obstruction 408 .

In at least one embodiment, the occupancy evaluator 142 may calculate the occupancy of the poses of the pose configuration space 112 and write the results to the occupancy space 412 in parallel. For example, each of the poses can be checked separately (e.g., by a thread) and the results written to the allocation space 412 in parallel (e.g., by the thread). The occupancy space 412 data may be used by the reachability manager 122 for subsequent processing and for path identification and evaluation.

With the data from the occupancy space obtained from the clearance manager 120, the path planner 110 can run an algorithm to generate a recommended path from a current pose (x _c , y _c , θ _c ) given the non-holonomic constraints of the vehicle. to find a target pose set (x _min , y _min , θ _min ), (x _max , y _max , θ _max ). Both the current pose and the target pose may be provided as inputs to the path planner 110 from one or more other motion planners (e.g., upon detection of a parking space, a position allowing a robot to pick up a pallet, etc.).

In at least one embodiment, the path planner 110 may be configured to evaluate one or more predetermined turn or trajectory types relative to the current or starting pose in the pose configuration space 112, where a turn or trajectory type corresponds to a given turn radius and direction (e.g., forward or backward). For example, a trajectory type may define multiple trajectories through the pose configuration space 112, and the achievable Accessibility manager 122 may use reachability evaluator 148 to evaluate any or almost any of these poses relative to an initial pose or current pose. Here, trajectories that are at least portions of turns are mainly described as examples, but the disclosed embodiments are more generally applicable to trajectories of other types, which may consist of, for example, turn primitives.

In at least one embodiment, the reachability evaluator 148 may determine reachability, for example in terms of whether the poses in a trajectory are reachable from the vehicle's current or initial pose C (or a target pose T or other pose, in some embodiments). An iterative approach may be used, in which the reachability manager 122 assesses the reachability of a set of trajectories (e.g., one or more turn types) and uses the results of the assessment as inputs to determine the reachability of the set of trajectories (or another set of trajectories) in a subsequent iteration. For example, a trajectory of a subsequent iteration may be reachable by at least that it is reachable by (e.g., linked to) at least one trajectory from a previous iteration and that the starting pose is reachable by that at least one trajectory from the previous iteration is.

The reachability assessor 148 may assess reachability for any number of trajectory executions (e.g., up to a threshold number of trajectory executions and/or until a target pose is reachable). For example, the reachability manager 122 may assess reachability if the vehicle in the pose configuration space 112 were to perform a single turn or a multipoint turn (e.g., up to a certain number of turns, e.g., 8 turns). In at least one embodiment, reachability assessor 148 may also assess the reachability of trajectories in terms of whether their poses are clear (e.g., based on clearance manager 120 determinations). For example, an obstacle in a trajectory can automatically make all subsequent poses in the trajectory inaccessible.

The reachability assessor 148 may also, in some embodiments, determine and/or record costs associated with reaching poses of the trajectories, for example when evaluating the reachability of the poses. As described here, a cost value can be used as the reachability indicator, for example with a 1 or 0 indicating whether a position is reachable or not, a maximum cost value indicating "unreachable" and another cost value indicating "reachable". indicates, etc. In other examples, the availability manager 122 may store cost values separately from the availability indicators. In various embodiments, one or more obstacles may not necessarily result in a pose being classified as unreachable in a trajectory, but may incur a cost greater than the cost value if no obstacle is present. Also, different obstacles can have different cost values.

As described herein, one or more of the trajectories associated with a pose in pose configuration space 112 may be a turn, and each "turn" may involve the vehicle moving forward or backward any distance while the steering wheel is stationary remains (thereby maintaining the turning radius). 5 Figure 12 illustrates examples of turns of different types of turns that can be evaluated in pose configuration space 112, including a sharp left turn 510, a slight left turn 512, a straight 514 (θ=0), a slight right turn 516, and a sharp right turn 518. Each of the various turn types can have a respective turning radius, with left turns carrying a negative (-) and right turns carrying a positive (+) sign. For example, the radius for the sharp left turn 510 can be "-10 m" and the radius for the slight left turn 512 can be "-20 m", while the radius for the sharp right turn 518 can be "10 m" and the Radius for the slight right turn 516 can be "20 m".

durch θ parametrisiert, wobei Q der Wendungsradius ist, q die Wendungsrichtung ist (z. B. +1 für rechts bei Vorwärtsfahrt oder -1 für links bei Vorwärtsfahrt), und (x₀, y₀) die Position ist, wenn θ null ist.In at least one embodiment, when traversing a turn type, the vehicle tracks the trajectory of poses in pose configuration space 112, which can be defined by Equation (1): $$P\left(\theta \right)=\left(x_0+qQ\text{sin}\theta {\text{, y}}_0+qQ\left(\text{cos}\theta \text{-1}\right),\theta \right),$$

parametrized by θ, where Q is the turning radius, q is the turning direction (e.g. +1 for right when driving forward or -1 for left when driving forward), and (x ₀ ,y ₀ ) is the position when θ is zero .

The reachability evaluator 148 can iterate through all potential combinations of phrases, including all variations of phrase types (e.g., forward first, then backward). The reachability manager 122 may update a reachability space 102 after each iteration of the reachability score with an indication of whether each pose in the pose configuration space 112 is reachable in that particular iteration (e.g., reachable all the way back to the home pose without being blocked by an obstacle) and /or with a cost of reaching the pose. Running through each trajectory at each iteration can represent a significant amount of processing and in some cases can be greatly speeded up by parallel processing.

In at least one embodiment, the reachability estimator 148 may analyze disjoint trajectories in the pose configuration space 112 to process trajectories in parallel to determine the reachability of the poses contained therein, allowing for independent processing. According to disclosed embodiments, the disjoint trajectories may include a set of trajectories of a common trajectory or turn type (e.g., turn radius and direction). In processing the trajectories or portions thereof (e.g., by a respective thread), the pose translator 146 of the reachability manager 122 may be configured to translate poses of the pose configuration space 112 based at least on the translation of the poses along the pose configuration space 112 at least of an axis of the pose configuration space 112 to determine the shifted poses of the disjoint trajectories. The disjoint trajectories (or portions thereof) may be evaluated by parallel processing (e.g., by a respective thread), allowing the reachability evaluator 148 to quickly assess the reachability of the shifted poses.

Examples of approaches that may be used by the pose translator 146 to translate the poses of the pose configuration space 112 are provided with reference to FIG 6A-6B described. Now on 6A referring, is 6A an example of a "corkscrew" of a twist 610 described in a pose configuration space 112 according to embodiments of the present invention. As shown, the turn 610 can be conceptually mapped onto a corkscrew in the pose configuration space 112 by following the trajectory of poses defined by equation (1). That is, as a rotation progresses through space (e.g., in following the poses defined by Equation (1)), the rotation may progress from current (x,y) coordinates to the next (x,y) coordinates , while they are also along the θ-axis through θ-planes like e.g. B. the θ-plane 616 advances. In other words, each of the x, y, and θ coordinates may change as the turn progresses to those specified in the in 6A change in the corkscrew-like manner shown.

In pose configuration space 112, any turn that has the same turn radius and initial orientation (same turn type) can be a translated copy of every other turn, but with a different starting point (x,y), due to the vehicle moving independently behaves in the same way from its initial position. 6A For example, FIG. 6 shows turn 610, which has a fixed turning radius and begins at (x,y) coordinates 614 in θ-plane 616. Any of the other twists that have the same twist radius but start at different (x,y) coordinates in the θ-plane 616 may be a translated copy of the twist 610 in the pose configuration space 112 . For example on 6B Referring shows 6B an example of a corkscrew of turn 610 from 6A along with a corkscrew of a twist 612 that is a translated copy of twist 618 and that is described in pose configuration space 112 in accordance with embodiments of the present invention. Turns that have the same turning radius can have shift parallelism that can be exploited by the pose translator 146 for parallel processing. While turns are described, other types of trajectories and/or portions thereof may also exhibit displacement parallelism, which may similarly be exploited for the parallel processing described herein.

In at least one embodiment, the pose translator 146 can translate the poses of the pose configuration space 112 along at least one axis (e.g., the θ-axis), and the reachability evaluator 148 can process each of the poses in parallel along the at least one axis to convert the Assess reachability (e.g. with one thread per turn or a section of it). For example, the parallelism of turn types can be revealed by having the pose translator 146 perform a parallel translation that makes the set of corkscrews in the pose configuration space 112 that correspond to the same turn radius a bundle of parallel lines along the θ-axis. The parallel lines can traverse the pose configuration space 112 in parallel as disjoint trajectories, allowing them to be processed independently.

als eine Funktion von θ durchführen und die Transformation, wie durch Gleichung (3) definiert: $$\left(x,y,\theta \right)\to \left(x-t_x\left(\theta \right),y-t_y\left(\theta \right),\theta \right)$$

auf den Posenkonfigurationsraum 112 anwenden (z. B. mit einem Warp). Dies kann auf eine Parallelverschiebung jeder konstanten θ-Ebene des Posenkonfigurationsraums 112 hinauslaufen.In at least one embodiment, for a given turn radius and direction (a turn type), the pose translator 146 can translate as defined by Equation (2): $$t\left(\theta \right)=\left(t_x\left(\theta \right),t_y\left(\theta \right)\right)=\left(qQ\text{sin}\theta \text{,}qQ\left(\text{cos}\theta \text{-1}\right)\right)$$

as a function of θ and perform the transformation as defined by equation (3): $$\left(x,y,\theta \right)\to \left(x-t_x\left(\theta \right),y-t_y\left(\theta \right),\theta \right)$$

apply to the pose configuration space 112 (e.g., with a warp). This may amount to a translation of each constant θ plane of pose configuration space 112 .

was einer Menge von Linien entspricht, die parallel zueinander und zur θ-Achse sind. Zum Beispiel auf 6C Bezug nehmend, zeigt 6C ein Beispiel für verschobene Trajektorien, die Ausführungsformen der vorliegenden Erfindung gemäß durch Parallelverschiebung der Korkenzieher von 6B gebildet wurden. Insbesondere stellt 6C konzeptionell den Posenkonfigurationsraum 112 zusammen mit den Wendungen 610 und 618 dar, wie sie durch den Posen-Parallelverschieber 146 parallelverschoben wurden. Wie gezeigt, sind die Wendungen 610 und 618 (z. B. mit dem gleichen Wendungsradius und der gleichen Richtung) nun parallele Linien entlang der θ-Achse und können als solche verarbeitet werden. Diese Transformation ist möglich, da die Trajektorien in Bezug auf die Parallelverschiebung entlang x und y in dem Sinne translationsinvariant sind, dass die Operationen des Folgens der Wendungskurve und der Parallelverschiebung kommutieren. Die Trajektorien des Wendungstyps können als parallelverschobene Kopien voneinander betrachtet werden und haben unabhängig von den (x₀, y₀)-Koordinaten des Ausgangspunkts die gleiche Form. Die Parallelität entlang der θ-Achse trägt zur Massenparallelität des Wegplaners 110 in einer Weise bei, die für einen modernen Parallelprozessor geeignet ist. Optional können die Daten transponiert werden, um die θ-Achse und die x--Achse zu vertauschen, sodass die Trajektorien stets parallel zur x-Achse (oder θ-Achse) sind.The transformed version of the turn trajectories of a turn type can be defined by equation (4): $$P\left(\theta \right)=\left(x_0,y_0,\theta \right),$$

which corresponds to a set of lines parallel to each other and to the θ-axis. For example on 6C Referring shows 6C an example of shifted trajectories, the embodiments of the present invention according to parallel shifting of the corkscrew of FIG 6B were formed. In particular represents 6C 12 conceptually represents the pose configuration space 112 along with the twists 610 and 618 as translated by the pose translator 146. FIG. As shown, turns 610 and 618 (e.g., having the same turning radius and direction) are now parallel lines along the θ-axis and can be processed as such. This transformation is possible because the trajectories are translation-invariant with respect to the translation along x and y in the sense that the operations of turning curve following and translation are commutated. The turn-type trajectories can be viewed as translated copies of each other and have the same shape regardless of the (x ₀ ,y ₀ ) coordinates of the starting point. The parallelism along the θ-axis contributes to the mass parallelism of the path planner 110 in a way that is appropriate for a modern parallel processor. Optionally, the data can be transposed to swap the θ-axis and the x- axis so that the trajectories are always parallel to the x-axis (or θ-axis).

was durch das Inverse der oben definierten Transformation gegeben ist. Diese Transformation kann leicht invertierbar und daher bijektiv sein. Es kann wünschenswert sein, dass diese Bijektivität bei Verwendung einer diskretisierten Implementierung der Transformation erhalten bleibt. Zum Beispiel kann durch eine bijektive Transformation eine komplette parallele Trennung zwischen Threads, die einzelne Trajektorien (oder Abschnitte davon) verarbeiten, beibehalten werden, ohne dass auf atomare Operationen zurückgegriffen werden muss oder Wettlauf-Bedingungen für das Lesen oder Schreiben zu befürchten sind. Dies kann auch die Fähigkeit für jede Zelle im diskretisierten Posenkonfigurationsraum verbessern, bei der Verarbeitung eines Wendungstyps von einem der Threads bedient zu werden. Die Bijektivität kann zum Beispiel erreicht werden, indem t(θ) in jeder konstanten θ-Ebene auf gleiche Weise quantisiert wird. Die vom Posen-Parallelverschieber 246 durchgeführte Transformation des Posenkonfigurationsraums 112 kann dann, als Bild betrachtet, eine Parallelverschiebung jeder konstanten θ-Ebene um eine ganzzahlige Anzahl von Pixeln sein. Dies ist für parallele Implementierungen ideal, da der verschobene Speicherzugriff während der Verarbeitung erfolgen kann, wodurch der Speicherzugriff, der ein limitierender Faktor sein kann, reduziert wird. Zusätzlich kann die Differenz in Parallelverschiebungen aufeinanderfolgender Ebenen reduziert werden (z. B. um nicht mehr als ein Pixel oder einen anderen Schwellenwert), um große Sprünge entlang der diskretisierten Trajektorienverarbeitung zu vermeiden. Dies kann durch eine ausreichende Winkelauflösung des Posenkonfigurationsraums 112 erreicht werden.In one or more embodiments, the pose translator 146 may transform the pose configuration space 112 (e.g., virtually by an access pattern or actually by copying data around) for each turn type (e.g., having the same radius and direction) so that the turning curves of this type of turning form the parallel lines. The reachability assessor 148 and/or other system components can then traverse each of the lines in parallel. The processing can be done parallel to the θ-axis along each of the parallel lines chosen according to equation (5) by (x ₀ ,y ₀ ) while accessing the original pose configuration space: $$\left(x_0+t_x\left(\theta \right),y_0+t_y\left(\theta \right),\theta \right),$$

which is given by the inverse of the transformation defined above. This transformation can be easily invertible and therefore bijective. It may be desirable that this bijectivity is preserved when using a discretized implementation of the transform. For example, a bijective transformation can maintain complete parallel separation between threads processing individual trajectories (or portions thereof) without resorting to atomic operations or fear of read or write race conditions. This can also improve the ability for each cell in the discretized pose configuration space to be serviced by one of the threads when processing a turn type. Bijectivity can be achieved, for example, by quantizing t(θ) in each constant θ plane in the same way. The transformation of the pose configuration space 112 performed by the pose translator 246, viewed as an image, may then be a translator of each constant θ plane by an integer number of pixels. This is ideal for parallel implementations as the deferred memory access can occur during processing, reducing the memory access that can be a limiting factor. Additionally, the difference in parallel displacements of successive planes can be reduced (e.g., by no more than a pixel or other threshold) to avoid large jumps along the discretes to avoid oriented trajectory processing. This can be achieved by providing sufficient angular resolution of the pose configuration space 112 .

durch u parametrisiert, wobei θ der nun konstante Steuerkurswinkel ist und (x₀, y₀) die Ausgangsposition ist, wenn u null ist. Im Posenkonfigurationsraum 112 können die Trajektorien als Linien mit einem Winkel θ in jeder θ-Ebene dargestellt werden. Zum Beispiel stellt 7 Trajektorien 710 und 712 dar, die bei jeweiligen (x, y)-Koordinaten anfangen und beide dem Winkel θ, der einer θ-Ebene 714 zugeordnet ist, entsprechend ausgerichtet sind. Darüber hinaus stellt 7 Trajektorien 716 und 718 dar, die bei jeweiligen (x, y)-Koordinaten anfangen und beide einem anderen Winkel 0, der einer θ-Ebene 720 zugeordnet ist, entsprechend ausgerichtet sind.In the reachability assessment, the path planner 110 may assess both curved turns and straight-line maneuvers (e.g., turns with an infinite turning radius). Traversing a trajectory of poses along a straight path, the poses can be defined by equation (6): $$P\left(\mathrm{and}\right)=\left(x_0+\mathrm{and}\text{cos}\theta {\text{y}}_0+\mathrm{and}\text{sin}\theta \text{,}\theta \right),$$

parametrized by u, where θ is the now constant heading angle and (x ₀ ,y ₀ ) is the home position when u is zero. In the pose configuration space 112, the trajectories can be represented as lines with an angle θ in any θ plane. For example puts 7 Illustrate trajectories 710 and 712 starting at respective (x,y) coordinates and both oriented according to the angle θ associated with a θ plane 714 . In addition, 7 trajectories 716 and 718 starting at respective (x,y) coordinates and both oriented according to a different angle θ associated with a θ plane 720. FIG.

A transformation (e.g., by the pose translator 146) from straight lines to parallel lines aligned with one of the coordinate axes can be accomplished in a number of ways. For example, in at least one embodiment, each θ plane can be rotated by the angle -θ, and as a continuous transform, this approach is bijective. For example, warps of work can be defined, where each warp rotates each constant θ-plane independently by an angle −θ about its center such that all lines are parallel to the x-axis. This approach can be efficient in a GPU since the warps in each constant θ plane consist of an image rotation, which can be achieved using the "nearest" interpolation method (e.g. to preserve the integrity of the integers in the warped space) . Rotation allows corners to be rotated outside of the original space, which can be accommodated by using a padded version of the space. Any cells that are outside of the original space can be treated as unreachable and impermissible, as with obstacles or high costs.

die auch als Gleichung (7) geschrieben werden kann: $$\{\begin{array}{c}P\left(x\right)=\left(x,y_0+x\text{ tan}\theta ,\theta \right).\left|\text{tan}\theta \right|\le 1\\ P\left(y\right)=\left(x_0-y\text{ tan}\left(\theta +\frac{\pi }{2}\right),y,\theta \right),\left|\text{tan}\theta \right|>1\text{'}\end{array}$$

A discretized image rotation cannot easily maintain a bijectivity of the transform, even if a bijective image rotation is achievable by three coordinate shearing transforms. Alternatively, a single shear transform per θ-plane can be applied, which produces less discretization noise and offers the possibility to perform the transform on-the-fly by simply parallel-shifting the memory access vector operations. For example, if the direction of the lines is closer to the x-axis direction than the y-axis direction (|tan θ| ≤ 1) or some other threshold, the pose translator 146 can shear the trajectories to be parallel to the x-axis or otherwise become the y-axis. In the former case (eg, closer to the x-axis), the pose translator 146 can set x ₀ = 0, and in the latter case (eg, closer to the y-axis), the pose translator 146 can set y _{Set 0} = 0 and still consider all lines. These operations can give the family of trajectories defined by equation (6): $$P\left(\mathrm{and}\right)=\{\begin{array}{c}\left(\mathrm{and}\text{cos}\theta {\text{<figure-callout id="0" label="y" filenames="DE102022114521A1\_0021.png,DE102022114521A1\_0027.png" state="{{state}}">y</figure-callout>}}_0+\mathrm{and}\text{sin}\theta \text{,}\theta \right),\left|\text{tan}\theta \right|\le 1\\ \left({\text{x}}_0+\mathrm{and}\text{cos}\theta \text{,}\mathrm{and}\text{sin}\theta \text{,}\theta \right),\left|\text{tan}\theta \right|>1\text{'}\end{array}$$

which can also be written as equation (7): $$\{\begin{array}{c}P\left(x\right)=\left(x,{\mathrm{<figure-callout\; id="0"\; label="y"\; filenames="DE102022114521A1\_0021.png,DE102022114521A1\_0027.png"\; state="\{\{state\}\}">y</figure-callout>}}_0+x\text{tan}\theta ,\theta \right).\left|\text{tan}\theta \right|\le 1\\ P\left(y\right)=\left(x_0-y\text{tan}\left(\theta +\frac{\pi }{2}\right),y,\theta \right),\left|\text{tan}\theta \right|>1\text{'}\end{array}$$

The bijective shear transformation can then be defined by equation (8): $$\left(x,y,\theta \right)\to \{\begin{array}{c}\left(x,y-x\text{tan}\theta ,\theta \right),\left|\text{tan}\theta \right|\le 1\\ \left(x+y\text{tan}\left(\theta +\frac{\pi }{2}\right),y,\theta \right),\left|\text{tan}\theta \right|>1\text{'}\end{array}$$

was einem Satz von Linien entspricht, die entweder parallel zur x-Achse oder zur y-Achse sind.After this transformation, the trajectories can take the form of equation (9): $$\{\begin{array}{c}P\left(x\right)=\left(x,{\mathrm{<figure-callout\; id="0"\; label="y"\; filenames="DE102022114521A1\_0021.png,DE102022114521A1\_0027.png"\; state="\{\{state\}\}">y</figure-callout>}}_0,\theta \right)\left|\text{tan}\right|\le 1\\ P\left(y\right)=\left(x_0,y,\theta \right)\left|\text{tan}\right|>1\end{array}$$

which corresponds to a set of lines parallel to either the x-axis or the y-axis.

was durch das Inverse der Schertransformation gegeben sein kann.As in the case of the curved turn, the reachability evaluator 148 can perform the processing along these lines while accessing the original pose configuration space according to equation (10): $$\{\begin{array}{c}\left(x,y+x\text{tan}\theta ,\theta \right),\left|\text{tan}\theta \right|\le 1\\ \left(x-y\text{tan}\left(\theta +\frac{\pi }{2}\right),y,\theta \right),\left|\text{tan}\theta \right|>1\text{'}\end{array}$$

which can be given by the inverse of the shear transformation.

As described herein, an iterative approach may be used in which the reachability manager 122 assesses the reachability of a set of trajectories (e.g., one or more turn types) and uses the results of the assessment as inputs to determine the reachability of the trajectory set (or one other trajectory set) in a subsequent iteration. To this end, the reachability evaluator 148 can be used to determine whether a pose in the pose configuration space 112 (e.g., as transformed by the pose translator 146) is reachable from a current position by, in one iteration, one or more Trajectories are processed and then one or more subsequent iterations (e.g. up to a given maximum number of turns) are performed to determine whether the pose becomes reachable in a subsequent iteration. For example, the reachability evaluator 148 can assess whether a pose is reachable by determining whether the pose is blocked or occupied (e.g., based on the occupancy space 412) and in one or more trajectories (e.g., 510, 512, 514, 516, 518, etc.) of an iteration.

In at least one embodiment, the reachability evaluator 148 may start at a current or initial pose (x _c , y _c , θ _c ) with only that pose marked as reachable. In a first iteration, the reachability assessor 148 may annotate one or more first iteration reachability spaces 102 to indicate (e.g., mark) whether each pose is reachable from the current pose using the trajectories in the first iteration ( e.g., along a turn trajectory of at least one of the trajectories without being blocked by an obstacle from the current pose). For example, the reachability rater 148 can rate the reachability for each turn type. This can result in a reachability space 102 that indicates a set of reachable poses after the first iteration. In a second iteration, the reachability assessor 148 may annotate one or more other reachability spaces 102 (or the same reachability spaces 120, in some embodiments) for the second iteration to indicate (eg, mark) whether each pose differs from the current pose from is achievable with the trajectories in the second iteration (e.g. for each turn type and/or different turn types). Accordingly, the iterations may continue until the reachability evaluator 148 has reached a maximum number of iterations (e.g., 8 turns), has discovered a target pose, and/or has met some other end condition. The route evaluator 130 may use the annotated reachability spaces 102 to determine which route, if any, is recommended based on a cost function (e.g., least number of turns, shortest route, or other cost score) or some other approach.

The reachability evaluator 148 may process the pose configuration space 112 in a variety of ways as it iterates. In at least one embodiment, for each trajectory type, the reachability evaluator 148 can reference the occupancy space 112 parameterized by (x, y, θ) and a separate reachability space 102, also parameterized by (x, y, θ) (e.g., a reachability space 102 for forward travel sharp left, an accessibility space 102 for reverse travel sharp left, an accessibility space 102 for forward travel straight ahead, etc.). In addition, each cell corresponding to a single pose (x, y, θ) in a corresponding reachability space 102 may be updated as the reachability assessor 148 evaluates whether the respective pose in the iteration is free and reachable.

Now on 8th Referring shows 8th Figure 8 shows an exemplary computation flow graph 800 that may be used to process phrases where reachability is encoded with a binary value, according to embodiments of the present invention. Calculation flow graph 800 may be suitable for embodiments where a binary value is used to store whether or not a position is reachable. In one or more embodiments, the reachability evaluator 148 may start with a shared reachability space at each iteration and process each phrase type in parallel to calculate a corresponding reachability space. For example, the reachability evaluator 148 may start at iteration 802A with a reachability space 808A and enter a sharp left turn type to create a reachability space 810A, a sharp left turn type to create a reachability space 810A, a slight left turn type to create a reachability space 812A process a straight turn type in parallel to create a reachability space 814A, a turn slightly right type to create a reachability space 816A, and a turn sharp right type to create a reachability space 818A. Although not shown, both forward and reverse turn types can be processed.

At each iteration, the reachability evaluator 148 may also merge the reachability spaces for the phrase types to provide the shared reachability space for a subsequent iteration. For example, iteration 802A may include reachability evaluator 148 performing a merge 820A of reachability spaces 810A, 812a, 814A, 816A, and 818A to generate reachability space 808B as input to an iteration 802B. In one or more embodiments, the merge 820A may include a logical OR operation. A logical OR may be appropriate to reflect that the reachability spaces may cover different paths to particular cells, each of which may be valid. In at least one embodiment, the shared reachability space may be cached in shared memory (e.g., a GPU) that each thread has access to.

In one aspect of the present invention, reachability evaluator 148 propagates reachability along an entire turn trajectory such that each pose that lies along the turn trajectory of the reachable single pose (e.g., shaded boxes in 9 ) and is not separated from the reachable single pose by an obstacle, is marked as reachable. To determine which pose(s) are reachable from a previous iteration, the path planner iterates through each and every turn that has that turn radius and direction, and since the parallel translation has been applied, this can be done in parallel. At each pose (x, y, θ), the path planner turns reachability off when an obstacle is found (e.g., based on occupancy space) and turns reachability on when reachability from a previous iteration is found.

This allows the reachability evaluator 148 to transform a reachability space from a previous iteration into a reachability space after the iteration when running through an iteration. In at least one embodiment, the reachability evaluator 148 may apply a rule stating that if a pose on a trajectory (e.g., a particular turn of a turn type) being processed was previously reachable (as indicated by the reachability space) , all poses along the trajectory that are not separated by an obstacle from any of the previously reachable points are reachable. In this way, reachability can propagate along the trajectory until it is stopped by an obstacle.

8th contains pseudocode 830 to illustrate how a thread of the reachability assessor 148 can traverse M pose cells of a trajectory with a kernel loop that turns off the reachability represented by a local variable r for a pose cell i when the thread encounters an obstacle, as represented by an element of a occupancy space entry F, and turns on reachability for pose cell i if the thread finds a reachability from a previous iteration, as represented by a reachability space entry Ri, and then reachability r into a reachability space -Output Ro writes out. As described herein, the pose translator 146 can be used to shift the poses for processing in advance or on the fly.

In one or more embodiments, all variables r, F, Ri, Ro can be processed as bit vectors. For example, if the variables are expressed as 32-bit unsigned integers, 32 bits and thus 32 parallel trajectories can be processed as bit vectors in parallel. The processing must not include any conditional processing or branching - just a logical AND to stop propagation when the free space ends and a logical OR to stop the Start or restart propagation when reachability is given in the input from the previous iteration. In addition, the processing of this core loop can be parallel and precisely synchronized between many threads, each responsible for a trajectory, so that the threads perform large vector operations effectively synchronized.

Therefore, this approach may be appropriate for a modern parallel processor that benefits from large vector operations that are well coordinated in memory in a Simultaneous Instruction Multiple Thread (SIMT) manner, similar to Simultaneous Instruction Multiple Data (SIMD), however different. For example, some parallel processing architectures may use a basic unit of parallel processing such as B. have a warp or wavefront of 32 threads striving to run concurrently to increase efficiency. Here, a "warp" may more generally refer to a basic unit of parallel processing. Each streaming multiprocessor can process one or more basic units at a time, and there can be numerous multiprocessors. A single thread can process 32 trajectories (e.g. twists) as a bit vector at the same time, so each basic unit (e.g. warp) can process 32x32=1024 trajectories in parallel. In addition, there may be eight or more multiprocessors, each handling many basic units simultaneously to achieve tens of thousands of parallelism. By being able to evaluate each trajectory (e.g., turn) in each iteration, the path planner 110 is able to do all the work rather than apply a heuristic to hopefully pick a promising path. This allows the path planner 110 to avoid recommendations resulting from inapplicable heuristics that attempt to limit operations to the best work first.

The path planner 110 may go through one or more iterations, and the path estimator 130 may use the results of the iteration(s) to identify and/or recommend at least one path to maneuver the vehicle to the target pose. To this end, the reported reachability spaces of each iteration can be preserved to support backtracking as described herein. Since the reachability spaces only require one bit per cell, this can only require N bits per cell in the pose configuration space. With a reasonable number of iterations, this is no more than what would have been used by representing the spaces by an 8, 16, or 32 bit fixed or floating point number.

To identify and/or select a path, the path estimator 130 may include the trajectory estimator 150, which evaluates one or more costs associated with trajectories (e.g., turns), and there are many cost functions of varying complexity that are applicable are to identify a recommended path. A simpler cost function may consider the number of trajectories (e.g. turns) required to reach a given pose and give higher priority to paths that reach the pose in fewer trajectories. A more complex cost function can take into account how many resources are required to follow a path. For example, the trajectory estimator 150 may apply a cost function that models the time taken to complete a trajectory, which may include a penalty for shifting from forward to reverse or back, and a time taken to complete a trajectory as a function of its travel distance rated. In an even more complex example, a state space can be used by adding the turning curvature and sign (the underlying state can be steering wheel and road wheel position and gear) and speed. This enables the modeling of costs incurred when engaging reverse gear by braking to a standstill or requiring large changes in steering wheel position.

The trajectory estimator 150 can apply different cost models depending on the goals and application of the path planner 110 . For example, a cost model may omit an explicit speed component and instead have costs that depend, at least in part, on the distance traveled by a turn, with a penalty for changing the type of turn (including engaging reverse gear) dependent on the type of turn before and after the change is dependent. This approach approaches the understanding that turns are negotiated at a fixed, low speed that is reached relatively quickly, and that switching between turn types carries an additional disadvantage of having to slow down to turn the steering wheel and/or to change gear. This cost model can be handled by a reachability space 102 in the form of a cost volume parameterized by (x,y,θ) containing for each type of turn (e.g. including turn radius and forward/reverse gear) the lowest cost to be expended, to reach that pose state with the appropriate twist type as the final twist. This model can essentially use a four-dimensional State space work in which the fourth dimension contains the possible turn radii times the two gears.

The path planner 110 can go through the update steps that perform one turn at a time to update the state space from considering all paths up to n turns to n+1. For each twist type, for the (n+1). te turn is used, the trajectory estimator 150 can start with one of the other turn types and pay the cost of the transition first. The smallest of these possibilities can be considered the least expensive if one is willing to start a turn of this type from this pose, and the path planner 110 can only consider the most efficient one. Then the path planner 110 can traverse the turn to, after completion of the (n+1). th turn to achieve the lowest cost with this turn type. In one example, traversing N turns with K turn radii may require the processing of 2NK turns (e.g., assuming reverse requires a separate processing step). Therefore, before each turn, the trajectory estimator 150 can determine the minimum for the transition to the beginning of the turn from 2K cost volume. In one aspect of the disclosure, this step can be performed for all 2K turn types simultaneously, reading the 2K volumes, computing the (2K) ² transitions, finding the 2K minima, and writing back to the volumes at the same time. This processing updates the pre-turn cost to the post-turn cost for the next turn through memory accesses to 4NK volumes. Each turn can also read back the cost, read the free space, and write out the new cost, resulting in 6NK additional memory accesses to the volume. This approach can result in a total of 10 NK volume accesses.

The cost function can be further simplified, for example, by valuing the transition costs between turn types the same regardless of which types of turns are involved in the transition. In addition, the cost model may assume that the cost of making a turn is negligible compared to the transition cost. With this approach, the cost can be calculated based on the number of turns. Although this approach greatly simplifies operations and does not take into account some situations (e.g. two short turns with a transition that take less time than one long turn, or when the transition between two turns of similar radius in the same aisle is faster than a major change in steering or a gear change), this cost model can still represent a workable heuristic. In this simpler model, a three-dimensional state space can be used instead of a four-dimensional state space. A single state volume can now state whether or not a corresponding pose could be reached with n turns, corresponding or similar to the approach represented by pseudocode 830. With this simpler cost model, the processing of each turn can be gleaned from a common previous cost volume since the transition penalty is always paid and there is no need to remember which turn type was the last. The occupancy space is still read and the new cost is written out for the first turn type and combined by logical OR with the previous results for the following turn types, involving a read and a write. This corresponds to memory access to a total of 8NK - 2N volumes. In addition, the cost volume can be represented by a single bit per cell (reachable or not) instead of 8, 16 or even 32 bits, depending on the resolution of the cost function. The processing speed of many modern kernels is largely determined by the amount of memory accesses, and this approach can require a small amount of memory accesses.

Now on 9 Referring shows 9 Figure 9 shows an exemplary computation flow graph 900 that may be used to process idioms where reachability is encoded with a non-binary value, according to embodiments of the present invention. The computational flowchart 900 may be appropriate when the reachability evaluator 148 is a more general cost function than the computational flowchart 800 of FIG 8th used.

Like the computational flow graph 800, the computational flow graph 900 may include N iterations such as B. go through an iteration 902A and an iteration 902B, each time considering K types of trajectories (e.g. sharp left, slightly left, straight, slightly right, sharp right, etc.). Instead of using a shared reachability space, each iteration may include a cost update of the post-trajectory cost outputs from the previous iteration to generate a pre-trajectory cost for the iteration. For example, iteration 802B may include a cost update 820 of post-trajectory cost output spaces 910A, 912A, 914A, 916A, and 918A from iteration 902A to generate pre-trajectory cost output spaces 910B, 912B, 914B, 916B, and 918B for iteration 902B. In one or more embodiments, costs may follow the trajectory represent the lowest cost of reaching a pose ending with the corresponding trajectory type and cost before trajectories may represent the lowest cost of reaching the pose and being ready for the corresponding trajectory type without penalty. The turn kernel processing of an iteration can then convert the pre-trajectory cost to a post-trajectory cost, and the process can be repeated. For example, iteration 902B may include converting the pre-trajectory cost output spaces 910B, 912B, 914B, 916B, and 918B into corresponding post-trajectory cost output spaces, as shown. In 9 each vertical line represents a trajectory type rating.

9 8 includes pseudocode 930 to illustrate how a thread of reachability assessor 148 can traverse M pose cells of a trajectory with a core loop that operates on a more general cost function than pseudocode 830 that involves cost c. Here the variables are no longer bit vectors to capture non-binary cost values. In addition, the logical operations are replaced by max/min operations. Obstacles can be represented by maximum costs, which can be unchangeable, or they can be proportional to the size of the obstacle (e.g., if obstacles don't necessarily block the vehicle). In addition, the cost can increase by one at each step.

In one or more embodiments, an iteration can process each trajectory in both the forward and reverse directions. If it is cyclic, which is the case with turns, unless they are even, it can also be processed for two cycles. This is because it is not trivial to know a priori where a cycle should start, and a worst case is that reachability must be propagated from the last cell through an entire second cycle, although certain heuristics can be developed to take this into account. Therefore, the trajectories can be processed in four runs (sweeps) with code that does not have lengthy branch differences between threads to avoid thread divergences that impede concurrency. However, much of this processing can be avoided while still allowing for more parallelism by splitting each trajectory into any number of sections that can be processed in parallel.

By using the disclosed approaches to dividing a trajectory into sections that can be processed in parallel, much of the cost of running cyclic turns back and forth in two cycles can be avoided. In one or more embodiments, the reachability manager 122 divides the processing of one or more trajectories into independent parallel sections. The results between sections of a turn cannot be totally independent, as reachability can affect two cycles through the entire trajectory, both forward and backward. The disclosed approaches can handle this dependency with a concurrency reduction pattern that collects the results hierarchically, performs a small amount of processing on the collected results, and then propagates the results back as inputs to individual sections.

The results of a section may be independent of other sections, except for reachability, which occurs at the beginning of a section (or at the end if processing is in the reverse direction). With the disclosed approaches, the forward and/or backward reachability that occurs at the beginning or at the end of a section can be calculated, allowing independent processing of the sections with these inputs. For small chunks, the input occupancy space and reachability space can be loaded once, and both forward and backward traversals can be processed simultaneously, halving global memory accesses. This can be desirable since keeping the data in local registers or shared memory near the processing cores is usually much faster than accessing global memory.

To determine the forward and backward reachability entering each chunk, a processing pass can be performed that calculates for each chunk (e.g., in parallel): The forward reachability exiting the chunk in the forward direction, as found in the chunk originating (SRƒ), reverse reachability leaving the span in the reverse direction as it originated in the span (SRb), and span clearance, i. H. whether the section is entirely open space (SF).

The section-based forward and backward reachability that emerges from this flow is reachability that originates in this section. It may not be possible to locally detect the reachability that is propagated from outside the span, for example the reachability that enters at the beginning of the span and traverses it completely, because the span consists entirely of free space. This can be taken into account by calculating the section clearance so that these globally propagated effects can be determined in a much smaller processing flow that works with sections rather than individual cells.

wobei m die Abschnittslänge darstellt und s ein Index für einen Abschnitt ist.The following is example pseudocode for the processing pass, where a backward pass that is interspersed with the forward pass has been omitted:

where m represents the section length and s is an index for a section.

Another processing pass can be used that works similarly to the processing pass in sections with sections, but instead uses the reachability and free space of sections. This processing pass can loop through the section outputs in two (in the case of cyclic turns). In the first loop, the reachability that could be propagated can be "warmed up". The second loop can complete the full propagation for cyclic turns and write back the result. The following is example pseudo-code for the processing pass, showing only the forward pass as the backward pass is entirely analogous:

Even if this processing runs through the section outputs four times (e.g. in parallel), only 4/m times as much work is incurred in this step, since this decreases with longer sections (choice of a larger m).

Another processing pass of the section processing can be almost identical to the basic loop, with the change that the section reachability is used as input instead of starting the reachability at zero. The following is example pseudocode for the processing pass, where a backward pass has been omitted:

The processing pass of writing the result back can be changed to a logical OR with what was already there for each but the first phrase type. Section processing with the more general cost function works analogously, except that the reachability r can correspond to the lowest cost of exiting or entering a section.

Using a concurrency reduction pattern as described herein may involve 2 reads and 3/m writes per cell of the pose configuration space 112 for the processing pass involving the computation of reachability originating in a section. The processing pass involving the propagation of the section outputs may involve 8/m reads and 2/m writes. The processing pass that takes the section reachability as input can 2 (or 3) reads and 1 write. Therefore, the concurrency reduction pattern can be achieved in 5+8/m reads and 1+5/m writes, or a total of 6+13/m accesses. This can be compared to processing without chunks, which can be 2 reads and 1 write x 3 passes and 3 reads and 1 write on a final pass, or a total of 9 reads, 4 writes and 13 total memory accesses. This allows section processing to reduce memory accesses by more than a factor of two. In addition, further savings can be achieved by caching the free space or occupancy and reachability inputs in shared memory between the first and third passes, saving the two read operations on the third pass and the resulting 4 + 13 /m accesses to save memory accesses by a factor of more than three. The szq parallelism reduction pattern approach also exhibits more parallelism since there are M/m sections in each turn that can be processed in parallel.

As described herein, the trajectory estimator 150 of the path estimator 130 may use the reachability space(s) annotated by the reachability estimator 148 over one or more iterations to identify and/or select a path using the backtracker 152 . In one or more embodiments, the backtracker 152 may perform backtracking to find a path that has the smallest number of trajectories or turns (or more generally, the path of lowest cost) to reach a destination. The backtracker 152 may be executed by a CPU and/or with parallel processing (e.g., by at least one GPU).

In at least one embodiment, the backtracker 152 may search for one or more cells in a target set of one or more poses for which the final reachability output is set in the reachability space(s). For example, the backtracker 152 can loop over a target set of poses or, if parallelism is desired, use parallelism reduction. If there is more than one cell whose reachability is set, one or more poses can be chosen by a preference function, e.g. B. based on proximity to a selected pose. If there is no such pose, this may indicate that there is no path plan with Al turns that achieves the target set, and in this case a plan with (N+1) turns can be evaluated.

Once a cell has been chosen from which to start a backtrack, it can be assumed that the cell was reached after i iterations and one of the turn types reached the pose from another pose one iteration earlier (at iteration i - 1) has been reached. The backtracker 152 can therefore backtrack all K terminology types from this cell and will find somewhere along it a cell set to reachable in a reachability space output from iteration i-1. Because the turns have repeatable coordinate definitions, they can be traced back exactly as scored. This backtracking of K turns can be performed sequentially by a CPU or by concurrency reduction. A cyclic ambiguity may arise since a turn may have occurred in either the forward or reverse direction. The ambiguity can be resolved by examining the clearance along the traced turns and stopping the traceback when an obstacle is found (in embodiments where obstacles may completely block a path).

In the case of a more general cost function, the criterion can be to find the cell with the lowest cost among the cells for which the cost reduction corresponds to the cost of the traced turn. If more than one cell meets the criterion, a heuristic can be used to select a cell, such as B. Selecting the cell that can be reached with the shortest turn. If the set goal was reached in less than N turns, some of the initial backtracking steps will find a reachable cell from the previous iteration as the same cell (because that turning step was not required). The backtracker 152 can determine that this criterion is met and remove this turn from a path solution. Once a cell is found, the process can be repeated from that cell until a reachable cell is found after iteration 1. The backtracking can conclude with the current pose, which represents the reachable space before each turn, and can be performed with the current pose to resolve the cyclic ambiguity.

Now on 10 With reference, each block of the method 1000 described herein includes a computational process that may be performed with any combination of hardware, firmware, and/or software. For example, various functions may be performed by a processor executing instructions stored in memory. The methods may also be embodied in computer-executable instructions stored on computer storage media be. The methods may be provided by a standalone application, service, or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name just a few examples. Additionally, the methods are exemplified with respect to the path planner 110 of FIG 1A and 1B described. However, these methods may additionally or alternatively be performed by any system or combination of systems, including but not limited to those described herein.

10 10 is a flowchart showing a method 1000 for determining translated poses of a pose configuration space to determine a path through the pose configuration space, according to embodiments of the present invention. The method 1000 includes in block B1002 determining shifted poses based at least on the parallel shift of poses of a pose configuration space to generate shifted trajectories containing the shifted poses. For example, the pose translator 146 of the reachability manager 122 may translate poses of the pose configuration space 112 representing poses of an object (e.g., the vehicle 1300) in an environment based at least on a translation of a set of poses representing trajectories (e.g., B. turns 618 and 610 in 6B ) in pose configuration space 112 along at least one axis (e.g., the θ-axis) to generate shifted trajectories containing the shifted poses (e.g., turns 618 and 610 in 6C ).

The method 1000 includes in block B1004 determining a path through the pose configuration space based at least on the assessment of the reachability of the shifted poses by parallel processing of the shifted trajectories. For example, the path evaluator 130 can calculate a path from the current pose C in 1A to a target pose T in 1A based at least on the reachability evaluator 148, which evaluates the reachability of the shifted poses of the shifted trajectories from the first pose by parallel processing of the shifted trajectories (e.g. corresponding to pseudocode 830 or 930).

Now on 11 referring, is 11 Flowchart showing a method 1100 according to embodiments of the present invention for translating the trajectories of poses in a pose configuration space to translated trajectories to determine a path through the pose configuration space. The method 1100 includes, in block B1102, a parallel translation of trajectories from poses in a pose configuration space to translated trajectories that include one or more sections that are parallel to each other and to at least one axis of the pose configuration space. For example, the pose translator 146 may translate trajectories formed by poses in the pose configuration space 112 into translated trajectories that include at least portions that are parallel to each other and to at least one axis of the pose configuration space 112 .

In block B1104, the method 1100 includes the parallel processing of at least the portions of the trajectories shifted along the at least one axis in order to calculate reachability indicators for the shifted trajectories. For example, the reachability evaluator 148 may process at least the portions of the shifted trajectories along the at least one axis in parallel to compute reachability indicators for the shifted trajectories (e.g., corresponding to pseudocode 830 or 930).

The method 1100 includes, in block B1106, determining a path through the pose configuration space based at least on the reachability indicators. For example, the path evaluator 130 may determine a path through the pose configuration space 112 based at least on the reachability indicators.

12 FIG. 12 is a flowchart showing a method 1200 according to embodiments of the present invention for evaluating the reachability of sections of a trajectory in parallel. The method 1200 includes in block B1202 calculating, for a first section among the sections of a trajectory of poses in a pose configuration space, one or more reachability indicators upon exiting the first section. For example, the reachability evaluator 148 for a first section among the sections of the phrase 618 of 6B and 6C , which is formed by poses in the pose configuration space 112, calculate a reachability indicator when leaving the first section.

The method 1200 includes in block B1204 calculating a reachability indicator when entering a second section among the sections based on the reachability indicator when exiting the first section. For example, the reachability assessor 148 may include a reachability indicator calculate when entering a second section among the sections based on the accessibility indicator when leaving the first section.

The method 1200 includes, in block B1206, calculating a reachability indicator for a pose in the first leg based on the reachability indicator when entering the second leg. For example, the reachability evaluator 148 can provide a reachability indicator for a pose in the first section (e.g., relative to the current pose C in 1A ) calculated using the accessibility indicator when entering the second section. The method 1200 may be performed in parallel as part of a parallelism reduction pattern by parallel processing for a plurality of trajectories.

EXEMPLARY AUTONOMOUS VEHICLE

13A 13 is an illustration of an exemplary autonomous vehicle 1300 in accordance with embodiments of the present invention. The autonomous vehicle 1300 (alternatively referred to herein as “vehicle 1300”) may include, but is not limited to, a passenger vehicle, such as an automobile. a truck, bus, rescue vehicle, shuttle, electric or motorized bicycle, motorcycle, fire engine, police vehicle, ambulance, boat, construction vehicle, underwater vehicle, drone, trailer vehicle and/or another type of vehicle (e.g., an unmanned vehicle and/or a vehicle that accommodates one or more passengers). Autonomous vehicles are generally described in terms of automation levels defined by the National Highway Traffic Safety Administration (NHTSA), a division of the US Department of Transportation, and the Society of Automotive Engineers (SAE) "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806, published June 15, 2018, Standard No. J3016-201609, published September 30, 2016, and previous and future versions of this standard). The vehicle 1300 may be capable of performing functionalities corresponding to one or more of autonomous driving levels 3-5. For example, the vehicle 1300 may be capable of conditional automation (level 3), high automation (level 4), and/or full automation (level 5), depending on the embodiment.

The vehicle 1300 may include components such as a z. B. chassis, body, wheels (z. B. 2, 4, 6, 8, 18, etc.), tires, axles and other vehicle components include. The vehicle 1300 may include a propulsion system 1350 such as B. include an internal combustion engine, a hybrid electric drive, a pure electric motor and / or a drive system of another type. The propulsion system 1350 may be coupled to a powertrain of the vehicle 1300, which may include a transmission, to enable the vehicle 1300 to be propelled. The propulsion system 1350 may be controlled in response to receiving signals from a throttle/accelerator 1352 .

A steering system 1354, which may include a steering wheel, may be used to steer a vehicle 1300 (e.g., along a desired path or route) when the propulsion system 1350 is operational (e.g., when the vehicle is in motion). The steering system 1354 can receive signals from a steering actuator 1356 . The steering wheel can be optional for fully automatic functionality (level 5).

The brake sensor system 1346 may be used to apply the vehicle brakes in response to receiving signals from the brake actuators 1348 and/or brake sensors.

Controllers 1336, including but not limited to one or more systems on chip (SoCs) 1304 ( 13C ) and/or GPU(s) may provide signals (e.g., representing commands) to one or more components and/or systems of the vehicle 1300 . For example, the controllers may send signals to actuate the vehicle brakes via one or more brake actuators 1348, to actuate the steering system 1354 via one or more steering actuators 1356, and to actuate the powertrain system 1350 via one or more throttle/acceleration controllers 1352. Controllers 1336 may include one or more onboard (e.g., integrated) computing units (e.g., supercomputers) that process sensor signals and issue operational commands (e.g., signals representing commands) to enable autonomous driving and/or or assist a human driver in driving the vehicle 1300 . The controllers 1336 may include a first controller 1336 for autonomous driving functions, a second controller 1336 for functional safety functions, a third controller 1336 for artificial intelligence (e.g., computer vision) functionality, a fourth controller 1336 for infotainment functionality, a fifth controller 1336 for emergency redundancy and/or other controllers. In some examples, a single controller 1336 can perform two or more of the above functions functionalities, two or more controllers 1336 can assume a single functionality, and/or any combination thereof.

Controllers 1336 may provide the signals to control one or more components and/or systems of the vehicle 1300 in response to sensor data received from one or more sensors (eg, sensor inputs). The sensor data can be, for example, but not limited to, global navigation satellite system (GNSS) sensor(s) 1368 (e.g., GPS sensor(s)), RADAR sensor(s) 1360, ultrasonic sensor(s) 1362, LIDAR sensor(s) 1364, Inertial Measurement Unit (IMU) sensor(s) 1366 (e.g. accelerometer, gyroscope(s), magnetic compass(es), magnetometer, etc.), Microphone(s) 1396, Stereo camera( s) 1368, wide angle camera(s) 1370 (e.g., fisheye cameras), infrared camera(s) 1372, surround view camera(s) 1374 (e.g., 360 degree cameras), long range and/or mid range camera (s) 1398, speed sensor(s) 1344 (e.g. for measuring the speed of the vehicle 1300), vibration sensor(s) 1342, steering sensor(s) 1340, brake sensor(s) (e.g. as part of the brake sensor system 1346 ) and/or other sensor types.

One or more controllers 1336 may receive input (e.g., in the form of input data) from an instrument cluster 1332 of the vehicle 1300 and via a human-machine interface (HMI) display 1334, an audible indicator, a speaker, and/or or provide outputs (e.g., in the form of output data, display data, etc.) via other components of the vehicle 1300 . The outputs can include information such as B. Vehicle speed, RPM, map data (e.g. the HD map 1322 from 13C ), location data (e.g., location of vehicle 1300 on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and status of objects as received from controller(s) 1336 is detected, and so on. For example, the HMI display 1334 may display information about the presence of one or more objects (e.g., a road sign, a warning sign, a traffic light that is changing, etc.) and/or information about maneuvers that the vehicle has performed, is performing, or is performing (eg, change lanes now, take exit 34B in two miles, etc.).

The vehicle 1300 also includes a network interface 1324 that may use wireless antenna(s) 1326 and/or modem(s) to communicate over one or more networks. For example, network interface 1324 may enable communication over LTE, WCDMA, UMTS, GSM, CDMA2000, and so on. Wireless antenna(s) 1326 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) by using local area networks such as e.g. B. Bluetooth, Bluetooth LE Z-Wave, ZigBee etc. and / or low-power wide area networks (LPWANs) such. B. Use LoRaWAN, SigFox etc.

13B 13 is an example of camera positions and fields of view for the example autonomous vehicle 1300 of FIG 13A according to embodiments of the present invention. The cameras and their respective fields of view are exemplary and not limiting. For example, additional and/or alternative cameras may be included and/or cameras may be located at other vehicle 1300 locations.

Camera types for the cameras may include, but are not limited to, digital cameras that may be adapted for use with vehicle 1300 components and/or systems. The camera(s) may operate at automotive safety integration level (ASIL) B and/or other ASIL level. The camera types may be able to record an image rate such as e.g. 60 frames per second (fps), 120 fps, 240 fps, etc. The cameras can use rolling shutters, global shutters, another type of shutter, or a combination of these. In some examples, the color filter array may be a red-clear-clear-clear color filter array (RCCC), a red-clear-clear-blue color filter array (RCCB), a red-blue-green-clear color filter array (RBGC), a Foveon X3 color filter array, a Bayer sensor color filter array (RGGB), a monochrome sensor color filter array, and/or another type of color filter array. In some embodiments, clear pixel cameras such as e.g. B. Cameras with an RCCC, an RCCB and/or an RBGC color filter array can be used.

In some examples, one or more of the cameras may be used to perform adaptive driver assistance systems (ADAS) functions (e.g., as part of a redundant or failsafe design). For example, a multi-function mono camera can be installed to provide functions including Lane Departure Warning, Traffic Sign Assist, and Intelligent Headlight Control. One or several of the cameras (e.g. all cameras) can record and provide image data (e.g. video) at the same time

One or more of the cameras may be mounted in a mounting bracket, e.g. B. in a custom-designed (3D-printed) mount to eliminate stray light and reflections from the vehicle interior (e.g. reflections from the dashboard reflected in mirrors of the windshield) that could affect the image data collection capabilities of the camera. As for side mirror mounting brackets, the side mirror brackets can be custom 3D printed so that the camera mounting plate matches the shape of the side mirror. In some examples, camera(s) may be integrated into the side view mirror. For side view cameras, the cameras can also be integrated into the four pillars at each corner of the passenger compartment.

Cameras with a field of view that includes portions of the environment in front of the vehicle 1300 (e.g., forward-facing cameras) may be used for all-around vision to aid in detecting paths and obstacles in the forward direction and using one or more Controllers 1336 and/or control SoCs may provide information critical to establishing an occupancy grid and/or determining preferred vehicle paths. Forward-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including but not limited to emergency braking, pedestrian detection, and collision avoidance. Forward-facing cameras may also be used for ADAS functions and systems including, but not limited to, lane departure warning (LDW), autonomous cruise control (ACC), and/or other functions such as traffic sign recognition.

Various cameras can be used in a forward-facing configuration, e.g. B. a monocular camera platform with a CMOS (Complementary Metal Oxide Semiconductor) color image sensor. Another example is wide-angle cameras 1370, which can be used to detect objects coming into view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Even if in 13B only one wide-angle camera is shown, any number of wide-angle cameras 1370 may be present on vehicle 1300 . In addition, long-range cameras 1398 (e.g., a long-range stereo camera pair) can be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. Long range cameras 1398 can also be used for object detection and classification, as well as basic object tracking.

One or more stereo cameras 1368 may also be included in a forward-facing configuration. The stereo camera(s) 1368 may include an integrated controller with a scalable processing unit that can provide programmable logic (FPGA) and a multi-core microprocessor with an integrated CAN or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle's surroundings, including a distance estimate for all points in the image. Alternative 1368 stereo cameras may include compact stereo vision sensor(s) that may include two camera lenses (one each left and right) and an image processing chip that measures the distance from the vehicle to the target object and the information (e.g. metadata) generated can use to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1368 may be used in addition or as an alternative to those described herein.

Cameras with a field of view that includes portions of the environment to the side of the vehicle 1300 (e.g., side view cameras) may be used for the surround view and provide information used to create and update the occupancy grid and generate side impact alerts. For example, surround view camera(s) 1374 (e.g. four surround view cameras 1374 as in 13B shown) on the vehicle 1300 may be arranged. Perimeter cameras 1374 may include wide angle camera(s) 1370, fisheye camera(s), 360 degree camera(s), and/or the like. For example, four fisheye cameras may be located at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle may use three surround view cameras 1374 (e.g., left, right, and rear) and one or more other cameras (e.g., a front-facing camera) as the fourth surround view Use view camera.

Cameras with a field of view that includes portions of the environment behind the vehicle 1300 (e.g., rear view cameras) may be used for parking assistance, surround visibility, rear collision alerts, and to create and update occupancy grids. A wide range of cameras ver including but not limited to cameras that are also suitable as front-facing cameras (e.g., long-range and/or mid-range camera(s) 1398, stereo camera(s) 1368, infrared camera(s) 1372, etc .), as described here.

13C 13 is a block diagram of an example system architecture for the example autonomous vehicle 1300 of FIG 13A according to embodiments of the present invention. It should be understood that these and other arrangements described herein are exemplary only. Other arrangements and elements (e.g., engines, interfaces, functions, sequences, groups of functions, etc.) may be used in addition to or in place of those shown, and some elements may be omitted altogether. Furthermore, many of the elements described herein are functional units that can be implemented as single or distributed components, or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by devices may be performed by hardware, firmware, and/or software. For example, various functions may be performed by a processor executing instructions stored in memory.

All components, features and systems of the vehicle 1300 in 13C are shown connected via a bus 1302 . Bus 1302 may include a Controller Area Network (CAN) data interface (alternatively referred to herein as "CAN bus"). A CAN may be a vehicle 1300 resident network configured to support control of various vehicle 1300 features and functionality such as: B. actuation of the brakes, acceleration, braking, steering, wipers, etc. is used. A CAN bus can be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus can be read to determine steering wheel angle, vehicle speed, engine speed per minute (RPM), knob positions, and/or other vehicle condition indicators. The CAN bus can be ASIL B compliant.

Although the bus 1302 is described herein as a CAN bus, this is not meant to be limiting. For example, FlexRay and/or Ethernet can be used in addition to or as an alternative to the CAN bus. Although a single line is used to represent bus 1302, this is not intended to be limiting. For example, there may be any number of buses 1302, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and/or one or more other type buses with a different protocol. In some examples, two or more buses 1302 may be used to perform different functions and/or used for redundancy. For example, a first bus 1302 may be used for collision avoidance functionality and a second bus 1302 may be used for actuation control. In each example, each bus 1302 can communicate with any component of the vehicle 1300, and two or more buses 1302 can communicate with the same components. In some examples, each SoC 1304, controller 1336, and/or computer within the vehicle may have access to the same input data (e.g., inputs from vehicle 1300 sensors) and communicate with a common bus, such as a bus. B. be connected to the CAN bus.

The vehicle 1300 may include one or more controllers 1336 such as those referred to herein 13A have been described. Controllers 1336 can be used for a variety of functions. Controllers 1336 may be coupled to various other vehicle 1300 components and systems and used for vehicle 1300 control, vehicle 1300 artificial intelligence, vehicle 1300 infotainment, and/or the like.

The vehicle 1300 may include one or more systems on chip (SoC) 1304 . SoC 1304 may include CPU(s) 1306, GPU(s) 1308, processor(s) 1310, cache(s) 1312, accelerators 1314, memory 1316, and/or other components and features not shown. SoC(s) 1304 can be used in a variety of platforms and systems to control the vehicle 1300 . For example, SoC(s) 1304 in a system (e.g., vehicle's 1300 system) may be combined with an HD card 1322 that may be accessed via a network interface 1324 from one or more servers (e.g., server(s) 1378 from 13D ) may receive map refreshes and/or updates.

The CPU(s) 1306 may comprise a CPU cluster or CPU complex (also referred to herein as "CCPLEX"). CPU(s) 1306 may include multiple cores and/or L2 caches. For example, in some embodiments, CPU(s) 1306 may include eight cores in a coherent multiprocessor configuration. In some embodiments, CPU(s) 1306 may include four dual-core clusters, where each cluster has a dedicated L2 cache (e.g., a 2 MB L2 cache). The CPU(s) 1306 (e.g., the CCPLEX) may be configured to support cluster concurrent operation, such that any cluster combination of CPU(s) 1306 may be active at any time.

CPU(s) 1306 may implement power management functions, including one or more of the following: individual hardware blocks may be automatically clocked when idle to conserve dynamic power; each core clock can be clocked when the core is not actively executing instructions due to execution of WFI/WFE instructions; each core can be independently current controlled; each core cluster can be independently clocked if all cores are clocked or power-driven; and/or each core cluster may be independently current controlled if all cores are current controlled. CPU(s) 1306 may also implement an advanced power state management algorithm, where allowable power states and expected wake-up times are set, and the hardware/microcode determines the best power state for the core, cluster, and CCPLEX. The processor cores can support simplified power state entry sequences in software, offloading the workload to microcode.

GPU(s) 1308 may include an integrated GPU (alternatively referred to herein as an “iGPU”). GPU(s) 1308 can be programmable and efficient for parallel workloads. GPU(s) 1308 may use an extended tensor instruction set, in some examples. GPU(s) 1308 may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96 KB of storage capacity), and two or more streaming microprocessors may include one Share an L2 cache (for example, a 512 KB L2 cache). In some embodiments, GPU(s) 1308 may include at least eight streaming microprocessors. GPU(s) 1308 may use one or more computational programming interfaces (API(s)). Additionally, GPU(s) 1308 may use one or more parallel computing platforms and/or programming models (eg, NVIDIA's CUDA).

GPU(s) 1308 may be power optimized for optimal performance in automotive and embedded applications. GPU(s) 1308 may be fabricated on a fin field effect transistor (FinFET), for example. This is not limiting, however, and GPU(s) 1308 may be fabricated using other semiconductor fabrication techniques. Each streaming microprocessor may include a number of mixed-precision processor cores divided into multiple blocks. For example, 64 PF32 cores and 32 PF64 cores can be divided into four processor blocks, but are not limited to this. In such an example, each processor block can have 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit and/or allocated a 64K register file. In addition, streaming microprocessors can include independent parallel integer and floating point data paths to enable efficient execution of workloads with a mix of computation and addressing computation. Streaming microprocessors may include independent thread scheduling capability to allow finer grained synchronization and cooperation between parallel threads. Streaming microprocessors may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

GPU(s) 1308 may include high-bandwidth memory (HBM) and/or a 16GB HBM2 memory subsystem to provide peak memory bandwidth of about 900GB/second in some examples. In some examples, in addition to or as an alternative to HBM memory, synchronous graphics random access memory (SGRAM), such as B. a "Graphics double data rate type five synchronous random-access memory" (GDDR5) can be used,

GPU(s) 1308 may include unified memory technology with access counters to enable more accurate allocation of memory pages to the processor accessing them most frequently, thereby increasing the efficiency of memory areas shared by multiple processors. to increase. In some examples, address translation services (ATS) support may be used to allow GPU(s) 1308 direct access to CPU(s) 1306 page tables. In such examples, an address translation request may be sent to CPU(s) 1306 when the memory management unit (MMU) of GPU(s) 1308 fails. In response, GPU(s) 1308 may look up their page tables for a virtual-physical mapping for the address and return the translation to GPU(s) 1308. This allows unified memory technology to provide a single, unified virtual address space for the memory of both CPU(s) 1306 and also from GPU(s) 1308, thereby simplifying programming of GPU(s) 1308 and porting of applications to GPU(s) 1308.

Additionally, GPU(s) 1308 may include an access counter that may track the number of times GPU(s) 1308 have accessed the memory of other processors. The hit count can help ensure that memory pages are moved to the physical memory of the processor that is accessing the pages most frequently.

SoC(s) 1304 may include any number of cache(s) 1312, including those described herein. For example, cache(s) 1312 may include an L3 cache that is available to both CPU(s) 1306 and GPU(s) 1308 (e.g., because it is shared with both CPU(s) 1306 and GPU(s) 1308 is connected). Cache(s) 1312 may include a write-back cache, e.g. by using a cache coherency protocol (e.g. MEI, MESI, MSI, etc.) to track the state of rows. The L3 cache can be 4MB or larger depending on the implementation, although smaller cache sizes can also be used.

SoC(s) 1304 may include an arithmetic logic unit(s) (ALU(s)) that may be utilized in performing processing related to one of the many tasks or operations of the vehicle 1300, such as processing DNNs. Additionally, SoC(s) 1304 may include one or more floating point units (FPU(s)) - or other types of math coprocessors or numeric coprocessors - for performing math operations within the system. For example, SoC(s) 104 may include one or more FPUs integrated with one or more CPU(s) 1306 and/or GPU(s) 1308 as execution units.

SoC(s) 1304 may include one or more accelerators 1314 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, SoC(s) 1304 may include a hardware acceleration cluster, which may include optimized hardware accelerators and/or large on-chip memory. The large on-chip memory (e.g. 4MB SRAM) can enable the hardware acceleration cluster to accelerate neural networks and other calculations. The hardware acceleration cluster may be used to complement the GPU(s) 1308 and take over some tasks from the GPU(s) 1308 (e.g., to free up more cycles of the GPU(s) 1308 to perform other tasks). For example, accelerators 1314 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to be eligible for acceleration. The term "CNN" as used herein can encompass all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and Fast RCNNs (e.g., for object detection).

Accelerators 1314 (e.g., the hardware acceleration cluster) may include a deep learning accelerator (DLA). DLA(s) may include one or more tensor processing units (TPUs) that may be configured to provide an additional tens of trillion operations per second for deep learning applications and inferencing. TPUs can be accelerators configured and optimized to perform image processing functions (e.g. for CNNs, RCNNs, etc.). DLA(s) may also be optimized for a specific set of neural network types and floating point operations and inferencing. The design of the DLA(s) can offer more performance per millimeter than a general-purpose GPU and far exceeds the performance of a CPU. TPU(s) can perform multiple functions, including a single-instance convolution function, e.g. B. INT8, INT16 and FP16 data types are supported for both features and weights, as well as post-processor functions.

DLA(s) can quickly and efficiently execute neural networks, particularly CNNs, with processed or unprocessed data for various functions including, for example but not limited to: a CNN to identify and detect objects using data from camera sensors; a CNN for distance estimation using data from camera sensors; a CNN to detect and identify emergency vehicles using data from microphones; a CNN for facial recognition and identification of vehicle owners using data from camera sensors; and/or a CNN for security-related events.

DLA(s) may perform any function of GPU(s) 1308, and a developer may provide for use of an inferencing accelerator for each function of DLA(s) or GPU(s) 1308, for example. For example, the developer may concentrate processing of CNNs and floating point operations on DLA(s) and leave other functions to GPU(s) 1308 and/or other accelerators 1314.

Accelerators 1314 (e.g., hardware acceleration clusters) may include a programmable vision accelerator (PVA), also referred to herein as a computer vision accelerator. The PVA(s) may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and/or augmented reality (AR) and/or virtual reality (VR) applications . PVA(s) can offer a balance between performance and flexibility. For example, each PVA may include, but is not limited to, any number of reduced instruction set (RISC), direct memory access (DMA), and/or any number of vector processors.

RISC cores may interact with image sensors (e.g., image sensors of any of the cameras described herein), image signal processors, and/or the like. Each of the RISC cores can contain any amount of memory. RISC cores can use a number of protocols depending on the embodiment. In some examples, the RISC cores can run a real-time operating system (RTOS). RISC cores may be implemented with one or more integrated circuits, application specific integrated circuits (ASICs), and/or memory devices. the RISC cores may include an instruction cache and/or tightly coupled RAM, for example.

DMA may allow components of PVA(s) to access system memory independently of CPU(s) 1306. The DMA can support any number of features to optimize the PVA, including but not limited to supporting multi-dimensional addressing and/or circular addressing. In some examples, the DMA may support up to six or more dimensions of addressing, which may include, but are not limited to, block width, block height, block depth, horizontal block stagger, vertical block stagger, and/or depth stagger.

The vector processors may be programmable processors that may be configured to efficiently and flexibly perform computer vision algorithm programming and provide signal processing capabilities. In some examples, the PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, DMA engine(s) (eg, two DMA engines), and/or other peripherals. The vector processing subsystem may operate as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and/or vector memory (e.g., VMEM). A VPU core can be a digital signal processor such as e.g. B. a Single Instruction, Multiple Data (SIMD), Very Long Instruction Word (VLIW) digital signal processor. The combination of SIMD and VLIW can increase throughput and speed.

Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. Thus, in some examples, each of the vector processors can be configured to operate independently of other vector processors. In other examples, vector processors included in a particular PVA may be configured to use data parallelism. For example, multiple vector processors contained within a single PVA can run the same computer vision algorithm, but for different regions of an image. In other examples, vector processors included in a particular PVA may concurrently run different image processing algorithms on the same image, or even run different algorithms on consecutive images or portions of an image. Among other things, any number of PVAs can be included in the hardware acceleration cluster, and any number of vector processors can be included in each PVA. In addition, PVA(s) may include additional memory with Error Correcting Code (ECC) to increase overall system security.

Accelerator 1314 (e.g., the hardware acceleration cluster) may include a on-chip computer vision network and static random access memory (SRAM) to provide accelerators 1314 with high-bandwidth, low-latency SRAM. In some examples, the on-chip memory may include at least 4MB of SRAM, consisting of, for example but not limited to, eight field-configurable memory blocks accessible to both the PVA and the DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and the DLA can access the storage via a backbone that allows the PVA and the DLA to access the storage at high speed. The backbone may include a single-chip computer vision network that connects the PVA and DLA (e.g., via APB) to memory.

The on-chip computer vision network may include an interface that determines that both the PVA and DLA provide complete and valid signals before transmitting control signals/address/data. Such an interface may provide separate phases and separate channels for control signal/address/data transmission and burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, although other standards and protocols can be used.

In some examples, the SoC(s) 1304 may include a real-time ray tracing hardware accelerator, such as. B. in the U.S. Patent Application No. 16/101,232 which was filed on August 10, 2018. The real-time ray-tracing hardware accelerator can be used for fast and efficient determination of the position and extent of objects (e.g. within a world model), generation of real-time visualization simulations, RADAR signal interpretation, sound propagation synthesis and/or analysis, simulation by SONAR systems, for general wave propagation simulation, for comparison with LIDAR data for localization purposes and/or other functions and/or for other purposes. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

Accelerators 1314 (e.g., the hardware acceleration cluster) have a wide range of uses for autonomous driving. The PVA can be a programmable vision accelerator that can be used for key processing steps in ADAS and autonomous vehicles. The capabilities of PVAs are well suited to algorithmic domains that require predictable, low-power, and low-latency processing. In other words, the PVA is well suited for semi-dense or dense regular computations, even with small data sets that require predictable, low-latency, low-power runtimes. In the context of autonomous vehicle platforms, PVAs are therefore designed to run classical computer vision algorithms because they are efficient at object recognition and work mathematically with integers.

For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. A semi-global matching algorithm may be used in some examples, although this is not intended to be limiting. Many Level 3-5 autonomous driving applications require motion estimation/stereo matching while driving (e.g. motion structure, pedestrian detection, lane detection, etc.). the PVA can perform a computer stereo vision function based on inputs from two monocular cameras.

In some examples, the PVA can be used to realize a dense optical flow. For example by processing raw RADAR data (e.g. with a 4D Fast Fourier Transform) to provide processed RADAR data. In other examples, PVA is used for deep time-of-flight (ToF) processing, for example by processing raw time-of-flight data to provide processed time-of-flight data.

The DLA can be used to operate any type of control and driving safety enhancement network including, for example, a neural network that outputs a confidence measure for each object detection. Such a confidence value can be interpreted as a probability, or as providing a relative "weight" of each detection compared to other detections. This confidence value allows the system to make further decisions about which detections to consider as true positives and not false positives. For example, the system can set a confidence threshold and only consider detections that exceed the threshold as true positive detections. In an automatic emergency braking (AEB) system, false positive detections would result in the vehicle automatically performing emergency braking, which is obviously undesirable. Therefore, only the safest detections should be considered as triggers for emergency braking (AEB). The DLA can run a neural network to regress the confidence value. The neural network can take as input at least a subset of parameters such as: B., among others, bounding box dimensions, the floor area estimate obtained (e.g., from another subsystem), the output of IMU sensor(s) 1366 correlating to the orientation of the vehicle 1300, the distance, 3D - Position estimates of an object obtained from the neural network and/or other sensors (e.g. LIDAR sensor(s) 1364 or RADAR sensor(s) 1360).

SoC(s) 1304 may include data storage 1316 (e.g., memory). Data storage 1316 may be on-chip memory of SoC(s) 1304 that may store neural networks running on the GPU and/or the DLA. In some examples, the capacity of data stores 1316 may be large enough to store multiple neural network instances for redundancy and security. Data stores 1312 may include L2 or L3 cache(s) 1312. Data storage 1316 may refer to the working memory associated with the PVA, DLA, and/or other accelerators 1314 as described herein.

SoC(s) 1304 may include one or more processors 1310 (e.g., embedded processors). Processors 1310 may include a boot and power management processor, which may be a dedicated processor and subsystem that performs boot and power management functions and security enforcement related thereto. The boot and power management processor may be part of the SoC(s) 1304 boot sequence and provide runtime power management services. The boot and power management processor may provide clock and voltage programming, support for system transitions to low power states, control of SoC(s) 1304 thermals and temperature sensors, and/or control of SoC(s) 1304 power states. Each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC(s) 1304 can use the ring oscillators to detect the temperature of the CPU(s) 1306, GPU(s) 1308, and/or accelerator 1314. If temperatures are determined to exceed a threshold, the boot and power management processor will initiate a temperature fault routine and place SoC(s) 1304 in a low power state and/or place vehicle 1300 in a hold mode (e.g., place vehicle 1300 in a safe bring a stop).

Processors 1310 can also include a variety of embedded processors that can serve as an audio processing engine. The audio processing engine can be an audio subsystem that enables full hardware support for multi-channel audio over multiple interfaces and a wide and flexible range of audio I/O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital Signal processor with dedicated RAM.

Processors 1310 may also include an always-on processor engine that may provide necessary hardware functions to support low-power management of the sensors and wake-up use cases. The always-on processor engine may include a processor core, tightly coupled RAM, supporting peripherals (e.g., clocks and interrupt controllers), various I/O control peripherals, and routing logic.

Processors 1310 may also include a security cluster engine that includes a dedicated processor subsystem for security management of automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., clocks, an interrupt controller, etc.), and/or routing logic. In a security mode, the two or more cores can operate in a lockstep mode and function as a single core with comparison logic to detect differences between their operations.

Processors 1310 may also include a real-time camera engine, which may include a dedicated processor subsystem for real-time camera management.

Processors 1310 may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

Processors 1310 may include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to generate the final image for the playback window. The video image compositor may apply lens distortion correction to wide-angle camera(s) 1370, surround view camera(s) 1374, and/or indoor surveillance camera sensors. The interior surveillance camera sensor is preferably monitored by a neural network running on another instance of the Advanced SoC, configured to identify and respond to interior events. An interior system may perform lip reading to activate cellular service and make a call, dictate email, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or speak enable controlled surfing on the Internet. Certain features are only available to the driver when the vehicle is operating in an autonomous mode and are otherwise disabled.

The video image compositor may include improved temporal denoising for both spatial and temporal denoising. For example, when motion occurs in a video, noise reduction appropriately weights spatial information, thereby reducing the weight of information from adjacent frames. If there is no motion in an image or part of an image, the noise reduction performed by the video image compositor can use temporal information from the previous image to reduce noise in the current image.

The video image compositor can also be configured to perform stereo rectification on input stereo images. The video image compositor can also be used to render the user interface when the operating system desktop is in use and does not require GPUs 1308 to continuously render new interfaces. Even when GPUs 1308 are powered on and actively doing 3D rendering, the video image compositor can be used to offload GPUs 1308, thereby improving performance and responsiveness.

SoC(s) 1304 may also include a MIPI serial camera interface for receiving video and input from cameras, a high-speed interface, and/or a video input block that may be used for camera and related pixel input functions. SoC(s) 1304 may also include one or more input/output controllers that can be controlled by software and used to receive I/O signals that are not associated with a specific task.

SoC(s) 1304 may also include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management, and/or other devices. SoC(s) 1304 can be used to process data from cameras (e.g. via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g. LIDAR sensor(s) 1364, RADAR sensor(s) 1360, etc., which can be connected via ethernet), data from bus 1302 (e.g. speed of vehicle 1300, steering wheel position, etc.), data from GNSS sensor(s) 1358 (e.g. connected via ethernet or CAN bus) etc. are used. SoC(s) 1304 may also include dedicated high-performance mass storage controllers, which may include their own DMA engines and may be used to offload routine data management tasks from CPU(s) 1306.

SoC(s) 1304 can be an end-to-end platform with a flexible architecture that covers automation levels 3 through 5, thereby providing a comprehensive functional security architecture that efficiently leverages computer vision and ADAS techniques to address diversity and Ensure redundancy and provide a platform for a flexible, reliable driving software stack along with deep learning tools. SoC(s) 1304 can be faster, more reliable, and even more power and space efficient than traditional systems. For example, the accelerators 1314 in combination with CPU(s) 1306, GPU(s) 1308, and memory(s) 1316 can form a fast, efficient platform for Level 3-5 autonomous vehicles.

The technology therefore offers capabilities and functions that are not achievable with traditional systems. For example, computer vision algorithms can be executed on CPUs using a high-level programming language such as B. the C programming language can be configured to perform a variety of processing algorithms on a wide variety of visual data. However, CPUs are often unable to meet the performance requirements of many computer vision applications, such as B. those related to execution time and power consumption. In particular, many CPUs are unable to execute complex object detection algorithms in real-time, which is a requirement in in-vehicle ADAS applications and is a requirement in Level 3-5 autonomous vehicles.

In contrast to traditional systems, the technology described here, by providing a CPU complex, a GPU complex and a hardware acceleration cluster, makes it possible to execute multiple neural networks simultaneously and/or sequentially and to combine the results with each other to achieve level 3 autonomous driving functions. 5 to allow. For example, a CNN running on the DLA (Deep Learning Accelerator) or a dGPU (e.g. GPU(s) 1320) may include text and word recognition that enables the supercomputer to read traffic signs and to understand, including characters for which the neural network has not been specially trained. Of the DLA may also include a neural network capable of identifying, interpreting and semantically understanding characters and passing this semantic understanding to path planning engines running on CPU complex.

Another example is that multiple neural networks can run simultaneously as required for level 3, 4 or 5 driving. For example, a warning sign that reads "Caution: Flashing lights indicate black ice" together with an electric light can be interpreted independently or jointly by several neural networks. The sign itself can be identified as a road sign by a first deployed neural network (e.g., a trained neural network), the text "Flashing lights indicate black ice" can be interpreted by a second deployed neural network, which the vehicle's path planning software (the runs preferentially on the CPU complex) informed that there is black ice when blinking lights are detected. The blinker can be identified by running a third deployed neural network over multiple frames and informing the vehicle's path planning software of the presence (or absence) of blinkers. All three neural networks can run simultaneously, e.g. B. in the DLA and/or on GPU(s) 1308.

In some examples, a CNN may use data from camera sensors for facial recognition and vehicle owner identification to detect the presence of an authorized driver and/or owner of the vehicle 1300 . The always-on sensor processing engine can be used to unlock the vehicle when the owner approaches the driver's door and turn on the lights, and to disarm the vehicle in security mode when the owner exits the vehicle. In this way, SoC(s) 1304 provide security against theft and/or carjacking.

In another example, an emergency vehicle detection and identification CNN may use data from microphones 1396 to detect and identify emergency vehicle sirens. Unlike traditional systems that use general classifiers to detect sirens and manually extract features, the SoC(s) 1304 use the CNN to classify environmental and city sounds, as well as to classify visual data. In a preferred embodiment, the CNN running on a DLA is trained to detect the relative closing speed of an emergency vehicle (e.g., by utilizing the Doppler effect). The CNN may also be trained to identify emergency vehicles specific to the area as identified by GNSS sensor(s) 1358 . For example, in Europe, CNN will attempt to detect European sirens, and in the United States, CNN will attempt to identify only North American sirens. Once an emergency vehicle is detected, a control program can be used to run an emergency vehicle safety routine that uses ultrasonic sensors 1362 to slow the vehicle, pull to the curb, park the vehicle, and/or allow the vehicle to coast until emergency vehicles respond have passed.

The vehicle may include CPU(s) 1318 (e.g., discrete CPU(s) or dCPU(s)) that may be coupled to SoC(s) 1304 via a high-speed interconnect (e.g., PCIe). CPU(s) 1318 may include an X86 processor, for example. CPU(s) 1318 may be used to perform various functions including, for example, arbitrating potentially inconsistent results between ADAS sensors SoC(s) 1304 and/or monitoring the status and health of the controllers 1336 and/or the infotainment SoC 1330.

Vehicle 1300 may include CPU(s) 1320 (e.g., discrete CPU(s) or dCPU(s)) that may be coupled to SoC(s) 1304 via a high-speed interconnect (e.g., NVIDIA's NVLINK). GPU(s) 1320 may include additional AI functionality such as by implementing redundant and/or other neural networks, and may be used to train and/or update neural networks based at least in part on input (e.g., sensor data) from vehicle 1300 sensors.

The vehicle 1300 may also include a network interface 1324, which may include one or more wireless antennas 1326 (e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface 1324 may be used to establish a wireless connection over the Internet to the cloud (e.g., to servers 1378 and/or other network devices), to other vehicles, and/or to computing devices (e.g., passenger client devices). ) to allow. To communicate with other vehicles, a direct connection between two vehicles and/or an indirect connection (e.g. via networks and the Internet) can be established. Direct connections can be made via a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link may provide the vehicle 1300 with information about vehicles in the vicinity of the vehicle 1300 (e.g., vehicles in front of, beside, and/or behind the vehicle 1300). This functionality may be part of a cooperative vehicle 1300 adaptive cruise control function.

Network interface 1324 may include an SoC that provides modulation and demodulation functionality and allows controllers 1336 to communicate over wireless networks. The network interface 1324 may include a radio frequency front end for baseband to radio frequency upconversion and radio frequency to baseband downconversion. Frequency conversions are done by well-known methods and/or by super-heterodyne methods. In some examples, the radio frequency front end functionality may be provided by a separate chip. The network interface can include wireless functionalities for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN and/or other wireless protocols.

The vehicle 1300 may also include one or more data stores 1328, which may also include off-chip memory (e.g., off-SoC(s) 1304). Data storage 1328 may include one or more storage elements including RAM, SRAM, DRAM VRAM Flash, hard drives, and/or other components and/or devices capable of storing at least one bit of data.

The vehicle 1300 may also include GNSS sensor(s) 1358 . GNSS sensor(s) 1358 (e.g., GPS, supported GPS sensors, differential GPS (DGPS) sensors, etc.) to support mapping, perception, occupancy grid generation, and/or path planning functions. Any number of GNSS sensors 1358 can be used, e.g. B. and, to be limited to this, a GPS using a USB port with an ethernet-to-serial (e.g. RS-232) bridge.

The vehicle 1300 may also include RADAR sensor(s) 1360 . RADAR sensor(s) 1360 can be used by vehicle 1300 for long-range vehicle detection even in darkness and/or poor weather conditions. The RADAR functional safety levels can be ASIL B. RADAR sensor(s) 1360 may use CAN and/or bus 1302 (eg, to transmit data generated by RADAR sensor(s) 1360) for control and access to object tracking data, wherein access on raw data is done over Ethernet in some examples. A variety of RADAR sensor types can be used. For example, RADAR sensor(s) 1360 may be suitable for use as front, rear, and side RADAR. In some examples, pulse Doppler RADAR sensor(s) are used.

RADAR sensor(s) 1360 can have various configurations, e.g. long range and narrow field of view, short range and wide field of view, short range and side coverage, etc. In some examples, long range RADAR can be used for adaptive cruise control. Long range RADAR systems can provide a wide field of view, achieved by two or more independent scans e.g. B. is realized within a range of 250 m. RADAR sensor(s) 1360 can help distinguish between static and moving objects and can be used by ADAS systems for emergency braking and front collision warning. Long-range RADAR sensors may include monostatic multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interface. In a six antenna example, the four central antennas may produce a focused beam pattern configured to map the vehicle's 1300 environment at higher speeds with minimal interference from traffic in adjacent lanes. The other two antennas may expand the field of view to allow for quick detection of vehicles entering or exiting the vehicle 1300 lane.

For example, medium-range RADAR systems can have a range of up to 1360 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 1350 degrees (rear). ? Short-range RADAR systems may include, but are not limited to, RADAR sensors designed to be installed on both ends of the rear bumper. When installed at both ends of the rear bumper, such a RADAR sensor system can generate two beams that constantly monitor the blind spot at the rear and sides of the vehicle.

Short-range RADAR systems can be used in the ADAS system for blind spot detection and/or lane change assistance.

The vehicle 1300 may also include ultrasonic sensor(s) 1362 . Ultrasonic sensors 1362, which may be located at the front, rear, and/or sides of the vehicle 1300, may be used for parking assistance and/or to generate and update an occupancy grid. A wide variety of ultrasonic sensors 1362 can be used, and different ultrasonic sensors 1362 can be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensor(s) 1362 can operate at functional safety levels of ASIL B.

Vehicle 1300 may include LIDAR sensor(s) 1364 . LIDAR sensor(s) 1364 may be used for object and pedestrian detection, emergency braking, collision avoidance, and/or other functions. The LIDAR sensor(s) 1364 can operate at functional safety levels of ASIL B. In some examples, the vehicle 1300 may include multiple LIDAR sensors 1364 (e.g., two, four, six, etc.) that may use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

In some examples, LIDAR sensor(s) 1364 may be able to provide a list of objects and their distances for a 360 degree field of view. Commercially available LIDAR sensors 1364, for example, can have a range of approximately 1300 m, with an accuracy of 2 to 3 cm and support for a 1300 Mbps Ethernet connection. ? In some examples, one or more non-protruding LIDAR sensors 1364 may be used. In such examples, LIDAR sensor(s) 1364 may be implemented as a small device that may be embedded in the front, rear, sides, and/or corners of vehicle 1300 . LIDAR sensors 1364 in such examples can provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees with a range of 200 m, even for objects with low reflectivity. Front-mount LIDAR sensor(s) 1364 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

In some examples, LIDAR technologies such as B. 3D Flash LIDAR can be used. 3D Flash LIDAR uses a laser flash as a transmission source to illuminate the vehicle's surroundings up to a distance of approximately 200 m. A Flash LIDAR unit includes a receiver that records the travel time of the laser pulse and the reflected light in each pixel, which in turn corresponds to the distance from the vehicle to objects. Flash LIDAR can enable the generation of highly accurate and distortion-free images of the environment with each laser flash. In some examples, four flash LIDAR sensors may be deployed, one on each side of the vehicle 1300. Available 3D flash LIDAR systems include a solid state 3D staring array LIDAR camera that has no moving parts other than a fan (i.e., a non-scanning LIDAR device). The Flash LIDAR device can use a 5 nanosecond Class I (eye safe) laser pulse per frame and capture the reflected laser light in the form of point clouds in 3D space and with registered intensity data. By using flash LIDAR, and because flash LIDAR is a solid state device with no moving parts, LIDAR sensor(s) 1364 may be less susceptible to motion blur, vibration, and/or shock.

The vehicle may also include IMU sensor(s) 1366 . IMU sensor(s) 1366 may be located at the center of the rear axle of vehicle 1300 in some examples. IMU sensor(s) 1366 may include, for example but not limited to, accelerometers, magnetometers, gyroscope(s), magnetic compass(es), and/or other types of sensors. In some examples, such as For example, in six-axis applications, the IMU sensor(s) 1366 may include accelerometers and gyroscopes, while in nine-axis applications, the IMU sensor(s) 1366 may include accelerometers, gyroscopes, and magnetometers.

In some embodiments, IMU sensor(s) 1366 may be implemented as a miniaturized, high-performance GPS-based inertial navigation system (GPS/INS) that combines microelectromechanical systems (MEMS) with inertial sensors, a highly sensitive GPS receiver, and advanced Kalman filter algorithms to generate estimates of position, speed and orientation. In some examples, IMU sensor(s) 1366 can allow a vehicle 1300 to estimate heading without requiring input from a magnetic sensor by observing speed changes directly from GPS and correlating with IMU sensor(s) 1366 . In some examples, the IMU sensor(s) 1366 and the GNSS sensor(s) 1358 may be combined into a single integrated unit.

The vehicle may include microphone(s) 1396 located in and/or around the vehicle 1300 . The microphone(s) 1396 may be used, among other things, to detect and identify emergency vehicles.

The vehicle may also include any number of camera types, including stereo camera(s) 1368, wide-angle camera(s) 1370, infrared camera(s) 1372, surround-view camera(s) 1374, long-range and/or mid-range camera(s) 1398 and/or other camera types. The cameras can be used to capture image data around the entire perimeter of the vehicle 1300 . The types of cameras used are dependent on the vehicle 1300 embodiments and requirements, and any combination of camera types can be used to provide the required coverage around the vehicle 1300 . In addition, the number of cameras can vary depending on the embodiment. For example, the vehicle may include six, seven, ten, twelve, and/or other numbers of cameras. For example, but not limited to, the cameras may support Gigabit Multimedia Serial Link (GMSL) and/or Gigabit Ethernet. Each of the cameras is on here 13A and 13B Reference described in more detail.

The vehicle 1300 may also include one or more vibration sensors 1342 . Vibration sensor(s) 1342 can detect vibrations from components of the vehicle such as B. the axis (s) measure. For example, changes in vibration can indicate a change in the road surface. In another example, when two or more vibration sensors 1342 are used, differences between vibrations can be used to determine friction or slippage of the road surface (e.g., when the vibration difference is between a driven axle and a freely rotating axle ).

The vehicle 1300 may include an ADAS system 1338 . ADAS system 1338 may include an SoC in some examples. ADAS system 1338 can include autonomous/adaptive/automatic cruise control (ACC), cooperating adaptive cruise control (CACC), frontal collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane departure warning (LKA), a blind spot warning system (BSW), a rear cross traffic warning system (RCTW), a collision warning system (CWS), a lane departure warning system (LC) and/or other systems, features and/or functions.

ACC (cruise control) systems may use RADAR sensor(s) 1360, LIDAR sensor(s) 1364, and/or camera(s). ACC systems may include longitudinal ACC and/or transverse ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately ahead of vehicle 1300 and automatically adjusts speed to maintain a safe distance from vehicles in front. The lateral ACC performs distance maintenance and instructs the vehicle 1300 to change lanes if necessary. Cross ACC is interoperable with other ADAS applications such as B. LCA and CWS related.

CACC uses information from other vehicles, which may be received via network interface 1324 and/or radio antenna(s) 1326 via a wireless connection from other vehicles or indirectly via a network connection (e.g., via the Internet). Direct connections can be made through a vehicle-to-vehicle (V2V) communication link, while indirect connections can be made through an infrastructure-to-vehicle (12V) communication link. In general, the V2V communication scheme provides information about vehicles immediately ahead (e.g., vehicles immediately ahead and in the same lane as vehicle 1300), while the I2V communication scheme provides information about traffic further ahead. CACC systems can include both I2V and V2V information sources. Because of the information about vehicles in front of the vehicle 1300, CACC can be more reliable and has the potential to improve traffic flow and reduce traffic congestion.

FCW systems are designed to warn the driver of a hazard so the driver can take corrective action. FCW systems use a forward-looking camera and/or RADAR sensor(s) 1360 coupled to a dedicated processor, DSP, FPGA, and/or ASIC that is coupled with driver feedback such as B. a display, a speaker and / or a vibrating component is electrically coupled. FCW systems can issue a warning e.g. B. in the form of a tone, a visual warning, a vibration and / or a quick braking impulse.

AEB systems detect an imminent collision with another vehicle or object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. AEB systems may use the AEB system forward-looking camera(s) and/or RADAR sensor(s) 1360 coupled to a dedicated processor, DSP, FPGA, and/or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid the collision, and if the driver fails to take corrective action, the AEB system can automatically apply the brakes to mitigate the impact to prevent or at least mitigate the predicted collision. AEB systems may include techniques such as Dynamic Brake Assist and/or Imminent Impact Braking (CIB).

LDW systems give optical, acoustic and/or tactile warnings such as e.g. B. steering wheel or seat vibrations to warn the driver when the vehicle crosses 1300 lane markings. An LDW system will not activate if the driver indicates intention to leave the lane by using the turn signal. LDW systems may use forward-facing cameras coupled to a dedicated processor, DSP, FPGA, and/or ASIC that is coupled with driver feedback such as B. a display, a speaker and / or a vibrating component is electrically coupled.

LKA systems are a variant of LDW systems. LKA systems correct the vehicle 1300 through steering interventions or braking when the vehicle 1300 begins to leave the lane.

BSW systems detect and warn the driver of vehicles in a vehicle's blind spot. BSW systems can issue a visual, audible and/or tactile warning to indicate that it is unsafe to merge into or change lanes. The system can issue an additional warning if the driver activates a turn signal. BSW systems may utilize rear-facing camera(s) and/or RADAR sensor(s) 1360 coupled to a dedicated processor, DSP, FPGA, and/or ASIC that is coupled with driver feedback such as B. a display, a speaker and / or a vibrating component is electrically coupled.

RCTW systems can output a visual, audible, and/or tactile notification when an object is detected that is out of range of the rear view camera while reversing the vehicle 1300 . Some RCTW systems include an Automatic Emergency Braking (AEB) system to ensure vehicle brakes are applied to avoid an accident. RCTW systems may use one or more rear-facing RADAR sensors 1360 coupled to a dedicated processor, DSP, FPGA, and/or ASIC that is coupled with driver feedback such as B. a display, a speaker and / or a vibrating component is electrically coupled.

Traditional ADAS systems can be prone to false positives which, while annoying and distracting to the driver, are typically not catastrophic because traditional ADAS systems warn the driver and allow the driver to decide if a safety condition actually exists , and act accordingly. However, in an autonomous vehicle 1300, in the event of conflicting results, the vehicle 1300 must decide for itself whether to consider the result of a primary processor or a secondary processor (e.g., the first controller 1336 or the second controller 1336). For example, in some embodiments, ADAS system 1338 may be a backup and/or secondary computer that provides perception information to a backup computer rationality module. The backup computer rationality monitor can run redundant diverse software on hardware components in order to detect errors in perception and in dynamic driving tasks. ADAS system 1338 outputs may be forwarded to a monitoring MCU. If there are conflicting outputs from the primary and secondary processors, the monitoring MCU must determine how to resolve the conflict to ensure safe operation.

In some examples, the primary computer may be configured to provide the monitoring MCU with a confidence value indicative of the primary computer's confidence in the selected outcome. If the confidence value exceeds a threshold, the monitoring MCU can follow the direction of the primary computer, regardless of whether the secondary computer returns a conflicting or inconsistent result. If the confidence value does not meet the threshold and the primary and secondary computers report different results (e.g., in the event of a conflict), the monitoring MCU can arbitrate between the computers to determine the appropriate result.

The monitoring MCU may be configured to execute one or more neural networks that are trained and configured to determine conditions under which the secondary computer will issue false alarms based on the outputs of the primary computer and the secondary computer. This allows neural networks in the monitoring MCU to learn when the secondary computer's output is trustworthy and when it is not. For example, if the secondary computer is a RADAR-based FCW system, neural networks in the higher-level MCU can learn when the FCW system identifies metallic objects that actually pose no threat, such as metal objects. B. a drain grate or a manhole cover, which trigger an alarm. Accordingly, if the secondary computer is a camera-based LDW system, a neural network in the monitoring MCU can learn to override the LDW system when cyclists or pedestrians are present and lane departure is indeed the safest manoeuvre. In embodiments that include neural networks running on the monitoring MCU, the monitoring MCU may include at least one DLA or GPU with associated memory capable of executing the neural networks. In preferred embodiments, the monitoring MCU may comprise and/or be included as a component of the SoC(s) 1304 .

In other examples, ADAS system 1338 may include a secondary computer that performs ADAS functionalities using conventional computer vision rules. The secondary computer can use classic computer vision (if-then) rules, and the presence of neural networks in the monitoring MCU can increase reliability, safety and performance. For example, different implementation and the intentional disparity make the overall system more fault tolerant especially to faults caused by the software (or software-hardware interface) functionality. For example, if there is a software error in the software running on the primary computer and the non-identical software code on the secondary computer gives the same overall result, the monitoring MCU can be more confident that the overall result is correct and the error is in the primary computer's software or hardware did not cause any significant error.

In some examples, the output of the ADAS system 1338 may be input to the host processor's perception block and/or the host processor's dynamic driving task block. For example, if the ADAS system 1338 is displaying a collision accident warning due to an object immediately ahead, the perception block can use this information in identifying objects. In other examples, the secondary computer may have its own neural network that is trained and therefore reduces the risk of false positives (false alarms) as described herein.

The vehicle 1300 may also include the infotainment SoC 1330 (eg, an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 1330 can include a combination of hardware and software that can be used to provide the vehicle 1300 with audio (e.g., music, a PDA, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g. hands-free kit), network connectivity (e.g. LTE, WiFi etc.) and/or information services (e.g. navigation systems, parking assistance, a data radio system, vehicle-related information such as E.g. fuel level, total distance travelled, brake fluid level, oil level, door open/close, air filter information etc.). For example, the 1330 infotainment SoC can power radios, CD players, navigation systems, video players, USB and Bluetooth connectivity, carputers, in-car entertainment, Wi-Fi, steering wheel audio controls, hands-free calling, a heads-up display ( HUD), an HMI display 1334, a telematics device, a control panel (e.g., for controlling and/or interacting with various components, functions, and/or systems), and/or other components. The infotainment SoC 1330 may also be used to provide users of the vehicle with information (e.g., visually and/or audibly) such as e.g. B. Information from the ADAS system 1338, autonomous driving information such. B. planned vehicle maneuvers, trajectories, environment information (z. B. intersection information, vehicle information, road information, etc.) and / or other information.

The infotainment SoC 1330 may include GPU functionality. The infotainment SoC 1330 can communicate with other devices, systems, and/or components of the vehicle 1300 via the bus 1302 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1330 may be coupled to a monitoring MCU such that the infotainment system's GPU may perform some self-driving functions if the primary control unit(s) 1336 (e.g., the vehicle's primary and/or backup computer 1300) fail. In such an example, the infotainment SoC 1330 may place the vehicle 1300 in a hold mode, as described herein.

The vehicle 1300 may also include an instrument cluster 1332 (eg, a digital dashboard, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 1332 may include a controller and/or a supercomputer (eg, a discrete controller or supercomputer). The instrument cluster 1332 can include any number and combination of instruments such as B. Speedometer, fuel gauge, oil pressure gauge, tachometer, odometer, turn signals, shift position indicator, seat belt warning light(s), parking brake warning light(s), engine malfunction light(s), airbag system (SRS) information, lighting controls, security system controls, navigation information, etc. In some examples, information may be displayed and/or exchanged between the infotainment SoC 1330 and the instrument cluster 1332 . In other words, the instrument cluster 1332 may be included as part of the infotainment SoC 1330, or vice versa.

13D 13 is a system diagram for communication between cloud-based servers and the example autonomous vehicle 1300 of FIG 13A according to embodiments of the present invention. System 1376 may include servers 1378, networks 1390, and vehicles including vehicle 1300. Servers 1378 may include a variety of GPUs 1384(A)-1384(H) (collectively referred to herein as GPUs 1384), PCIe switches 1382(A)-1382(D) (collectively referred to herein as PCIe switches 1382), and/or or CPUs 1380(A)-1380(B) (collectively referred to herein as CPUs 1380). The 1384 GPUs, 1380 CPUs, and PCIe switches can be used with high-speed connections such as B., without being limited to the NVLink interfaces 1388 and/or PCIe connections 1386 developed by NVIDIA. In some examples, the GPUs 1384 are connected via an NVLink and/or NVSwitch SoC, and the GPUs 1384 and PCIe switches 1382 are connected via PCIe connections. Although eight GPUs 1384, two CPUs 1380, and two PCIe switches are shown, this is not intended to be limiting. Depending on the embodiment, each of the servers 1378 may include any number of GPUs 1384, CPUs 1380, and/or PCIe switches. For example, servers 1378 may include eight, sixteen, thirty-two, and/or more GPUs 1384.

Servers 1378 may receive image data over network(s) 1390 and from the vehicles showing unexpected or changing road conditions, such as traffic jams. B. show recently started road works. Servers 1378 may transmit neural networks 1392, updated neural networks 1392, and/or map information 1394, including information about traffic and road conditions, over network(s) 1390 and to the vehicles. The map information 1394 updates may be updates for an HD map 1322 such as e.g. B. information about construction sites, potholes, detours, flooding and / or other obstacles. In some examples, neural networks 1392, updated neural networks 1392, and/or map information 1394 may result from new training and/or experiences represented in data received from any number of vehicles in the area and/or based on training performed in performed at a data center (e.g., using servers 1378 and/or other servers).

The servers 1378 can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated by the vehicles and/or in a simulation (e.g. with a game engine). In some examples, the training data is tagged (e.g., when the associated neural network benefits from supervised learning) and/or undergoes other preprocessing, while in other examples, the training data is untagged and/or preprocessed (e.g., when the neural network does not require supervised learning). The training may be performed according to one or more classes of machine learning techniques, including but not limited to classes such as B.: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, diverse learning, representation learning (including sparse dictionary learning), rule-based machine learning, anomaly detection, and any variant or combination thereof. Once the machine learning models are trained, the machine learning models may be used by the vehicles (e.g., transmitted to the vehicles via network(s) 1390) and/or machine learning models may be used by server(s) 1378 to remotely monitor vehicles.

In some examples, servers 1378 may receive data from vehicles and apply the data to state-of-the-art neural networks in real-time for real-time intelligent inferencing. Servers 1378 may include deep learning supercomputers and/or GPU 1384 powered dedicated AI computers, such as B. the DGX and DGX Station machines developed by NVIDIA. However, in some examples, the servers 1378 may include a deep learning infrastructure using only CPU-backed data centers.

The servers 1378 deep leasing infrastructure may be able to perform the inferencing quickly and in real-time and use this capability to evaluate and check the state of processors, software and/or associated hardware in the vehicle 1300 . For example, in at least one embodiment, the deep learning infrastructure may receive periodic updates from the vehicle 1300, such as: B. a sequence of images and/or objects that the vehicle 1300 located in this sequence of images (e.g., via computer vision and/or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them to the objects identified by the vehicle 1300, and if the results do not match and the deep learning infrastructure concludes that the AI If the vehicle 1300 is malfunctioning, the server 1378 may send a signal to the vehicle 1300 instructing a fail-safe computer of the vehicle 1300 to take control, notify vehicle occupants, and perform a safe parking maneuver.

For inferencing, servers 1378 may include GPU(s) 1384 and one or more programmable inferencing accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-assisted servers and inferencing acceleration can enable real-time responsiveness. In other examples such as B. if the performance z. B. is less critical, CPU-based, FPGA-based and other processor-based servers can also be used for inferencing.

EXEMPLARY UNIT OF COMPUTER

14 14 is a block diagram of an exemplary computing unit 1400 suitable for use in embodiments of the present invention. The computing unit 1400 may include an interconnection system 1402 that directly or indirectly connects the following devices: memory 1404, one or more central processing units (CPUs) 1406, one or more graphics processing units (GPUs) 1408, a communications interface 1410, input/output (E /A) ports 1412, input/output components 1414, a power supply 1416, one or more presentation components 1418 (e.g., display(s)), and one or more logic units 1420. In at least one embodiment, computing unit(s) 1400 may be a or multiple virtual machines (VMs), and/or each of their components may include virtual components (e.g., virtual hardware components). As non-limiting examples, one or more of the GPUs 1408 may include one or more vGPUs, one or more of the CPUs 1406 may include one or more vCPUs, and/or one or more of the logical units 1420 may include one or more virtual logical units. As such, compute units 1400 may include discrete components (e.g., a full GPU dedicated to compute unit 1400), virtual components (e.g., a portion of a GPU dedicated to compute unit 1400), or a combination thereof .

Although the various blocks in 14 are shown connected to leads via the connection system 1402, this is not meant to be limiting and is for clarity only. For example, in some embodiments, a presentation component 1418, such as a a display device, may be considered an I/O component 1414 (e.g., if the display is a touch screen). As another example, CPUs 1406 and/or GPUs 1408 may include memory (e.g., memory 1404 may represent a storage device in addition to memory of GPUs 1408, CPUs 1406, and/or other components). In other words, the unit of account of 14 is only an example. Between categories such as "workstation", "server", "laptop", "desktop", "tablet", "client device", "mobile device", "handheld device", "game console", "electronic control unit (ECU)", "Virtual reality system" and/or other types of devices or systems are not distinguished, since they are all considered within the scope of the computing unit of 14 be viewed lying down.

Connection system 1402 may represent one or more connections or buses, such as B. an address bus, a data bus, a control bus or a combination thereof. Connection system 1402 may include one or more bus or link types, e.g. an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect (PCIe) bus Component Interconnect Express) and/or another bus or link type. In some embodiments, there are direct connections between components. For example, the CPU 1406 can be directly connected to the memory 1404. In addition, the CPU 1406 can be connected to the GPU 1408 directly. In the case of a direct or point-to-point connection between components, the connection system 1402 may include a PCIe link to establish the connection. In these examples, the computing unit 1400 need not include a PCI bus.

Memory 1404 may include various computer-readable media. Computer-readable media can be any available media that the computing device 1400 can access. Computer-readable media can include both volatile and non-volatile media, as well as removable and non-removable media. As a non-limiting example, the computer-readable media may include computer storage media and communications media.

Computer storage media may include both volatile and non-volatile and/or removable and non-removable data carriers used in any method or technology for storing information such as. B. computer-readable instructions, data structures, program modules or other data types can be implemented. For example, memory 1404 may store computer-readable instructions (e.g., those representing one or more programs and/or one or more program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, DVD or other optical storage, magnetic tape cartridge, magnetic tape, magnetic disk storage or other magnetic storage device, or any other Include a medium that can be used to store the desired information and that the computing unit 1400 can access. As used herein, computer storage media does not include signals per se.

The computer storage media typically may contain computer-readable instructions, data structures, program modules, or other types of data in a modulated data signal such as a signal. B. a carrier shaft or other transport mechanism embody and include any information delivery medium. The term "modulated data signal" means a signal that has one or more properties that are adjusted or changed to encode information into the signal. For example, but not limited to, computer storage media may include wired media such as B. a wired network or a direct wired connection and wireless media such. B. include acoustic, RF, infrared and other wireless media. Combinations of the above media also fall within the scope of computer-readable media.

The CPU(s) 1406 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1400 to perform one or more of the methods and/or processes described herein. CPU(s) 1406 may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing multiple software threads concurrently. CPU(s) 1406 may include any type of processor and may include different types of processors (e.g., processors with fewer cores for mobile devices and processors with more cores for servers) depending on the type of processing unit 1400 implemented. Depending on the type of computing unit 1400, the processor may be, for example, an Advanced RISC Machines (ARM) processor that uses Reduced Instruction Set Computing (RISC) or an x86 processor that uses Complex Instruction Set Computing (CISC). The computing unit 1400 may include one or more CPUs 1406 in addition to one or more microprocessors or additional co-processors such as e.g. B. include arithmetic coprocessors.

In addition to or in the alternative to CPU(s) 1406, GPU(s) 1408 may be configured to execute at least some of the computer-readable instructions for controlling one or more components of computing unit 1400 to perform one or more of the methods and/or processes described herein. One or more GPU(s) 1408 may be an integrated GPU (e.g., with one or more of the CPU(s) 1406) and/or one or more GPU(s) 1408 may be a discrete GPU. In embodiments, one or more GPU(s) 1408 may be a co-processor of one or more CPU(s) 1406. GPU(s) 1408 may be used by computing unit 1400 for graphics output (e.g., 3D graphics) or for performing general computations. GPU(s) 1408 can be used for general purpose computations in GPUs (GPGPU), for example. GPU(s) 1408 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPU(s) 1408 may generate pixel data for output images in response to rendering commands (e.g., rendering commands from CPU(s) 1406 received via a host interface). GPU(s) 1408 may include graphics memory such as B. a display memory for storing pixel data or other suitable data such. B. GPGPU data include. Display memory may be included as part of memory 1404 . GPU(s) 1408 may include two or more GPUs operating in parallel (e.g., via a link). The link can connect GPUs directly (e.g. using NVLINK) or connect via a switch (e.g. using NVSwitch). In combination with each other, each GPU can generate 1408 pixel data or each GPGPU data for different parts of an output or for different outputs (e.g. a first GPU for a first image and a second GPU for a second image). Each GPU can have its own memory or share memory with other GPUs.

In addition to or in the alternative to CPU(s) 1406 and/or GPU(s) 1408, logic unit(s) 1420 may be configured to execute at least some of the computer-readable instructions for controlling one or more components of computing unit 1400 to perform one or more of the herein carry out the procedures and/or processes described. In embodiments, CPU(s) 1406, GPU(s) 1408, and/or logic unit(s) 1420 may individually or collectively perform any combination of the methods, processes, and/or portions thereof. One or more of the logic units 1420 may be integrated into one or more CPU(s) 1406 and/or GPU(s) 1408, and/or one or more of the logic units 1420 may be standalone components or otherwise external to the CPU(s) 1406 and /or the GPU(s) 1408. In embodiments, one or more of the logic units 1420 may be a co-processor of one or more CPU(s) 1406 and/or one or more GPU(s) 1408 .

Examples of logic unit(s) 1420 include one or more computational cores and/or components thereof such as e.g. B. Data Processing Units (DPUs), Tensor Cores (TCs), Tensor Processing Units (TPUs), Pixel Visual Cores (PVCs), Vision Processing Units (VPUs), Graphics Processing Clusters (GPCs), Texture Processing Clusters (TPCs), Streaming -Multiprocessors (SMs), Tree Traversal Units (TTUs), AI Accelerators (AIAs), Deep Learning Accelerators (DLAs), Arithmetic Logic Units (ALUs), Application Specific Integrated Circuits (ASICs), Floating Point Units (FPUs), Input- /Output (I/O) elements, Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) elements, and/or the like.

Communications interface 1410 may include one or more receivers, transmitters, and/or transceivers that enable computing device 1400 to communicate with other computing devices over an electronic communications network, including wired and/or wireless communications. The communication interface 1410 may include components and functions that enable communication over a number of different networks, such as. B. wireless networks (e.g. Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g. communication via Ethernet or InfiniBand), low-power wide area networks (e.g . LoRaWAN, SigFox, etc.) and/or the Internet. In one or more embodiments, logical unit(s) 1420 and/or communication interface 1410 may include one or more data processing units (DPUs) to transmit data received over a network and/or over connection system 1402 directly to one or more GPU(s) 1408 ( e.g. a memory thereof).

I/O ports 1412 allow computing unit 1400 to logically communicate with other devices including I/O component 1414, presentation component(s) 1418, and/or other components, some of which may be incorporated into computing unit 1400 (e.g., integrated) can be to be coupled. Example I/O components 1414 include a microphone, mouse, keyboard, joystick, gamepad, game controller, satellite dish, scanner, printer, wireless device, and so on. The I/O components 1414 may provide a natural user interface (NUI) that processes a user's air gestures, speech, or other physiological input. In some cases, the inputs can be transmitted to an appropriate network element for further processing. A NUI may include any combination of speech recognition, pen recognition, face recognition, biometric recognition, both on-screen and off-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) in the context of a display of the computing unit 1400 to implement. The computing unit 1400 may include depth cameras such as e.g. B. stereoscopic camera systems, infrared camera systems, RGB camera systems, touch screen technology and combinations thereof for gesture detection and recognition and - control. Additionally, computing unit 1400 may include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit (IMU)) that enable detection of motion. In some examples, accelerometer or gyroscope output may be used by computing unit 1400 to render immersive augmented reality or virtual reality.

The power supply 1416 can be a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1416 may provide power to the computing device 1400 to enable the components of the computing device 1400 to operate.

The presentation component(s) 1418 can be a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other types of displays, or a combination thereof), speakers and/or other display components. The rendering component(s) 1418 may receive data from other components (e.g., GPU(s) 1408, CPU(s) 1406, DPUs, etc.) and display the data (e.g., as an image, video, sound, etc.) output.

EXEMPLARY DATA CENTER

15 15 illustrates an exemplary data center 1500 that may be used in at least one embodiment of the present invention. The data center 1500 may include a data center infrastructure layer 1510, a framework layer 1520, a software layer 1530, and/or an application layer 1540.

As in 15 As illustrated, the data center infrastructure layer 1510 may include a resource orchestrator (resource manager) 1512, clustered compute resources 1514, and node compute resources ("node RR") 1516(1)-1516(N), where "N" represents any positive integer . In at least one embodiment, node RRs 1516(1)-1516(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), Graphics processors or graphics processing units (GPUs), etc.), memory (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input/output (NW I/O) devices, network switches, virtual machines (VMs), power supply modules and/or cooling modules, etc. In some embodiments, one or more node RRs among node RRs 1516(1)-1516(N) may correspond to a server having one or more of the above computing resources. Additionally, in some embodiments, node RRs 1516(1)-15161(N) may include one or more virtual components such as B. include vGPUs, vCPUs and/or the like, and/or one or more of the node RR 1516(1)-1516(N) may correspond to a virtual machine (VM).

In at least one embodiment, clustered computing resources 1514 may include separate clusters of node RRs 1516 housed in one or more racks (not shown) or numerous racks in data centers in different geographic locations (also not shown). Separate groupings of Node RRs 1516 within grouped compute resources 1514 may include grouped compute, network, or storage resources that may be configured or designated to support one or more workloads. In at least one embodiment, multiple Node RRs 1516, including CPUs, GPUs, DPUs, and/or other processors, may be grouped in one or more racks to provide compute resources to support one or more workloads. The rack(s) may also include any number of power supply modules, cooling modules, and/or network switches in any combination.

The resource orchestrator 1512 may configure or otherwise control one or more node RRs 1516(1)-1516(N) and/or clustered computing resources 1514. In at least one embodiment, resource orchestrator 1512 may comprise a data center 1500 software design infrastructure (SDI) manager. The resource orchestrator 1512 may include hardware, software, or a combination thereof.

In at least one embodiment, as in 15 As shown, the framework layer 1520 may include a job scheduler 1533, a configuration manager 1534, a resource manager 1536, and/or a distributed file system 1538. The framework layer 1520 may include a system for supporting the software 1532 of the software layer 1530 and/or one or more application(s) 1542 of the application layer 1540. Software 1532 or application(s) 1542 may be web-based service software or applications such as B. include those provided by Amazon Web Services, Google Cloud and Microsoft Azure. The framework layer 1520 can be, but is not limited to, a free and open source software web application framework, such as Apache Spark™ (hereinafter "Spark"), which is designed to process large amounts of data (e.g., "Big Data") a distributed File system 1538 can use. In at least one embodiment, the job scheduler 1533 may include a Spark driver to facilitate scheduling of the workloads supported by different tiers of the data center 1500. The configuration manager 1534 may be able to configure various layers such as the software layer 1530 and the framework layer 1520 including Spark and the distributed file system 1538 to support processing of large amounts of data. The resource manager 1536 may be able to manage pooled or grouped computing resources allocated or allocated in support of the distributed file system 1538 and the job scheduler 1533 . In at least one embodiment, pooled or grouped computing resources may be grouped Computing resources 1514 at the infrastructure level 1510 of the data center. The resource manager 1536 can coordinate with the resource orchestrator 1512 to manage these allocated or assigned computing resources.

In at least one embodiment, the software 1532 included in the software layer 1530 may include software generated by at least a portion of the node RRs 1516(1)-1516(N), the clustered computing resources 1514, and/or the distributed file system 1538 of the framework layer 1520 is used. One or more types of software may include, but are not limited to, Internet web site search software, e-mail virus scanning software, database software, and video content streaming software.

In at least one embodiment, the application(s) 1542 contained in the application layer 1540 may include one or more application types that are governed by at least a portion of the node RR 1516(1)-1516(N), the clustered computing resources 1514, and/or the distributed Framework level file system 1538 1520 can be used. Application types can include, but are not limited to, any number of applications such as: B. a genomics application, cognitive computing, and a machine learning application, including training or inferencing software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and/or other machine learning applications, used in connection with one or more embodiments.

In at least one embodiment, configuration manager 1534, resource manager 1536, and resource orchestrator 1512 may implement any number and type of self-modifying actions based on any amount and type of data collected in any technically feasible manner. Self-modifying actions can relieve a data center operator of the data center 1500 from potentially making bad configuration decisions to avoid underutilized and/or malfunctioning parts of a data center.

The data center 1500 may include tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, one or more machine learning models may be trained by calculating weight parameters of a neural network architecture using software and/or computing resources described above with respect to data center 1500 . In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1500 using weighting parameters defined by one or more Training techniques such as, but not limited to, those described herein have been calculated.

In at least one embodiment, the data center 1500 may use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and/or other hardware (or equivalent virtual computing resources) to perform training and/or inferring (deriving) using the resources described above . Additionally, one or more of the software and/or hardware resources described above may be configured as a service that allows users to view information such as B. to train or infer image recognition, speech recognition or other AI services.

EXEMPLARY NETWORK ENVIRONMENTS

Network environments suitable for use in implementing embodiments of the invention may include one or more client devices, servers, network attached storage (NAS), other backend devices, and/or other types of devices. Client devices, servers, and/or other device types (e.g., any device) may reside in one or more instances of the computing unit(s) 1400 of 14 be implemented - e.g. e.g., each device may include similar components, features, and/or functionality of the computing unit(s) 1400. Additionally, when backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may be part of a data center 1500 referred to herein 15 was described as an example.

The components of a network environment may communicate with one another over one or more networks, which may be wired, wireless, or both. The network can have multiple network works or a network of networks. For example, the network may be one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet and/or a public switched telephone network (PSTN) and/or one or more private networks. If the network comprises a wireless telecommunications network, components such as e.g. B. a base station, a communication tower or even access points (as well as other components) provide wireless connectivity.

Compatible network environments may include one or more peer-to-peer network environments - in which case a network environment may not contain a server - and one or more client-server network environments - in which case a network environment may contain one or more servers be - include. In peer-to-peer network environments, the functionality described herein in relation to one or more servers may be implemented on any number of client devices.

In at least one embodiment, a network environment may include one or more cloud-based network environments, a distributed computing environment, a combination thereof, and so on. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and/or edge servers. A framework layer may include a system to support software of a software layer and/or one or more applications of an application layer. The software or application(s) may also include web-based service software or applications. In embodiments, one or more client devices may utilize web-based service software or applications (e.g., by accessing service software and/or applications through one or more application programming interfaces (APIs)). The framework layer can include, but is not limited to, some type of free and open source software web application framework such as For example, be one that uses a distributed file system to process large amounts of data (e.g., "big data").

A cloud-based network environment may provide cloud computing and/or cloud storage that performs any combination of the computing and/or data storage functions (or one or more portions thereof) described herein. Each of these various functions may be distributed across central or core servers in multiple locations (e.g., in one or more data centers that may be distributed within a state, region, country, worldwide, etc.). When a connection to a user (e.g., a client device) is relatively close to edge server(s), a core server may assign at least some functionality to an edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and/or a combination thereof (e.g., a hybrid cloud environment).

Client device(s) may include at least some of the components, features, and functionality of the example computing unit(s) 1400 described herein with reference to FIG 14 was described. A client device can be, for example but not limited to, a personal computer (PC), laptop, mobile device, smartphone, tablet computer, smart watch, wearable computer, personal digital assistant (PDA) , an MP3 player, a virtual reality headset, a global positioning system (GPS) or GPS device, a video player, video camera, surveillance device or system, vehicle, boat, flying craft, virtual machine, a drone, a robot, a wearable communication device, a clinical device, a gaming device or system, an entertainment system, a vehicle computing system, an embedded system controller, a remote control, an accessory, a consumer electronic device, a workstation, an edge device, and a any combination of these devices described, or any other suitable device.

The invention may be used in the general context of computer code or machine-readable instructions, including computer-executable instructions such as B. program modules are described, which of a computer or other machine such. B. a Personal Data Assistant (PDA) or other handheld device. Generally, program modules, including routines, programs, objects, components, structures, etc., refer to code that performs specific tasks or implements specific abstract data types. The invention can be implemented in a variety of system configurations including handheld devices, consumer electronics, general purpose computers, special purpose computers, and so forth. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing units that are linked through a communications network.

Here, when two or more elements are connected by "and/or", this is to be construed as only one element or a combination of elements. For example, "Item A, Item B, and/or Item C" can include only Item A, only Item B, only Item C, Item A and Item B, Item A and Item C, Item B and Item C, or Items A, B and C include. Furthermore, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

The subject matter of the present invention is described herein in a specific manner in order to meet legal requirements. The description is not intended to limit the scope of this invention. Rather, the inventors contemplated that the claimed subject matter could be implemented in other ways to include various steps or combinations of steps comparable to those described herein in conjunction with other present or future technologies. Although the terms "step" and/or "block" may have been used herein to denote various elements of the methods used, the terms should not be construed to imply any particular order among or between various steps disclosed herein, es unless the order of individual steps is explicitly described.

QUOTES INCLUDED IN DESCRIPTION

This list of the documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• US16101232 [0131]

Claims (20)

Method comprising: determining translated poses of a pose configuration space representing poses of a machine in an environment based at least on the parallel translation of a set of the poses corresponding to trajectories in the pose configuration space along at least one axis to generate translated trajectories containing the translated poses; determining a path from a first pose among the poses to a second pose among the poses based at least on the evaluation of reachability of the shifted poses from the first pose by parallel processing the shifted trajectories; and Causing one or more operations of a machine based on the determined path. procedure after claim 1 , wherein the shifted trajectories are parallel to each other along the at least one axis. procedure after claim 1 or 2 , wherein the at least one axis includes an axis representing an orientation of the object in the environment. A method according to any one of the preceding claims, wherein each trajectory among the trajectories comprises a turn having a fixed radius of turn throughout the trajectory. A method according to any preceding claim, wherein the parallel processing comprises evaluating a first subset of the shifted poses of a shifted trajectory among the shifted trajectories in parallel with a second subset of the shifted poses of the shifted trajectory. A method according to any preceding claim, wherein the parallel processing comprises evaluating a first subset of the shifted poses of a first shifted trajectory among the shifted trajectories in parallel with a second subset of the shifted poses of a second shifted trajectory among the shifted trajectories. A method according to any preceding claim, wherein the reachability is processed by a thread of a processor as a bit vector and each bit of the bit vector corresponds to a pose among the shifted poses and a trajectory among the shifted trajectories. A method according to any one of the preceding claims, wherein the parallel translation of the set of poses comprises accessing the pose configuration space using an access pattern that defines the translated poses at least during the reachability evaluation. Method according to one of the preceding claims, wherein the parallel translation of the set of poses generating, from the pose configuration space, a transformed pose configuration space containing the translated poses of the translated trajectories in memory, and evaluating the reachability of the translated poses using the transformed pose configuration space in memory included. System comprising: one or more processors; and one or more memory devices storing instructions that, when executed by one or more processors, cause the processor(s) to perform a method comprising: parallel translation of trajectories of a machine in a pose configuration space to translated trajectories, the translated trajectories comprising at least portions of the trajectories of the machine that are parallel to each other and to at least one axis of the pose configuration space; parallel processing of at least portions of the trajectories shifted along the at least one axis to calculate reachability indicators for the shifted trajectories; determining a path through the pose configuration space based at least on the reachability indicators; and Performing, based on the determined path, one or more operations corresponding to a behavior of the machine. system after claim 10 , where the pose configuration space is parameterized at least by (x, y, θ), where (x, y) represents a position of the object in a two-dimensional (2D) plane and θ represents a heading angle of the object at the position. system after claim 10 or 11 , wherein the reachability indicators comprise costs stored in one or more cost spaces parameterized at least by the poses of the pose configuration space, and determining the path includes tracing back from a first pose of the path to a second pose of the path using the or the includes cost areas. system according to one of the Claims 10 until 12 , further comprising: determining the clearance of the poses based at least on a collision check of a body of the object taking the poses against one or more obstacles; and storing indicators of free space in one or more occupancy spaces parameterized at least by the poses of the pose configuration space, the reachability indicators being based at least on the evaluation of the indicators of free space in the or each occupancy space against the shifted trajectories. system according to one of the Claims 10 until 13 wherein determining the path further comprises: calculating a first set of reachability indicators in a first iteration corresponding to a first pass of the shifted trajectories; and calculating, from the first set of reachability indicators, a second set of reachability indicators in a second iteration corresponding to a second run of the shifted trajectories. system according to one of the Claims 10 until 14 wherein the system is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented with an edge device; a system implemented with a robot; a system that integrates one or more virtual machines (VMs); a system implemented at least in part in a data center; or a system implemented at least in part with cloud computing resources. Processor comprising: one or more circuits for evaluating the reachability, by an object, of translated poses of a translated trajectory by parallel processing of sections of the translated trajectory based at least on the parallel translation of poses of a pose configuration space corresponding to a trajectory along at least one axis, and to determine a path of the object from a first pose in the pose configuration space to a second pose in the pose configuration space, at least based on the reachability of the shifted poses of the trajectories. processor after Claim 16 wherein the parallel processing of the sections comprises: calculating, for a first section among the sections, a reachability indicator upon exiting the first section as originating in the first section; calculating an availability indicator when entering a second section among the sections based on the availability indicator when exiting the first section; and calculating a reachability indicator for one or more of the displaced poses in the first section from the first pose based on the reachability indicator upon entering the second section. processor after Claim 16 or 17 , wherein the reachability indicator(s) for the displaced pose(s) comprises one or more costs associated with the object reaching the displaced pose(s). Processor after one of Claims 16 until 18 , where the reachability of the shifted poses is evaluated with respect to the first pose. Processor after one of Claims 16 until 19 , where the trajectory comprises a circle on a base of the environment formed by a turn having a fixed radius of turn throughout the trajectory.

Images