DE112022000498T5 - TRAINING PERCEPTION MODELS USING SYNTHETIC DATA FOR AUTONOMOUS SYSTEMS AND APPLICATIONS - Google Patents

Abstract

Various examples disclose systems and methods that use domain adaptation theory to minimize the reality gap between simulated and real domains for training machine learning models. For example, spatial priority sampling can be used to produce synthetic data that more closely matches the diversity of real-world data. To use this synthetic data to train models that perform well in the real world, the systems and methods of the present disclosure may use a discriminator that allows a model to learn domain-invariant representations to measure the divergence between the virtual world and the to minimize the real world in a latent space. The techniques described here enable a principled approach to learning neural-invariant representations and a theoretically inspired approach for samples from a simulator, which in combination enable the training of machine learning models with synthetic data.

Description

STATE OF THE ART

Autonomous and semi-autonomous driving systems, including advanced driver assistance systems (ADAS), can use sensors - such as cameras, LiDAR sensors and RADAR sensors - to form an understanding of the vehicle's surroundings. This understanding may include information about the location of objects, obstacles, traffic signs, road surfaces and/or other markings. Autonomous driving systems rely on machine learning models and/or neural networks to support the processes of information gathering and decision making. An effective machine learning model or neural network requires training with collections of real and/or synthetic training data.

A common strategy for training perceptual models is to use data collection vehicles that take massive amounts of real data, hire large numbers of annotators to label the data, and then train the models on that data using supervised learning. Although this approach is likely to be asymptotically successful, the financial and time costs increase with the amount of data to be recorded and labeled. Furthermore, changing the sensors can result in the effort having to be redone on a large scale. Some tasks, such as labeling ambiguous, distant, or obscured objects, may be difficult or even impossible for humans.

In comparison, collecting data from self-driving simulators has several advantages. First, a user has control over the content of a simulator, allowing all self-driving scenarios to be efficiently generated, regardless of how rare the event may be in the real world. Second, all information about the state of the world is known in the simulator, allowing near-perfect labels to be created for information that humans may not be able to correctly assign, as well as labels for completely occluded objects that are difficult or impossible to identify outside of the simulation would be labeled. Finally, extrinsic and intrinsic sensors are easier to control in a simulator, allowing data to be recorded for a fixed scenario using any sensor equipment. Given these properties of the simulation, it would be valuable to train operational perception models using data obtained from a simulation of the self-driving car.

However, there are two critical challenges when using a driving simulator. First, the sensor models in the simulation do not perfectly represent the sensors in the real world. Even with accurate calibration of the extrinsics and intrinsics, it is extremely difficult to perfectly label all the materials in the scene and to model complicated interactions between sensors and objects. Second, it is difficult to integrate real-world content into the simulation. The layout of the world may be different, the objects may not be diverse enough, and synthetic materials, textures, and weather conditions may not be well aligned with their real-world counterparts. Furthermore, machine learning models - and neural networks in particular - are very sensitive to these discrepancies and are known to generalize poorly from a virtual domain to a real domain.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure relate to training perceptual models using synthetic data for autonomous systems and applications. Systems and methods are disclosed that utilize domain adaptation theory to minimize the reality gap between simulated and real domains. For example, the system and methods of the present disclosure use sampling spatial priorities - e.g. B. independent of the structure of the road - to produce synthetic data that more closely matches the diversity of real-world data, to take into account the disadvantages of previous solutions. To train models on synthetic data that perform well in the real world, the systems and methods of the present disclosure use a discriminator that allows a model to learn domain-invariant representations to determine the divergence between the virtual world and the to minimize the real world in a latent space. By minimizing divergence in a representation space, the domain adaptation algorithm can be independent of different sensors and architectures. To further improve training, pseudo labels can be created using confidence Levels in the model's predictions are created to confront the model with real data during training. Thus, the techniques described here enable a principled approach to learning neural-invariant representations and a theoretically inspired approach to samples from a simulator, which in combination enable the training of machine learning models - and in particular neural networks - using simulated data. In this way, the amount of computational and time resources required to train an accurate and precise model for use in the real world is significantly reduced with respect to traditional training approaches.

BRIEF DESCRIPTION OF THE DRAWINGS

• 1 ist eine Veranschaulichung eines beispielhaften Datenflussdiagramms für einen Prozess des Trainings eines maschinellen Lernmodells unter Verwendung synthetischer Daten gemäß einigen Ausführungsformen der vorliegenden Offenbarung;
• 2 beinhaltet gemäß einigen Ausführungsformen der vorliegenden Offenbarung Veranschaulichungen von beispielhaften induzierten Randbedingungen für verschiedene Stichprobenstrategien;
• 3A beinhaltet gemäß einigen Ausführungsformen der vorliegenden Offenbarung Beispielvisualisierungen für die Verwendung verschiedener Stichprobenstrategien in einem Simulator;
• 3B veranschaulicht Diagramme, die gemäß einigen Ausführungsformen der vorliegenden Offenbarung verschiedenen Parametern zur Erzeugung von Szenen in einem Simulator entsprechen;
• 4 veranschaulicht das Training eines maschinellen Lernmodells unter Verwendung synthetischer Daten gemäß einigen Ausführungsformen der vorliegenden Offenbarung;
• 5 ist ein Flussdiagramm, das ein Verfahren zum Trainieren eines maschinellen Lernmodells unter Verwendung synthetischer Daten gemäß einigen Ausführungsformen der vorliegenden Offenbarung zeigt;
• 6A ist eine Veranschaulichung eines beispielhaften autonomen Fahrzeugs gemäß einigen Ausführungsformen der vorliegenden Offenbarung;
• 6B ist ein Beispiel für Kamerapositionen und Sichtfelder des beispielhaften autonomen Fahrzeugs aus 6A gemäß einigen Ausführungsformen der vorliegenden Offenbarung;
• 6C ist ein Blockdiagramm einer beispielhaften Systemarchitektur für das beispielhafte autonome Fahrzeug aus 6A gemäß einigen Ausführungsformen der vorliegenden Offenbarung;
• 6D ist ein Systemdiagramm für die Kommunikation zwischen (einem) cloudbasierten Server(n) und dem beispielhaften autonomen Fahrzeug aus 6A gemäß einigen Ausführungsformen der vorliegenden Offenbarung;
• 7 ist ein Blockdiagramm einer beispielhaften Rechenvorrichtung, die zur Verwendung bei der Implementierung einiger Ausführungsformen der vorliegenden Offenbarung geeignet ist; und
• 8 ist ein Blockdiagramm eines beispielhaften Rechenzentrums, das zur Verwendung bei der Implementierung einiger Ausführungsformen der vorliegenden Offenbarung geeignet ist.

The present systems and methods for training perception models using synthetic data for autonomous systems and applications are described in detail below with reference to the appended figures, provided that:

• 1 is an illustration of an example data flow diagram for a process of training a machine learning model using synthetic data, according to some embodiments of the present disclosure;
• 2 includes illustrations of example induced constraints for various sampling strategies, according to some embodiments of the present disclosure;
• 3A includes example visualizations for using various sampling strategies in a simulator, according to some embodiments of the present disclosure;
• 3B illustrates diagrams corresponding to various parameters for generating scenes in a simulator, according to some embodiments of the present disclosure;
• 4 illustrates training a machine learning model using synthetic data in accordance with some embodiments of the present disclosure;
• 5 is a flowchart depicting a method for training a machine learning model using synthetic data in accordance with some embodiments of the present disclosure;
• 6A is an illustration of an example autonomous vehicle in accordance with some embodiments of the present disclosure;
• 6B is an example of camera positions and fields of view of the exemplary autonomous vehicle 6A according to some embodiments of the present disclosure;
• 6C is a block diagram of an example system architecture for the example autonomous vehicle 6A according to some embodiments of the present disclosure;
• 6D is a system diagram for communication between cloud-based server(s) and the example autonomous vehicle 6A according to some embodiments of the present disclosure;
• 7 is a block diagram of an example computing device suitable for use in implementing some embodiments of the present disclosure; and
• 8th is a block diagram of an example data center suitable for use in implementing some embodiments of the present disclosure.

DETAILED DESCRIPTION OF THE INVENTION

Systems and methods related to training perceptual models for autonomous systems and applications using synthetic data are disclosed. Although the present disclosure may be described with respect to an example of an autonomous vehicle 600 (alternatively referred to herein as “vehicle 600” or “ego machine 600”), an example thereof is provided 6A-6D ) is described, this should not be understood as a restriction. For example, the systems and methods described herein may be used, without limitation, by non-autonomous vehicles, semi-autonomous vehicles (e.g. in one or more adaptive driver assistance systems (ADAS)), guided and unguided robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles equipped with a or multiple trailers, flying vessels, boats, shuttles, rescue vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones and/or other types of vehicles. Even if the present one As the disclosure relates to the training of perception models for use in autonomous or semi-autonomous systems and applications, this should not be construed as a limitation. The systems and methods described herein may be used in the areas of augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and/or any other technology area in which machine learning models can be used.

The systems and methods described herein may be used, without limitation, by non-autonomous vehicles, semi-autonomous vehicles (e.g. in one or more adaptive driver assistance systems (ADAS)), guided and unguided robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles equipped with a or coupled to multiple trailers, flying vessels, boats, shuttles, rescue vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones and/or other types of vehicles. Furthermore, the systems and methods described herein can be used for a variety of purposes, e.g. B. and without limitation for the control of machines, the locomotion of machines, the driving of machines, the generation of synthetic data, the training of models, perception, augmented reality, virtual reality, mixed reality, robotics, the Security and surveillance, simulation and digital twinning, autonomous or semi-autonomous applications of machines, deep learning, simulation of environments, simulation of objects or actors and/or digital twinning, data center processing, conversational computing, the Light traffic simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing and/or any other suitable application.

The disclosed embodiments may include a variety of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented using a robot, aviation systems , media systems, boat systems, systems for monitoring intelligent areas, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using an edge device, systems that are a or multiple virtual machines (VMs), systems for performing synthetic data generation operations, systems that are at least partially implemented in a data center, systems for performing conversational Kt operations, systems for performing light transport simulations, systems for Perform collaborative content creation for 3D assets, systems implemented at least in part using cloud computing resources, and/or other types of systems.

Although the outputs of the machine learning models described here are described primarily as bird's-eye segmentation maps, this should not be construed as a limitation. For example, the outputs of the machine learning model(s) may be from any perspective (e.g., bird's-eye view, perspective (e.g., sensor's perspective), projected, etc.), excluding the frame of the present revelation. In addition, the outputs of the machine learning model(s) may include another type of mask, classification, confidence, probability, binary output and/or another type of output, even if described primarily as a segmentation map for objects become.

With reference to 1 is 1 an example flowchart for a process 100 of training a machine learning model using synthetic data according to some embodiments of the present disclosure. It is to be understood that these and other arrangements described herein are provided as examples only. Other arrangements and elements (e.g., machines, interfaces, functions, commands, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Further, many of the elements described herein are functional units that may be implemented as discrete or distributed components or in conjunction with other components and in any suitable combination and position. Various functions described herein as being performed by devices may be performed by hardware, firmware, and/or software. For example, various functions can be performed by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein may use similar components, features, and/or functions to those of the example autonomous vehicle 600 out 6A-6D , the exemplary computing device 700 7 and/or the exemplary data center 800 8th be executed.

The process 100 includes an effective technique for sampling data, e.g. B. from a self-driving simulator, which is simulator-independent and easy to implement in practice. The process 100 may be built on the principle of reducing the distance between the edges of the labels, thereby reducing the domain gap and allowing adverse frameworks to learn meaningfully in a representation space (e.g., a latent space). The process 100 may include methods for minimizing discrepancies, e.g. B. Using Pearson x ² , generalize to dense prediction tasks. In some embodiments, the process 100 may include combining domain adversarial techniques with pseudo labels 116. Although embodiments are described in which labels are not available in the target domain (e.g., the real world), in some embodiments labels may be available in the target domain to further improve the performance of the process 100 when training one or more machine learning models 112 to increase.

bezeichnet werden. In einigen Ausführungsformen kann auch ein echter Datensatz (T) (z. B. Daten der realen Welt 110) mit unmarkierten Beispielen $\text{T}={\left\{\left(x_i^t\right)\right\}}_{i=1}^{n_t},$

die in der realen Welt aufgenommen wurden, verfügbar sein. Das Ziel des Prozesses 100 ist das Erlernen eines Modells (Hypothese) h für eine bestimmte Aufgabe (z. B. Objekterkennung, Objektverfolgung, Freiraumanalyse usw.) unter Verwendung von Daten aus dem beschrifteten Datensatz S (z. B. den synthetischen Daten 108), so dass h in der echten Welt gut durchgeführt wird. In Ausführungsformen kann der nicht beschriftete Datensatz T (z. B. die echten Daten 110) in den Lernprozess einbezogen werden, um die Verteilung in der echten Welt zu erfassen.The generalization capabilities of one or more machine learning models 112 under distribution changes may be learned by analyzing a corresponding binary classifier of the machine learning model(s) 112. As such, the process 100 may implement domain adaptation theory and assume that the domain of the output is y = {0, 1} and limit the mathematical analysis to the binary classification setting. A simulator 106 may automatically generate labels for a task, and these data samples obtained from the simulator 106 may be used as a synthetic (or simulated) data set (S) (e.g., synthetic data 108) with labeled data points $\text{S}={\left\{\left(x_i^s,y_i^s\right)\right\}}_{i=1}^{n_s}$

be referred to. In some embodiments, a real data set (T) (e.g., real world data 110) with unlabeled examples may also be used $\text{T}={\left\{\left(x_i^t\right)\right\}}_{i=1}^{n_t},$

recorded in the real world will be available. The goal of the process 100 is to learn a model (hypothesis) h for a specific task (e.g. object detection, object tracking, free space analysis, etc.) using data from the labeled data set S (e.g. the synthetic data 108) , so h is performed well in the real world. In embodiments, the unlabeled data set T (e.g., real data 110) may be included in the learning process to capture the real-world distribution.

The synthetic data 108 can be generated in any way - e.g. However, based on the performance results during the experiment, the synthetic data 108 (e.g., corresponding to the data set S) in the process 100 can be generated from a domain adaptation perspective, with the aim of addressing the lack of labeled Data in the target domain (e.g., in the real world 110, as represented by data set T) by transferring knowledge from a labeled source domain (e.g., a virtual or simulated world).

und die Zieleingaben $x_i^t$

unabhängig und identisch aus den Verteilungen P_s bzw. P_t, verteilt sein, beide über X. Die Domäne der Ausgabe kann Y = {0, 1} sein, und als Indikator für die Leistung kann das Risiko einer Hypothese h : X → Y bezogen auf die Beschriftungsfunktion f angenommen werden, unter Verwendung einer Verlustfunktion I : Y x Y →ℝ+, unter der Verteilung D oder formal $R_D^l\left(h,f\right):={\mathbb{E}}_D\left[l\left(h\left(x\right),f\left(x\right)\right)\right].$

Die Beschriftungsfunktion kann ein optimaler Bayes-Klassifikator f(x) = argmax_ŷ∈_yP(y = ŷ|x)sein, wobei P(y|x) die bedingte Klassenverteilung entweder für die Quellendomäne (z. B, virtuelle oder simulierte Welt) (P_s(y|x)) oder die Ziel-Domäne (z. B. die echte Welt) ((P_t(y|x). Der Einfachheit halber kann $R_S^l\left(h\right)$

als $R_S^l\left(h\right):=R_{P_s}^l\left(h,f_s\right)$

und $R_T^l\left(h\right)$

als $R_T^l\left(h\right):=R_{R_t}^l\left(h,f_t\right)$

definiert werden.In some embodiments, various assumptions may be made to perform process 100. For example, the source inputs $x_i^s$

and the target entries $x_i^t$

independent and identical from the distributions P _s and P _t , respectively, both over X. The domain of the output can be Y = {0, 1}, and as an indicator of performance the risk of a hypothesis h : based on the labeling function f, using a loss function I : Y x Y →ℝ+, under the distribution D or formally $R_D^l\left(H,f\right):={\mathbb{E}}_D\left[l\left(H\left(x\right),f\left(x\right)\right)\right].$

The label function can be an optimal Bayesian classifier f(x) = argmax _ŷ ∈ _y P(y = ŷ|x), where P(y|x) is the conditional class distribution for either the source domain (e.g., virtual or simulated world ) (P _s (y|x)) or the target domain (e.g. the real world) ((P _t (y|x). For simplicity can $R_S^l\left(H\right)$

as $R_S^l\left(H\right):=R_{P_s}^l\left(H,f_s\right)$

and $R_T^l\left(H\right)$

as $R_T^l\left(H\right):=R_{R_t}^l\left(H,f_t\right)$

To be defined.

The hypotheses h (e.g. the machine learning model(s) 112) can be interpreted as the composition of h = ĥ◦g with g : X→Z and ĥ:Z → Y, where Z is a representation space (e.g. a latent space). So the hypothesis class can be defined as H := {ĥ◦h:ĥ ∈ Ĥ, g ∈ G}, such as h ∈ H. A generalization limit can then be used to measure the generalization performance of a model h.

Als Ergebnis kann Theorem (1) wie folgt definiert werden: $$R_T^l\left(h\right)\le R_S^l\left(h\right)+D_{h,H}^{\varphi }\left(P_s|P_t\right)+\lambda *$$

wobei $D_{h,H}^{\varphi }\left(P_s|P_t\right):=su{p}_{h\text{'}\in H}|{\mathbb{E}}_{x\sim P_s}\left[l\left(h\left(x\right),h\text{'}\left(x\right)\right)\right]-{\mathbb{E}}_{x\sim P_t}[\varphi *\left(l\left(h\left(x\right),h\text{'}\left(x\right)\right)\right)|,$

und die Funktion ϕ: ℝ₊ → ℝ definiert eine bestimmte f-Divergenz, wobei ϕ ihre (Fenchel-)Konjugierte ist. Der erste Begriff im Theorem kann die Leistung des/der maschinellen Lernmodelle(s) 112 bei der Simulation erklären und zeigt an, dass das Modell 112 den synthetischen Datensatz S (z. B. die synthetischen Daten 108) gut durchführen muss. In some embodiments, the following theorem (1) may be used to determine how well one or more machine learning models 112 generalize from the virtual to the real world can be meant. Assume that I: YxY → [0,1] on and $\lambda *:=mi{n}_{H\in H}{R}_S^l\left(H\right)+R_T^l\left(H\right).$

As a result, Theorem (1) can be defined as follows: $$R_T^l\left(H\right)\le R_S^l\left(H\right)+D_{H,H}^{\varphi }\left(P_s|P_t\right)+\lambda *$$

where $D_{H,H}^{\varphi }\left(P_s|P_t\right):=su{p}_{H\text{'}\in H}|{\mathbb{E}}_{x\sim P_s}\left[l\left(H\left(x\right),H\text{'}\left(x\right)\right)\right]-{\mathbb{E}}_{x\sim P_t}[\varphi *\left(l\left(H\left(x\right),H\text{'}\left(x\right)\right)\right)|,$

and the function ϕ: ℝ ₊ → ℝ defines a certain f-divergence, where ϕ is its (fennel) conjugate. The first term in the theorem can explain the performance of the machine learning model(s) 112 in simulation and indicates that the model 112 must perform well on the synthetic data set S (e.g., the synthetic data 108).

The second term of the theorem corresponds to the discrepancy between the marginal distributions P _s (x) and Pt(x) - z. E.g. how dissimilar the virtual and real worlds are from the perspective of an observer. The third term measures the ideal common hypothesis (λ*), which includes the notion of adaptability and can be attributed to the dissimilarity between the label functions.

In traditional domain fitting, two approaches are most common, depending on what assumptions are made about λ*. The first approach assumes that the last term in Theorem (1) is negligible, and the learning is carried out adversarially by minimizing the risk in the domain of the source and the domain mismatch in Z. The second approach introduces reweighting schemes to account for the dissimilarity between the labeling marginals P _s (y) and Pt(y), which can be either implicit or explicit. However, in the process 100 there is control over the synthetic data 108 because the synthetic data 108 is retrieved from the simulator 106. The process 100 enables control, for example. B. the classes of objects, the frequency of occurrence of objects, the variations in the appearance or class of objects and / or the locations at which the objects are arranged. As a result, Theorem (1) indicates that the process of data generation should be carried out so that λ* is negligible. If this is the case, the focus is on learning invariant representations. To achieve this, the process 100 includes a data generation procedure using a prior generator 102 (e.g., to generate a spatial prior) and a training algorithm executed by a training machine 114. The training strategy was determined based on the importance of controlling the dissimilarity between the label edge values P _s (y) and Pt(y). This is illustrated by following a lower bound as described with reference to Theorem (2). For example, assuming that D _JS (P _s ( _y )||P _t ( _Y )) ≥ D _JS (P _s (z)||P _t (z)) and D _JS corresponds to the Jensen-Shannon divergence, then Theorem (2) can be defined as follows: $$R_T^l\left(H\right)+R_S^l\left(H\right)\ge \frac{1}{2}{\left(\sqrt{D_{JS}\left(P_s\left(y\right)\Vert P_t\left(y\right)\right)}-\sqrt{D_{JS}\left(P_s\left(\mathrm{e.g}\right)\Vert P_t\left(\mathrm{e.g}\right)\right)}\right)}^2$$

Theorem (2) is important for imposing a lower bound on the joint risk and is useful for algorithms that aim to learn domain-invariant representations. In the process 100, theorem (2) illustrates the importance of the method of generating data in the simulator 106. When the synthetic data 108 is intentionally taken from the simulator 106 and objects are positioned in such a way that there are deviations between the real and the virtual world simply minimizing the risk in the source's domain and the discrepancy between the source's domain and the target's domain in the representation space may not help. Thus, failure to align the marginalia may prevent the process 100 from using SoTA algorithms for adversarial learning.

To accommodate this, the data sampling techniques in the process 100 are effective to generate synthetic data 108 that reduces the likelihood of mismatch between the label marginals in the target and source domains. For example, in some embodiments, spatial priorities may be generated using the priority generator 102 targeted to a system in which there is no real labeled data. As another example, in embodiments in which a few data points in the target domain (e.g., in the real world) are labeled, these labels may be used to estimate a prior for the positions of the objects in the scene.

For example, in previous approaches, the positions of objects were determined based on the structure or layout of a road or path and/or based on the drivable area of a Map determined. As in 2 As illustrated, visualizations 202 and 204 illustrate the induced margin P _s (y) when the sampling strategy follows standard approaches for the structure of the street and/or the area of the map. In contrast, the visualization 206 illustrates the induced edge value when the locations of the objects are located using a spatial prior of the prior generator 102. Similarly, the visualization 208 illustrates the induced marginality when the locations of the objects are sampled using a target priority of the prior generator 102, where the target priority is generated based on one or more training data sets in the target domain (e.g., in the real world). . The visualization 210 corresponds to a validation set, e.g. B. for the segmentation of vehicles from a bird's eye view (BEV). As illustrated, sampling in visualization 206 using a spatial prior in a simulator 106 is most similar to sampling from the real training data set of visualization 208. By using the spatial prior of the present disclosure, the synthetic data 108 can be better matched to the distribution of objects in the real world, such that one or more machine learning models 112 trained on synthetic data 108 can be better aligned in the real world can be carried out well. Compared to the previous approaches, the Jensen-Shannon divergence (JSD) between the induced marginal value using the spatial prior minimizes the divergence between the marginal value of the task label (e.g. 5.7e ^-4 for the spatial prior approach of the present Revelation as opposed to 1.40e ^-3 for the previous road or map based approaches) and results in a more diverse synthetic data set. In particular, the JSD for the real training data set in the visualization 208 is of the same order of magnitude as the spatial prior of the visualization 206 (e.g. 5.7e ^-4 for the spatial prior and 5.4e ^-4 for the real labeled training data set), and the spatial prior does not require access to labeled data. In the experiments of the present disclosure, the nuScenes training data set was used as the training data set.

If no labels are available in the target domain, the divergence between the label boundary values P(y) between the generated and the real data cannot be measured. However, if e.g. For example, if a bird's-eye view segmentation map is available, an empirically selected spatial prior can be generated that represents positions with a high probability of finding an object. For example, as in the visualization 206 in 2 shown, a spatial prior can be generated that localizes positions of objects proportional to a longitudinal (e.g. from left to right) distance of the object(s) to the ego machine. For samples with the spatial prior, the probability of an object being at a particular location decreases as the lateral distance from the ego machine increases. For example, for the position (x1,x2)∈ℝ ² in meters relative to the ego machine, the spatial prior, P _spatial , can be represented according to the following equation (3): $$P_{r\ddot{a}umlicH}\left(x_1,x_2\right)\propto \{\begin{array}{cc}-\frac{1}{125}\left|x_2\right|+0.6& \left|x_2\right|\le 12.5\\ -\frac{1}{75}\left|x_2\right|-50& \left|x_2\right|>12.5\end{array}$$

These values of equation (3) can be a linear interpolation between a probability proportional to 0.0 for |x ₂ | = 50 meters, 0.5 for |x ₂ | = 12.5 meters and 0.6 for |x ₂ | = correspond to 0 meters. Thus, the spatial prior P _spatially models the observation that other objects are generally located along the long axis (e.g., front to back) of the ego machine. The density is independent of x ₁ , which _spatially models the P on a straight road. Therefore, for each x, it is equally likely that an object x̂ ₁ meter is in front of or behind the ego machine x̂ ₁ ∈ ℝ.

For example illustrated 3A the difference between previous sampling methods that rely on the structure of streets or the drivable area of a map and the spatial approach of the present disclosure. For example, visualization 302 illustrates the potential locations of objects that can be located using previous approaches. In contrast, visualization 304 illustrates the potential locations of objects located using a spatial prior. As you can see, the diversity of locations for objects is increased when a spatial prior is used. This allows for a more diverse synthetic data set that is more similar to the diversity of a real data set (e.g. as described above in relation to the JSD). In the visualization 304, this diversity in the synthetic data 108 can force the machine learning models 112 to focus on the task at hand (e.g., object detection, object segmentation, etc.) rather than focusing too much on the conventional arrangement of objects on one path, even if the objects are not on the path of the ego machine or in a non-traditional way (e.g. a vehicle's position may be perpendicular to the path or in the opposite direction to the direction of travel, etc.). For example, the synthetic data 108 generated in this manner forces the machine learning model(s) 112 to learn essential features appropriate to the task at hand (e.g., learning essential features of an object of interest for object recognition ). As a result, the divergence between the boundary conditions of the task labeling is also reduced, at least in relation to here 2 described.

In some embodiments, in addition or alternatively to using spatial priority, a target priority may be generated based on a data set in a target domain that includes labels. For example, in practical applications there is often only a small proportion of data available in the target domain. In such scenarios, a natural sampling prior would be to calculate P _t (y) and generate the synthetic data based on that. Depending on how large the amount of data is in the target domain, the prior can be a mixture of the spatial prior and the target prior (e.g. αPspatial + (1 - α)Pt). include. In such an example, the locations of the objects can be chosen from the target prior, the spatial prior, or a combination thereof for any α ∈ [0,1].

In addition to selecting the locations of objects from the spatial and/or target priorities, other scene information 104 may also be used to create a synthetic data set 108 that is diverse and minimizes the divergence between the target and source domains. For example, the scene information may include the number of objects, the position of the objects, the color of the objects, the type of objects, the intrinsic sensor parameters, the extrinsic camera parameters, the weather, and/or other scene information 104. In some embodiments, the scene information 104 may be sampled randomly based on an existing data set and/or using another technique. For example illustrated 3B Scene information from a sample data set in the target domain. Diagram 310 illustrates the distribution of the number of vehicles per scene, Diagram 312 illustrates the distribution of the field of view of the cameras of an ego machine, Diagram 314 illustrates the greed relative to the ego machine coordinate system of the cameras (e.g. 6 peaks for 6 different camera directions), and diagram 316 illustrates the height of a LiDAR sensor of the ego machine.

In some embodiments, these statistics may be reconciled when data (e.g., scene information 104) is retrieved from the simulator 106.

Regarding the weather, there can be 106 different weather settings depending on the simulator. For example, there may be controls for cloud cover, precipitation, precipitation deposits (e.g. puddles), wind strength, wetness, fog density, sun angle, etc. In embodiments, the weather can be randomly selected from x ~ Unif(0,v) for v ∈ {0,2,0,4,0,6,0,8,1,0} and/or categorically from the simulator's respective weather settings 106 can be removed. Based on experiments, it was determined that using a “sunny” weather setting for all data and sampling the object positions using a spatial prior can compensate for much of the loss in weather diversity, especially when compared to sampling only from the “sunny” weather setting and sampling object positions based on previous approaches.

In some embodiments, the generated synthetic data 108 may be post-processed before being used to train the machine learning model(s) 112. For example, the synthetic data 108 may be edited or enhanced to provide a more realistic impression, such as by applying vignette, grain flicker, brightening, autoexposure, lens flare, and/or depth of field effects in post-processing.

Using the synthetic data 108 generated by the simulator 106 in combination with the real data 110, in embodiments, the machine learning model(s) 112 may be trained (e.g., one or more parameters - such as weights and biases - may be of the machine learning model(s) 112 are updated or adjusted using the data 108 and 110 as input), using a training engine 114. For example, the parameters of the machine learning model(s) 112 may be updated until an acceptable level of accuracy or precision is reached (e.g., until convergence is achieved). Because the synthetic data 108 is generated using a simulator-independent sampling strategy, any simulator 106 can be used to generate the synthetic data 108 with sufficient diversity to minimize the distance between label edges. As determined based on theorem (1) above, the distance between the distributions of the inputs P _s (x) and Pt(x) should be minimized to improve real-world performance to improve. The training algorithm of the present disclosure achieves this by learning domain-invariant representations and minimizing the divergence between the virtual and real worlds in a latent space Z. Minimizing the divergence in a latent space enables the domain adaptation algorithms to be sensor and to be architecture independent. Without the sampling strategy of the present disclosure, effectively learning invariant representations through adversarial learning would be less effective. Therefore, the combination of the adverse training algorithm and the sampling strategy for generating the synthetic data 108 enables learning the invariant representations of the domain. Additionally, in embodiments, pseudo-labels 116 may be used to further improve the training algorithms and increase the accuracy and precision of the machine learning model(s) 112.

wobei I: ℝ^h·wxℝ^h·w → R₊ definiert ist als $l\left(p,q\right):=\frac{1}{h\cdot w}{\displaystyle {\sum }_{i=0}^{h\cdot w}\beta p_i}log{q}_i+\left(1-p_i\right)\text{log}\left(1-q_i\right)$

$$

und pi und qi ∈ [0,1] (der gemittelte Binärkreuz-Entropieverlust). B ist ein Hyperparameter, der die Bedeutung positiver Pixel in der pixelweisen binären Sementierunskarte ewichtet. d_st kann emäß Gleichuna (5) unten definiert werden: $$d_{st}:={\mathbb{E}}_{x\sim p_s}\left[{\mathbb{E}}_{h\cdot w}\left[\left(\widehat{h\text{'}}\cdot g\right)\right]\right]-{\mathbb{E}}_{x\sim p_t}\left[{\mathbb{E}}_{h\cdot w}\left[\frac{1}{4}{\left(\widehat{h\text{'}}\cdot g\right)}_i^2+\widehat{h\text{'}}\cdot g{\left(x\right)}_i\right]\right]$$

was der Pearson x²-Divergenz und ${\mathbb{E}}_{h\cdot w}\left[x\right]:=\frac{1}{h\cdot w}{\displaystyle {\sum }_{i=0}^{h\cdot w}{x}_i}$

entspricht. Anders als bei früheren Implementierungen von f-DAL Pearson kann d_st als Klassifikator für eine Domäne pro Standort 410 interpretiert werden, wie in 4 veranschaulicht. Zum Beispiel kann der Klassifikator 410 als Diskriminator fungieren, der identifiziert, ob die Daten in einem latenten Raum einer echten Domäne oder einer simulierten oder virtuellen Domäne entsprechen. Wenn die Werte im latenten Raum als zur Domäne der Quelle (und nicht zur Ziel-Domäne) gehörend klassifiziert werden, kann dies im d_st-Wert erfasst und von der Trainingsmaschine 114 verwendet werden, um die Parameter des/der maschinellen Lernmodells/Maschinenmodelle 112 zu aktualisieren. In Ausführungsformen kann eine Gradientenumkehrschicht verwendet werden, um das Min-Max-Ziel in einem einzigen Vorwärts-Rückwärts-Durchgang zu behandeln.The training algorithm can simultaneously minimize the source domain error and align the two distributions (e.g. virtual and real) in a representation space Z. One goal of the training is to find a hypothesis h that jointly minimizes the first two terms of Theorem (1). Suppose h:X→Y with Y ⊆ ℝ ^h·v , which corresponds to ^a pixel-wise binary segmentation map of dimensions h and w, respectively, where X is an unordered set of multiview images for camera bird's eye view (BEV) segmentation ^·3·H·W and a point cloud for LiDAR-based BEV segmentation X ⊆ ℝ ^3·N . The objective function can then be formulated in an adversarial way, for example by minimizing the objective of equation (4): $$\widehat{H}\in \stackrel{min}{\widehat{H},G}\in G\stackrel{max}{\widehat{H\text{'}}\in \widehat{H}}{\mathbb{E}}_{x,y\sim p_s}\left[l\left(\widehat{H}\cdot G,y\right)\right]+d_{st}$$

where I: ℝ ^{h w} xℝ ^{h w} → R ₊ is defined as $l\left(p,q\right):=\frac{1}{H\cdot w}{\displaystyle {\sum }_{i=0}^{H\cdot w}\beta p_i}lOG{q}_i+\left(1-p_i\right)\text{log}\left(1-q_i\right)$

$$

and pi and qi ∈ [0,1] (the averaged binary cross entropy loss). B is a hyperparameter that weights the importance of positive pixels in the pixel-wise binary segmentation map. d _st can be defined according to equation (5) below: $$d_{st}:={\mathbb{E}}_{x\sim p_s}\left[{\mathbb{E}}_{H\cdot w}\left[\left(\widehat{H\text{'}}\cdot G\right)\right]\right]-{\mathbb{E}}_{x\sim p_t}\left[{\mathbb{E}}_{H\cdot w}\left[\frac{1}{4}{\left(\widehat{H\text{'}}\cdot G\right)}_{\mathrm{<figure-callout\; id="2"\; label="i"\; filenames="DE112022000498T5\_0031.png"\; state="\{\{state\}\}">i</figure-callout>}}^2+\widehat{H\text{'}}\cdot G{\left(x\right)}_i\right]\right]$$

what the Pearson x ² divergence and ${\mathbb{E}}_{H\cdot w}\left[x\right]:=\frac{1}{H\cdot w}{\displaystyle {\sum }_{i=0}^{H\cdot w}{x}_i}$

corresponds. Unlike previous implementations of f-DAL Pearson, d _st can be interpreted as a one domain per site classifier 410 as in 4 illustrated. For example, the classifier 410 may function as a discriminator that identifies whether the data in a latent space corresponds to a real domain or a simulated or virtual domain. If the values in the latent space are classified as belonging to the source domain (and not the target domain), this can be captured in the d _st value and used by the training engine 114 to adjust the parameters of the machine learning model(s) 112 to update. In embodiments, a gradient inversion layer may be used to handle the min-max target in a single forward-backward pass.

As in 4 illustrated, process 400 can be used to train machine learning model(s) 112. For example, to calculate the notion of discrepancy, d _st , a domain classifier ĥ' is introduced per site, in embodiments prior to the final up-sampling module of the bird's eye view model encoder. The domain classifier can include two convolutional layers with LeakyRelu nonlinearity that predict whether a pixel in the H x W semantic map belongs to either a source domain (e.g. the virtual or simulated world) or a target domain (e.g. the real one world). The other output, ĥ, predicts the output of the machine learning model(s) 112 - in this example, a BEV segmentation map. g forms the backbone of the machine learning model(s) 112, which may include a CamEncoder and a BevEncoder (up to the last up-sampling module).

wobei τ ∈ [0,1], 1[x] der Anzeigefunktion p: = h◦g(x^t) entspricht und p_aug := h◦ g◦ aug(x^t) mit aug : X→Xeiner Funktion, die eine starke Augmentierung des Eingabedatenpunkts erzeugt. Bei Kamerasensoren kann aug eine Version von RandAugment sein, die zusätzlich Kameraaussetzer einbezieht. Bei LiDAR können die Bildvergrößerungen durch zufälliges Rauschen über die Punktpositionen sowie durch Punktausfälle ersetzt werden. T kann ein jeglicher Wert sein, wie beispielsweise 0,8, 0,9, 0,95 usw. In nicht einschränkenden Ausführungsformen wurde T auf 0,9 festgelegt. Das Ziel von Gleichung (5) ist es, Pseudo-Kennzeichnungen 116 unter Verwendung der durch T bestimmten Konfidenzpunkte in den Vorhersagen des Modells für die Ziel-Domäne auf einer nicht stark erweiterten Version desselben Datenpunktes zu erzeugen. Die Verwendung von Pseudo-Kennzeichnungen 116 und adversariales Training minimieren gemeinsam die Gleichungen (4) und (6).In some embodiments, in addition to the objective of equation (4), a pseudo loss may be used, such as in equation (6) below: $$l_{pseudO}\left(p.p_{auG}\right):={\sum }_{i=0}^{H\cdot w}〛\left[p_i\ge \tau \right]\left[\beta lOG{p}_{au{G}_i}\right]+〛\left[1-p_i\le 1-\tau \right]\text{log}\left(1-p_{au{G}_i}\right)$$

where τ ∈ [0,1], 1[x] corresponds to the display function p: = h◦g(x ^t ) and p _aug := h◦ g◦ aug(x ^t ) with aug : X→Xa function that is a strong augmentation of the input data point generated. For camera sensors, aug can be a version of RandAugment that also includes camera dropouts. With LiDAR, the image magnifications can be replaced by random noise about the spot positions as well as spot dropouts. T can be any value such as 0.8, 0.9, 0.95, etc. In non-limiting embodiments, T has been set to 0.9. The goal of equation (5) is to generate pseudo-labels 116 using the confidence points determined by T in the model's predictions for the target domain on a not greatly expanded version of the same data point. The use of pseudo-labels 116 and adversarial training jointly minimize equations (4) and (6).

The machine learning model(s) 112 may include any type of machine learning model or algorithm, such as a machine learning model(s) that supports linear regression, logistic regression, decision trees, etc Vector Machines (SVM), Naive Bayes, k-nearest neighbor (Knn), K means clustering, random forest, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., auto-encoders, convolution algorithms, recurrent algorithms , Perceptrons, Long/Short Term Memory/LSTM, Hopfield, Boltzmann, Deep Belief, Deconvolution, Generative Adversary Algorithms, Liquid State Machine, etc.), interest detection algorithms, computer vision algorithms and/or other types of algorithms or machine Learning models.

As in 5 As shown, each block of the method 500 described herein includes a process of computing that can be performed with any combination of hardware, firmware and/or software. For example, various functions can be performed by a processor executing instructions stored in memory. The method 500 may also be embodied in the form of computer-usable instructions stored on computer storage media. Method 500 may be provided through a standalone application, service, or hosted service (standalone or in combination with another hosted service), or a plug-in for another product, to name a few. In addition, the method 500 is based on the process 100 by way of example 1 described. However, this method 500 may additionally or alternatively be performed by any system or combination of systems, including but not limited to the systems described herein.

5 is a flowchart depicting a method 500 for training a machine learning model using synthetic data in accordance with some embodiments of the present disclosure. The method 500 includes, at block B502, sampling one or more locations of objects from a prior. For example, one or more locations of objects may be sampled from a spatial prior (if labeled target data is not available) and/or from a target prior (if labeled data can be used to determine a distribution). The prior can be generated using the priority generator 102.

NetworkThe method 500 includes, in block B504, generating synthetic data based at least in part on the locations of one or more objects and including one or more objects at the locations of one or more objects. For example, using the sampled locations of the objects and/or other scene information 104, a scene may be created in a simulator 106 and the synthetic data 108 may be generated from the scene.

The method 500 includes, at block B506, updating one or more parameters of a machine learning model using a discriminator based at least in part on the synthetic data and the real world data. For example, using a discriminator h', one or more machine learning models 112 can be trained for real-world use using the synthetic data 108 and the real data 110. In embodiments, pseudolabels may be generated for data points where the confidence of the output of the machine learning model(s) 112 is above a confidence threshold (e.g., 0.85, 0.9, etc.). In a further training epoch, the pseudo labels can then be used as a basis for further training of the machine learning model(s) 112.

EXEMPLARY AUTONOMOUS VEHICLE

6A is an illustration of an example autonomous vehicle 600 according to some embodiments of the present disclosure. The autonomous vehicle 600 (alternatively referred to herein as “vehicle 600”) may include, without limitation, a personal vehicle such as a car, a truck, a bus, an emergency vehicle, a shuttle, an electric or motorized bicycle, a motorcycle, a fire engine, a police vehicle, an ambulance, a boat, a construction vehicle, an underwater vehicle, a drone, a trailer-coupled vehicle, and/or any other type of vehicle (e.g., which is unmanned and/or accommodates one or more passengers). Autonomous vehicles are generally described in terms of automation levels as defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (e.g., Standard No. J3016-201806, published June 15, 2018, Standard No. J3016-201609, published September 30, 2016, and previous and future versions of this standard). . The vehicle 600 may be operable according to one or more of levels 3-5 of the autonomous driving levels. The vehicle 600 may be operable according to one or more of levels 1-5 of the autonomous driving levels. For example, depending on the embodiment, the vehicle 600 can have driver assistance (level 1), partial automation (level 2), conditional automation (level 3), high automation (level 4) and/or full automation (level 5). be able to. The term “autonomous,” as used herein, may include any and/or all types of autonomy for the vehicle 600 or other machine, such as being fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, assistive Providing autonomy, being semi-autonomous, being primarily autonomous, or some other name.

The vehicle 600 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. The vehicle 600 may include a propulsion system 650, such as. B. an internal combustion engine, a hybrid electric prime mover, a fully electric motor and / or another type of drive. The propulsion system 650 may be connected to a powertrain of the vehicle 600, which may include a transmission to enable propulsion of the vehicle 600. The propulsion system 650 may be controlled in response to receiving signals from the accelerator pedal/throttle lever 652.

A steering system 654, which may include a steering wheel, may be used to steer the vehicle 600 (e.g., along a desired path or route) when the propulsion system 650 is in operation (e.g., when the vehicle is in motion is). The steering system 654 may receive signals from a steering actuator 656. For full automation functionality (Level 5), the steering wheel can be optional.

The brake sensor system 646 may be used to actuate the vehicle brakes when signals are received from the brake actuators 648 and/or brake sensors.

Controller 636, which has one or more system-on-chips (SoCs) 604 ( 6C ) and/or GPUs may send signals (e.g., representative of commands) to one or more components and/or systems of the vehicle 600. For example, the controller may send signals to operate the vehicle brake via one or more brake actuators 648, to operate the steering system 654 via one or more steering actuators 656, to operate the drive system 650 via one or more accelerator pedals/levers 652. The controller 636 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and issue operational commands (e.g., signals representing commands) to enable autonomous driving and/or to support a human driver in driving the vehicle 600. The controller 636 may include a first controller 636 for autonomous driving functions, a second controller 636 for functional safety functions, a third controller 636 for artificial intelligence functionality (e.g. computer vision ), a fourth controller 636 for infotainment functionality, a fifth controller 636 for emergency redundancy, and/or other controllers. In some examples, a single controller 636 may handle two or more of the above functionalities, two or more controllers 636 may handle a single functionality, and/or any combination thereof.

The controller(s) 636 may provide the signals to control one or more components and/or systems of the vehicle 600 in response to sensor data received from one or more sensors (e.g., sensor inputs). The sensor data can, for example, and without limitation of global navigation satellite system ("GNSS") sensors 658 (e.g. Global Positioning System sensor(s)), RADAR sensor(s) 660, ultrasonic sensor(s) 662, LIDAR sensor(s) 664, inertial measurement units -Sensor(s) (IMU) 666 (e.g. accelerometer(s), gyroscope(s), magnetic compass(s), magnetometer(s), etc.), microphone(s) 696, stereo cameras 668, wide-angle cameras 670 (e.g. fisheye cameras), Infrared cameras 672, ambient cameras 674 (e.g. 360 degree cameras), long-range cameras and/or medium-range cameras 698, speed sensor(s) 644 (e.g. for measuring vehicle speed 600), vibration sensor(s) 642, steering sensor(s). ) 640, brake sensor(s) (e.g. as part of the brake sensor system 646) and/or other sensor types are received.

One or more of the controllers 636 may receive inputs (e.g., represented by input data) from an instrument cluster 632 of the vehicle 600 and outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (human-machine interface). machine interface (HMI) display 634, an acoustic signal generator, a loudspeaker and / or received via other components of the vehicle 600. The outputs may include information such as vehicle speed, RPM, time, imaging data (e.g., the high resolution (“HD”) map 622 6C ), location data (e.g., the location of the vehicle 600, such as on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and status of objects as indicated by the Controller 636 are perceived, etc. include. For example, the HMI display 634 may provide information about the presence of one or more objects (e.g., a street sign, a warning sign, a changing traffic light, etc.) and/or information about driving maneuvers that the vehicle has made, is making, or will make (e.g. that it is now changing lanes, taking exit 34B in two miles, etc.).

The vehicle 600 further includes a network interface 624 that may use one or more wireless antennas 626 and/or modems to communicate over one or more networks. For example, the network network interface 624 may be capable of communicating via LTE, WCDMA, UMTS, GSM, CDMA2000, etc. The wireless antenna(s) 626 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area network(s), such as Bluetooth LE, Z-Wave, ZigBee, etc., and/or low power wide-area networks (“LPWANs”), such as LoRaWAN, SigFox, etc.

6B is an example of camera positions and fields of view for the example autonomous vehicle 600 6A according to some embodiments of the present disclosure. The cameras and the corresponding fields of view represent an exemplary embodiment and should not be construed as limiting. For example, additional and/or alternative cameras may be included and/or the cameras may be located at different locations within the vehicle 600.

Camera types for the cameras may include, but are not limited to, digital cameras that may be adapted for use with the components and/or systems of the vehicle 600. The camera(s) may operate at automotive integrity safety level (ASIL) B and/or another ASIL. The camera types may be capable of any image capture rate depending on the embodiment, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The cameras may be capable of rolling shutters, global Shutters, another type of shutter, or a combination thereof. In some examples, the color filter assembly may be a red clear-clear (RCCC) color filter assembly, a red clear blue (RCCB) color filter assembly, a red-blue-green-clear ( red blue green clear (RBGC) color filter array, a Foveon-X3 color filter array, a Bayer Sensor (RGGB) color filter array, a monochrome sensor color filter array, and/or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras with an RCCC, an RCCB, and/or an RBGC color filter array, may be used in an effort to increase light sensitivity.

In some examples, one or more of the cameras may be used to perform functions of the advanced driver assistance systems (ADAS) (e.g., as part of a redundant or fail-safe design). For example, a multi-function mono camera can be installed, providing functions such as lane departure warning, traffic sign assistant and intelligent headlight control. One or more of the camera(s) (e.g. all cameras) can simultaneously record and provide image data (e.g. a video).

One or more of the cameras may be mounted in a mounting assembly, such as a custom-designed (3D printed) assembly, to capture stray light and reflections from inside the car (e.g., reflections from the dashboard reflected in the windshield mirrors). to exclude those that could affect the image data capture capabilities of the camera. Referring to side mirror mounting assemblies, the side mirror assemblies can be custom 3D printed so that the camera mounting plate matches the shape of the side mirror. In some examples, the camera(s) may be integrated into the side mirror. With side view cameras, the camera(s) can also be integrated into the four pillars at each corner of the driver's cab.

Cameras with a field of view that includes portions of the environment in front of the vehicle 600 (e.g., forward-facing cameras) may be used for surround vision to help identify and assist in forward-facing paths and obstacles , using one or more controllers 636 and/or control SoCs, to provide important information for generating an occupancy grid and/or determining preferred vehicle paths. Forward-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection and collision avoidance. Forward-facing cameras can also be used for ADAS functions and systems, including lane departure warnings (LDW), autonomous cruise control (ACC), and/or other functions such as traffic sign recognition.

A variety of cameras may be used in a forward-facing configuration, including, for example, a monocular camera platform that includes a CMOS (complementary metal oxide semiconductor) color imager. Another example is wide angle camera(s) 670, which may be used to perceive objects coming into view from the periphery (e.g., pedestrians, intersection traffic, or bicycles). Although in 6B Only one wide-angle camera is illustrated, there may be any number of wide-angle cameras 670 on the vehicle 600. Additionally, long-range camera(s) 698 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. The 698 long range camera(s) can also be used for object detection and classification as well as simple object tracking.

One or more stereo cameras 668 may also be included in a front-facing configuration. The one or more of the stereo camera(s) 668 include an integrated control unit that includes a scalable processing unit that can provide programmable logic (FPGA) and a multi-core microprocessor with an integrated CAN or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle's surroundings, which includes a distance estimate for all points in the image. (An) alternative stereo camera(s) 668 may include a compact stereo vision sensor(s), which may include two camera lenses (one each on the left and one on the right) and an image processing chip that may include the Measure the distance from the vehicle to the target object and use the generated information (e.g. metadata) to activate the autonomous emergency braking and lane departure warning functions. Other stereo camera types 668 may be used in addition to or as an alternative to those described here.

Cameras with a field of view that includes portions of the environment on the side of the vehicle 600 (e.g., side view cameras) may be used for surround vision and provide information to generate and update the occupancy grid and generate side collision warnings. For example, environmental camera(s) 674 (e.g., four environmental cameras 674, as in 6B shown) can be positioned on the vehicle 600. The ambient camera(s) 674 may include wide-angle cameras 670, fisheye cameras, 360-degree cameras, and/or similar cameras. For example, four fisheye cameras can be positioned at the front, rear and sides of the vehicle. In an alternative arrangement, the vehicle may use three surround cameras 674 (e.g., left, right, and rear) and use one or more other cameras (e.g., a front-facing camera) as a fourth surround-view camera.

Cameras with a field of view that includes portions of the environment at the rear of the vehicle 600 (e.g., rearview cameras) may be used for parking assistance, surround visibility, rear collision warning, and occupancy grid creation and updating. A variety of cameras may be used, including, but not limited to, cameras that are also suitable as front-facing cameras (e.g., long-range and/or medium-range cameras 698, stereo cameras 668, infrared cameras 672, etc.), as described herein.

6C is a block diagram of an example system architecture for the example autonomous vehicle 600 6A , according to some embodiments of the present disclosure. It is to be understood that these and other arrangements described herein are provided as examples only. Other arrangements and elements (e.g., machines, interfaces, functions, commands, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Further, many of the elements described herein are functional units that may be implemented as discrete or distributed components or in conjunction with other components and in any suitable combination and position. Various functions described herein as being performed by devices may be performed by hardware, firmware, and/or software. For example, various functions can be performed by a processor executing instructions stored in memory.

All components, features and systems of the vehicle 600 in 6C are shown as connected via bus 602. The bus 602 may include a Controller Area Network (CAN) data interface (alternatively referred to herein as a “CAN bus”). A CAN may be a network within the vehicle 600 that is used to assist in controlling various features and functionalities of the vehicle 600, such as brake application, acceleration, braking, steering, windshield wipers, etc. A CAN bus may be configured to: that it has dozens or even hundreds of nodes, each with its own unique identifier (e.g. a CAN ID). The CAN bus can be read to determine steering wheel angle, ground speed, engine revolutions per minute (RPMs), button positions and/or other vehicle status indicators. The CAN bus can be ASIL B compliant.

Although bus 602 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and/or Ethernet can also be used in addition to or as an alternative to the CAN bus. Again, although a single line is used to represent bus 602, this is not intended to be limiting. For example, there may be any number of buses 602, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and/or one or more other types of buses with a different protocol. In some examples, two or more buses 602 may be used to perform various functions and/or to provide redundancy. For example, a first bus 602 can be used to avoid collisions and a second bus 602 can be used for control. In each example, each bus 602 may communicate with any component of the vehicle 600, and two or more buses 602 may communicate with the same components. In some examples, each SoC 604, each controller 636, and/or each computer in the vehicle may access the same input data (e.g., inputs from sensors of the vehicle 600) and be connected to a common bus, such as the CAN bus.

The vehicle 600 may include one or more controllers 636, as described herein 6A is described. The 636 controller can be used for a variety of functions. The controller 636 may be coupled to any of the various other components and systems of the vehicle 600 and may be used to control the vehicle 600, the artificial intelligence of the vehicle 600, the infotainment for the vehicle 600, and/or the like.

The vehicle 600 may include a system on a chip (SoC) 604. The SoC 604 may include CPUs 606, GPUs 608, processor(s) 610, caches 612, accelerators 614, data storage 616, and/or other components and functions not shown. The SoC 604 can be used to control the vehicle 600 in a variety of platforms and systems. For example, the SoC 604 may be combined in a system (e.g., the system of the vehicle 600) with an HD card 622 that receives map updates and/or upgrades via a network interface 624 from one or more servers (e.g., servers ( n) 678 out 6D ).

The CPUs 606 may include a CPU cluster or a CPU complex (alternatively referred to herein as a “CCPLEX”). The CPUs 606 may include multiple cores and/or L2 caches. For example, in some embodiments, CPUs 606 may include eight cores in a coherent multi-processor configuration. In some embodiments, the CPUs 606 may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). The CPUs 606 (e.g., CCPLEX) may be configured to support concurrent cluster operation so that any combination of the CPUs 606 cluster(s) may be active at any given time.

The CPUs 606 may implement power management functions that include one or more of the following features: individual hardware blocks may be automatically clocked when idle to conserve dynamic power; each core clock can be controlled when the core is not executing active instructions due to the execution of WFI/WFE instructions; each core can be independently power controlled; each core cluster can be clocked independently if all cores are clocked or power driven; and/or each core cluster can be independently power controlled when all cores are power controlled. The CPUs 606 may implement an advanced power state management algorithm in which allowed power states and expected wake times are specified and the hardware/microcode determines the best power state for the core, cluster, and CCPLEX. The processing cores can support simplified performance status entry sequences in software, offloading the work to microcode.

The GPUs 608 may contain an integrated GPU (alternatively referred to here as “iGPU”). The GPUs 608 may be programmable and efficient for parallel workloads. The GPUs 608 may use an extended Tensor instruction set in some examples. The GPUs 608 may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96 KB of storage capacity) and two or more of the streaming microprocessors share one Share L2 cache (e.g. an L2 cache with a storage capacity of 512 KB). In some embodiments, GPUs 608 may include at least eight streaming microprocessors. The GPUs 608 may use computer application programming interfaces (API). Additionally, the GPUs 608 may use one or more parallel computing platforms and/or programming models (e.g., NVIDIA CUDA).

The GPUs 608 may be power optimized for the best performance in automotive and embedded use cases. For example, the GPUs 608 may be fabricated on a Fin field-effect transistor (FinFET). However, this is not intended to be limiting and the GPUs 608 may be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor may include a number of mixed-precision processing cores partitioned into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores could be partitioned into four processing blocks. In such an example, each processing block could have 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a Distribution unit and/or a 64 KB register file must be assigned. Additionally, streaming microprocessors can include independent parallel integer and floating point data paths to enable efficient execution of workloads with a mix of computation and addressing computations. The streaming microprocessors can include an independent thread scheduling function to enable finer-grained synchronization and cooperation between parallel threads. The streaming microprocessors can include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

The GPUs 608 may include high bandwidth memory (HBM) and/or a 16 GB HBM2 memory subsystem to provide peak memory bandwidth of approximately 900 GB/s in some examples. In some examples, in addition to or as an alternative to the HBM memory, a synchronous graphics random-access memory (SGRAM) may be used, such as a graphics double data rate type five synchronous random access memory (graphics double data rate type five - GDDR5).

The GPUs 608 may include unified memory technology including access processors to enable more accurate migration of memory pages to the processor that accesses them most frequently, thereby improving efficiency for memory areas shared between processors . In some examples, Address Translation Services (ATS) support may be used to allow the GPU(s) 608 to directly access the CPUs 606 page tables. In such examples, an address translation request may be sent to the CPUs 606 if the memory management unit (MMU) of the GPUs 608 has an error. In response, the CPUs 606 may search their page tables for the virtual-to-physical mapping for the address and transmit the translation back to the GPUs 608. Therefore, unified memory technology may provide a single unified virtual address space for the Enable memory of the CPUs 606 and the GPUs 608, which simplifies GPUs 608 programming and porting applications to the GPUs 608.

In addition, the GPUs 608 may include an access counter that can track the frequency of access of the GPUs 608 to the memory of other processors. The access counter can help move memory pages to the physical memory of the processor that accesses the pages most frequently.

The SoC 604 may include any number of caches 612, including those described herein. For example, caches 612 may include an L3 cache that is available to both CPU(s) 606 and GPU(s) 608 (e.g., attached to both CPUs 606 and GPUs 608). The cache(s) 612 may include a write-back cache that can track the status of lines, e.g. B. by using a cache coherency protocol (e.g. MEI, MESI, MSI, etc.). The L3 cache may include 4 MB or more depending on the embodiment, although smaller cache sizes may also be used.

The SoCs 604 may include arithmetic logic units (ALUs) that may be used in processing related to one of the various tasks or operations of the vehicle 600 - e.g. B. when processing DNNs. In addition, the SoCs 604 may include floating point units (FPUs) - or other types of mathematical coprocessors or numerical coprocessors - for performing mathematical operations in the system. For example, the SoCs 104 may include one or more FPUs integrated as execution units within a CPU 606 and/or GPU 608.

The SoC 604 may include one or more accelerators 614 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC 604 may include a hardware acceleration cluster, which may include optimized hardware accelerators and/or large on-chip memory. The large on-chip memory (e.g. 4 MB SRAM) can enable a hardware acceleration cluster to accelerate neural networks and other calculations. The hardware acceleration cluster may be used to supplement the GPUs 608 and offload some of the tasks of the GPUs 608 (e.g., freeing up more cycles of the GPUs 608 to perform other tasks). As an example, the accelerators 614 may be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to be suitable for acceleration. The term “CNN” as used herein may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

The accelerator(s) 614 (e.g., the hardware acceleration cluster) may include one or more deep learning accelerators (DLA). The DLA(s) may include one or more Tensor Processing Units (TPUs), which may be configured to provide an additional ten trillion operations per second for deep learning applications and inference. The TPUs may be accelerators configured and optimized to perform image processing functions (e.g. for CNNs, RCNNs, etc.). The DLA(s) may be further optimized for a specific set of types of neural networks and floating point operations as well as inference. The design of the DLA(s) can deliver more performance per millimeter than a typical general purpose GPU and far exceeds the performance of a CPU. The TPU(s) can perform multiple functions, including a single instance convolution function, e.g. B. INT8, INT16 and FP16 data types supported for both features and weights, as well as post-processing functions.

The DLA(s) can quickly and efficiently run neural networks, particularly CNNs, on processed or raw data for a variety of functions, including, for example, and without limitation: A CNN for object identification and recognition using data from camera sensors; a CNN for distance estimation with data from camera sensors, a CNN for detection and identification of emergency vehicles and detection with data from microphones; a CNN for facial recognition and vehicle owner identification using data from camera sensors and/or a CNN for security and/or safety-related events.

The DLA(s) may perform any function of the GPUs 608, and by using an inference accelerator, for example, a developer may target either the DLA(s) or the GPUs 608 for each function. For example, the designer can process CNNs and sliding Focus comma operations on the DLA(s) and leave other functions to the GPU(s) 608 and/or other accelerators 614.

The accelerator(s) 614 (e.g., the hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may alternatively be referred to as a vision accelerator in this document. The PVA(s) may be designed and/or configured to accelerate machine vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, augmented reality (AR) and/or virtual reality (VR) applications. The PVA(s) can provide a balance between performance and flexibility. By way of example and without limitation, all PVA(s) may include any number of Reduced Instruction Set Computer (RISC) cores, direct memory access (DMA), and/or any number of vector processors.

RISC cores may interact with image sensors (e.g., the image sensors of any of the cameras described herein), image signal processor(s), and/or the like. Each of the RISC cores can contain any amount of memory. The RISC cores may use any of a number of protocols depending on the embodiment. In some examples, the RISC cores may run a real-time operating system (RTOS). The RISC cores may be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and/or memory devices. For example, the RISC cores may include an instruction cache and/or tightly coupled RAM.

The DMA may allow components of the PVA(s) to access system memory independently of the CPU(s) 606. The DMA may support any number of features used to provide optimization of the PVA, including, but not limited to, support for multidimensional addressing and/or circular addressing. In some examples, the DMA may support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block gradation, vertical block gradation, and/or depth gradation.

The vector processors may be programmable processors that may be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, DMA engine(s) (e.g., two DMA engines), and/or other peripherals. The vector processing subsystem may operate as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and/or a vector memory (e.g., VMEM). A VPU core may include a digital signal processor, such as a single instruction multiple data (SIMD) very long instruction word (VLI W) digital signal processor. The combination of SIMD and VLIW can increase throughput and speed.

Each of the vector processors may include an instruction cache and be coupled to dedicated memory. Therefore, in some examples, each of the vector processors may be configured to execute independently of the other vector processors. In other examples, vector processors included in a specific PVA may be configured to employ data parallelism. For example, in some embodiments, the plurality of vector processors included in a single PVA may execute the same computer vision algorithm but on different regions of an image. In other examples, the vector processors included in a specific PVA may simultaneously execute different computer vision algorithms on the same image, or even execute different algorithms on sequential images or portions of an image. Among other things, any number of PVAs may be included in the hardware acceleration cluster and any number of vector processors may be included in each of the PVAs. Additionally, the PVA(s) may include additional error correction code (ECC) memory to increase overall system security.

The accelerator(s) 614 (e.g., hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator(s) 614. In some examples, the on-chip memory may include at least 4 MB of SRAM, e.g. B. and without limitation consists of eight field-configurable memory blocks that can be accessed by both the PVA and the DLA. Any pair of memory blocks can include an advanced peripheral bus (APB) interface, a configuration circuit, a controller and a multiplexer. Any type of storage can be used. The PVA and DLA can access the memory over a backbone that provides the PVA and DLA with high-speed access to the memory. The backbone may include an on-chip computer vision network that connects the PVA and DLA to the memory (e.g., using the APB).

The on-chip computer vision network may include an interface that determines that both a PVA and a DLA provide operational and valid signals before transmitting any control signal/address/data. Such an interface can provide separate phases and separate channels for control signal/address/data transmission as well as burst-like communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, although other standards and protocols can also be used.

In some examples, the SoC 604 may include a real-time hardware accelerator for ray tracing monitoring, as in the filing on August 10, 2018 US Patent Application No. 16/101,232 described. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the positions and extents of objects (e.g. within a world model), to generate real-time visualization simulations, for RADAR signal interpretation, for sound propagation synthesis and/or or analysis, for simulation of SONAR systems, for general wave propagation simulation, for comparison with LIDAR data for localization purposes and/or for other functions and/or for other applications. In some embodiments, one or more Tree Traversal Units (TTUs) may be used to perform one or more ray tracing-related operations.

The accelerator(s) 614 (e.g. the hardware accelerator cluster) have a variety of possible uses for autonomous driving. The PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. PVA's capabilities are a good complement for algorithmic domains that require predictable, low-power, low-latency processing. In other words, the PVA shows good performance for semi-dense or dense regular computations, even on small datasets that require predictable runtimes with low latency and low power. Consequently, the PVAs associated with autonomous vehicle platforms are designed to execute classical computer vision algorithms because they are efficient at object detection and operate on integer mathematics.

For example, according to one embodiment of the technology, PVA is used to perform machine stereo vision. An algorithm based on semi-global matching may be used, although this should not be construed as a limitation. Many Level 3-5 autonomous driving applications require motion estimation/stereo matching on the fly (e.g. structure from motion, pedestrian detection, lane detection, etc.). The PVA can perform a machine stereo vision function on inputs from two monocular cameras.

In some examples, the PVA can be used to perform dense optical flow. Pursuant to processing raw RADAR data (e.g. using a 4D Fast Fourier Transform) to provide processed RADAR. In other examples, the PVA is used for runtime deep processing, e.g. B. Raw runtime data is processed to provide processed runtime data.

The DLA can be used to run any type of network to improve control and driving safety, including, for example, a neural network that outputs a level of confidence for each object detection. Such a confidence value can be interpreted as a probability or as providing a relative "weight" of each detection compared to other detections. The confidence value allows the system to make further decisions about which detections should be considered true positive detections rather than false positive detections. For example, the system may set a confidence threshold and only consider detections that exceed the threshold as true positive detections. In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform emergency braking, which is of course undesirable. Therefore, only the safest detections should be considered as triggers for AEB. The DLA can Run a neural network to regression the confidence value. The neural network can use as input at least a subset of parameters, such as: B. Bounding box dimensions, ground plane estimation (e.g. from another subsystem), Inertial Measurement Unit (IMU) sensor 666 output associated with vehicle 600 orientation, distance, 3D position estimates of the object obtained from the neural network and/or other sensors (e.g., LIDAR sensor(s) 664 or RADAR sensor(s) 660), among others.

The SoC 604 may include data storage 616 (e.g., memory). The data memory 616 may be on-chip memory of the SoC 604, which may store neural networks running on the GPU and/or the DLA. In some examples, data storage 616 may be large enough to store multiple instances of neural networks for redundancy and security. Data storage 612 may include L2 or L3 caches 612. The reference to the data memory 616 may include reference to the memory of the PVA, DLA, and/or other accelerators 614, as described herein.

The SoC 604 may include one or more processors 610 (e.g., embedded processors). The processors 610 may include a boot and power management processor, which may be a dedicated processor and subsystem to handle the boot power and management functions and associated security enforcement. The boot and power management processor may be part of the SoC 604 boot sequence and may provide runtime power management services. The boot power and management processor may provide clock frequency and voltage programming, support for system power state transitions, management of SoC 604 thermals and temperature sensors, and/or management of SoC 604 power states. Each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC 604 can use the ring oscillators to detect temperatures of CPUs 606, GPUs 608, and/or accelerators 614. If temperatures exceed a threshold, the boot and power management processor may enter a temperature error routine and place the SoC 604 into a lower power state and/or place the vehicle 600 into a safe stop mode (e.g., the vehicle 600 into a safe stop).

The processors 610 may also include a set of integrated processors that may serve as an audio processor. The audio processing engine can be an audio subsystem that enables full hardware support for multi-channel audio across multiple interfaces, as well as a wide and flexible range of audio I/O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor with dedicated RAM.

The processor(s) 610 may also be equipped with an always-on processor that provides the necessary hardware features to support low-power sensor management and power-on use cases. The always-on processing engine may include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I/O controller peripherals, and routing logic.

The processor(s) 610 may also include a security cluster engine that includes a dedicated processor subsystem for security management for automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, an interrupt controller, etc.), and/or routing logic. In a safe mode, the two or more cores can operate in lockstep mode, functioning as a single core with comparison logic to detect any differences between their operations.

The processor(s) 610 may also include a real-time camera engine, which may include a dedicated processor subsystem for real-time camera management.

The processor(s) 610 may also include a high dynamic range signal processor that includes an image signal processor that is part of the camera processing pipeline as a hardware engine.

The processors 610 may include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to generate the final image for the player window . The video image compositor can provide lens distortion correction on wide-angle cameras 670, surrounding cameras 674 and/or on sensors of the passenger compartment surveillance camera. The in-cabin security camera sensor is preferably monitored by a neural network running on another instance of the Advanced SoC and configured to detect and respond to events in the cabin. An in-cabin system can perform lip reading to activate cellular service and place a call, dictate email, change a vehicle's destination, activate or change a vehicle's infotainment system and its settings, or provide voice-activated Internet surfing. Certain features are only available to the driver when the vehicle is operating in an autonomous mode and are disabled otherwise.

The video image compositor may include advanced temporal noise reduction for both spatial and temporal noise reduction. For example, if there is motion in a video, noise reduction will weight the spatial information accordingly by reducing the weight of the information provided by neighboring frames. When an image or portion of an image does not contain motion, temporal noise reduction performed by the video image compositor may use information from the previous image to suppress noise in a current image.

The video image compositor can also be configured to perform stereo rectification on stereo input lens frames. The video image compositor can also be used for user interface creation when the operating system desktop is used and the GPUs 608 are not required to continuously render new interfaces. Even when the GPUs 608 is turned on and actively doing 3D rendering, the video image compositor can be used to offload the GPUs 608 to improve performance and responsiveness.

The SoC 604 may also include a mobile industry processor interface (MIPI) serial port for receiving video and input from cameras, a high-speed interface, and/or a video input block that may be used for camera and related pixel input functions. The SoC 604 may also include an input/output controller that may be controlled by software and may be used to receive I/O signals that are not assigned to a specific role.

The SoC 604 may also include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management, and/or other devices. The SoC 604 can be used to process data from cameras (e.g. connected via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g. LIDAR sensor(s) 664, RADAR sensor(s) 660, etc., which can be connected via Ethernet), data from bus 602 (e.g. speed of vehicle 600, steering wheel position, etc.), data from GNSS sensor(s) 658 (e.g. connected via Ethernet or CAN bus). become. The SoC 604 may also include dedicated high-performance mass storage controllers that may include their own DMA engines and that may be used to free the CPUs 606 from routine data management tasks.

The SoC 604 can be an end-to-end platform with a flexible architecture that spans automation levels 3 to 5, thus providing a comprehensive functional safety architecture that efficiently leverages computer vision and ADAS techniques for diversity and redundancy, one platform at a time flexible, reliable driving software stack, along with deep learning tools. The SoC 604 can be faster, more reliable and even more energy efficient and space-saving than conventional systems. Thus, the accelerator 614 in combination with the CPU 606, the GPU 608 and the data storage 616 can provide a fast, efficient platform for level 3-5 autonomous vehicles.

The technology therefore offers capabilities and functionality that cannot be achieved with conventional systems. For example, computer vision algorithms can be executed on CPUs using a high-level programming language such as B. the C programming language, can be configured to execute a wide variety of processing algorithms over a wide variety of visual data. However, the CPUs are often unable to meet the performance requirements of many computer vision applications, such as: B. in terms of execution time and power consumption. In particular, many CPUs are unable to execute complex, real-time object detection algorithms required in in-vehicle ADAS applications and in practical Level 3-5 autonomous vehicles.

Unlike traditional systems, the technology described here, by providing a CPU complex, a GPU complex and a hardware acceleration cluster, allows multiple neural networks to be performed simultaneously and/or sequentially and the combination of the results to achieve Level 3-5 functionality for autonomous driving. For example, a CNN running on the DLA or dGPU (e.g., the GPU 620) may include text and word recognition that allows the supercomputer to read and understand traffic signs, including signs for which the neural Network has not been specially trained. The DLA may further include a neural network capable of identifying, interpreting and providing a semantic understanding of characters and passing this semantic understanding to the path planning modules running on the CPU complex.

As another example, multiple neural networks may be running simultaneously, as required for driving at Level 3, 4, or 5. For example, a warning sign that reads “Caution: Flashing lights indicate icing” along with an electric light can be interpreted independently or jointly by multiple neural networks. The sign itself can be identified as a traffic sign by a first deployed neural network (e.g. a trained neural network) and the text “Flashing lights indicate referral” can be interpreted by a second deployed neural network, which is the vehicle's path planning software (which preferably runs on the CPU complex) informs you that if blinking lights are detected, there is icing. The flashing light can be identified by operating a third deployed neural network over multiple frames and informing the vehicle's path planning software of the presence (or absence) of flashing lights. All three neural networks can run simultaneously, e.g. B. within the DLA and/or on the GPU 608.

In some examples, a facial recognition and vehicle identification CNN may use data from camera sensors to identify the presence of an authorized driver and/or vehicle owner 600. The always-on sensor processing engine may be used to unlock the vehicle when the owner is away from the vehicle Driver's door approaches and lights turn on, and to disarm the vehicle in security mode when the owner leaves the vehicle. In this way, the SoCs 604 ensure security against theft and/or theft.

In another example, an emergency vehicle detection and identification CNN may use data from microphones 696 to detect and identify emergency vehicle sirens. Unlike traditional systems that use general classifiers to detect sirens and extract features manually, the SoCs use 604 CNN to classify environmental and urban sounds as well as classify visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative approach speed of the emergency vehicle (e.g., using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the vehicle's local operational area, as identified by GNSS sensor(s) 658. Consequently, for example, when operating in Europe, the CNN attempts to identify European sirens, and in the United States, the CNN attempts to identify only North American sirens. Once an emergency vehicle is detected, a control program may be used to execute an emergency vehicle safety routine, slow the vehicle, pull to the side of the road, turn off the vehicle, and/or idle the vehicle using the ultrasonic sensors 662, until the emergency vehicle drives past.

The vehicle may include a CPU 618 (e.g., discrete CPU or dCPU) that may be coupled to the SoC 604 via a high-speed connection (e.g., PCIe). The CPU 618 can e.g. B. contain an X86 processor. The CPU 618 may be used to perform a variety of functions, including, for example, assessing potentially inconsistent results between ADAS sensors and the SoC 604 and/or monitoring the status and health of the controllers 636 and/or the infotainment SoC 630.

The vehicle 600 may include a GPU 620 (e.g., discrete GPU or dGPU) that may be coupled to the SoC 604 via a high-speed connection (e.g., NVLINK from NVIDIA). The GPU 620 can provide additional artificial intelligence functions such as: by executing redundant and/or different neural networks, and may be used to train and/or update neural networks based on inputs (e.g., sensor data) from sensors of the vehicle 600.

The vehicle 600 may also include the network interface 624, which may include one or more radio antennas 626 (e.g., one or more radio antennas for various communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network network interface 624 may be used to establish wireless connections over the Internet to the cloud (e.g., to the server 678 and/or other network devices), to other vehicles, and/or to computing devices (e.g., passenger client devices ) to enable. To communicate with other vehicles, a direct link can be established between the two vehicles and/or an indirect link can be established (e.g. via networks and via the Internet). Direct links can be established using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link may provide the vehicle 600 with information about vehicles in the vicinity of the vehicle 600 (e.g., vehicles in front of, to the side of, and/or behind the vehicle 600). This function may be part of a cooperative adaptive cruise control function of the vehicle 600.

The network network interface 624 may have an SoC that provides modulation and demodulation and allows the controller 636 to communicate over wireless networks. The network interface 624 may include a radio frequency front end for up-conversion from baseband to radio frequency and down conversion from radio frequency to baseband. The frequency conversions can be carried out by well-known processes and/or using superposition methods. In some examples, the high frequency front-end functionality may be provided by a separate chip. The network interface may include wireless functionality for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN and/or other wireless protocols.

The vehicle 600 may also include data storage 628, which may also include off-chip memory (e.g., off-the-SoC 604). The data memories 628 may include one or more storage elements including RAM, SRAM, DRAM, VRAM, flash, hard drives, and/or other components and/or devices capable of storing at least one bit of data.

The vehicle 600 may also include one or more GNSS sensors 658. The GNSS sensors 658 (e.g., GPS, supported GPS sensors, differential GPS (DGPS), etc.) assist in mapping, sensing, occupancy grid generation, and/or path planning. Any number of GNSS sensors 658 can be used, including e.g. B. a GPS device that uses a USB port with an Ethernet-to-serial bridge (RS-232).

The vehicle 600 may also include one or more RADAR sensors 660. The RADAR sensor 660 can be used by the vehicle 600 for remote vehicle detection, even in darkness and/or severe weather. The functional safety levels of the RADAR can be ASIL B. The RADAR sensor 660 may use the CAN and/or the bus 602 (e.g., to transmit the data generated by the RADAR sensor 660) to control and access data for object tracking, which may be accessed via Ethernet in some examples to access raw data. A wide variety of RADAR sensor types can be used. For example, the RADAR sensors 660 can be used without restrictions as front, rear and side RADAR. In some examples, pulse Doppler RADAR sensors are used.

The RADAR sensors 660 can include various configurations, e.g. E.g. long range with narrow field of view, short range with wide field of view, short side coverage, etc. In some examples, a long range RADAR may be used for adaptive cruise control. The long-range RADAR systems can provide a wide field of view realized by two or more independent scans, e.g. B. within a range of 250 m. The RADAR sensors 660 can help distinguish between static and moving objects and can be used by ADAS systems for emergency braking support and collision warning. Long-range RADAR sensors may include a monostatic multimodal RADAR with multiple (e.g. six or more) fixed RADAR antennas and a high-speed CAN and FlexRay interface. In a six antenna example, the central four antennas may produce a focused beam pattern designed to record the surroundings of the vehicle 600 at higher speeds with minimal interference from traffic in adjacent lanes. The other two antennas can expand the field of view, making it possible to quickly detect vehicles entering or leaving the path of vehicle 600.

For example, medium-range RADAR systems can include a range of up to 660 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 650 degrees (rear). Short-range RADAR systems may include, without limitation, RADAR sensors designed for installation at either end of the rear bumper. When the sensor system is installed at both ends of the rear bumper, such a RADAR sensor system can produce two beams that constantly monitor the blind spot behind and alongside the vehicle.

Short-range RADAR systems can be used in an ADAS system for blind spot detection and/or lane change assistance.

The vehicle 600 may also contain ultrasonic sensors 662. The ultrasonic sensors 662, which may be located at the front, rear, and/or sides of the vehicle 600, may be used for parking and/or for creating and updating an occupancy grid. A variety of ultrasonic sensors 662 and different ultrasonic sensors 662 for different detection ranges (e.g. 2.5 m, 4 m) can be used. The 662 ultrasonic sensors can work with functional safety level ASIL B.

The vehicle 600 may contain LIDAR sensors 664. The LIDAR sensors 664 can be used for object and pedestrian detection, emergency braking, collision avoidance and/or other functions. The LIDAR sensors 664 can be functional safety level ASIL B. In some examples, the vehicle 600 may include multiple LIDAR sensors 664 (e.g., two, four, six, etc.) that may use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

In some examples, the LIDAR sensors 664 may provide a list of objects and their distances for a 360-degree field of view. Commercially available LIDAR sensors 664 may have an advertised range of approximately 600 m, an accuracy of 2 cm to 3 cm, and, for example, support for a 600 Mbps Ethernet connection. In some examples, one or more non-protruding LIDAR sensors 664 may be used. In such examples, the LIDAR sensors 664 may be implemented as a small device that may be embedded in the front, rear, sides, and/or corners of the vehicle 600. The LIDAR sensor(s) 664 in such examples can provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees with a range of 200 m even for low reflectivity objects. Front-mounted LIDAR sensors 664 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

In some examples, LIDAR technologies, such as 3D flash LIDAR, may also be used. 3D flash LIDAR uses a laser flash as a transmission source to illuminate the vehicle's surroundings up to a distance of approximately 200 m. A flash LIDAR unit includes a receptor that records the laser pulse travel time and reflected light at each pixel, which in turn corresponds to the range from the vehicle to the objects. Flash LIDAR can enable highly accurate and distortion-free images of the environment to be created with each laser flash. In some examples, four LIDAR lightning sensors may be deployed, one on each side of the vehicle 600. Available 3D flash LIDAR systems include a solid-state 3D staring array LIDAR camera with no moving parts other than a fan (e.g . a non-scanning LIDAR device). The flash LIDAR device can use a 5-nanosecond Class I (eye-safe) laser pulse per image and can capture the reflected laser light in the form of the 3D range point clouds and co-registered intensity data. By using Flash LIDAR and because Flash LIDAR is a solid-state device with no moving parts, the 664 LIDAR sensor may be less susceptible to motion blur, vibration and/or shock.

The vehicle may also include IMU sensors 666. The IMU sensors 666 may be located in the center of the rear axle of the vehicle 600 in some examples. The IMU sensors 666 can e.g. B. include one or more accelerometers, magnetometers, gyroscopes, magnetic compasses and / or other types of sensors. In some examples, such as For example, in six-axis applications, the IMU sensors 666 may include accelerometers and gyroscopes, while in nine-axis applications, the IMU sensors may include 666 accelerometers, gyroscopes, and magnetometers.

In some embodiments, the IMU sensors 666 may be implemented as a miniature GPS-Aided Inertial Navigation System (GPS/INS) that includes micro-electro-mechanical systems (MEMS) inertial sensors, a highly sensitive GPS receiver, and advanced Kalman filtering algorithms combined to provide estimates of position, velocity and inclination. In some examples, the IMU sensors 666 may enable the vehicle 600 to estimate heading without input from a magnetic sensor by observing and correlating the speed changes from GPS directly on the IMU sensors 666. In some examples, IMU sensors 666 and GNSS sensors 658 may be combined into a single integrated unit.

The vehicle may be equipped with microphones 696 placed in and/or around the vehicle 600. The microphones 696 can be used, among other things, to detect and identify emergency vehicles.

The vehicle may also include any number of camera types, including stereo cameras 668, wide-angle cameras 670, infrared cameras 672, ambient cameras 674, long-range and/or medium-range cameras 698, and/or other camera types. The cameras can be used to capture image data around an entire periphery of the vehicle 600. The types of cameras used depend on the embodiments and requirements for the vehicle 600, and any combination of camera types can be used to provide the required coverage around the vehicle 600. In addition, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and/or another number of cameras. For example, and without limitation, the cameras may support Gigabit Multimedia Serial Link (GMSL) and/or Gigabit Ethernet. Each camera is referred to in this document 6A and 6B described.

The vehicle 600 may also include vibration sensors 642. The vibration sensors 642 can detect vibrations from components of the vehicle, e.g. B. the axes. For example, changes in vibration may indicate a change in the road surface. In another example, when using two or more vibration sensors 642, the differences between the vibrations can be used to determine the friction or slip of the road surface (e.g., if the difference in vibration is between an electrically powered and a freely rotating axle).

The vehicle 600 may include an ADAS system 638. The ADAS system 638 may include an SoC in some examples. The ADAS system 638 can provide autonomous/adaptive/automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (automatic emergency braking - AEB), lane detection system (lane departure warning - LDW), lane keeping assistant (lane keep assistant - LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning systems (collision warning system - CWS), lane centering (LC) and/or other features and functionalities.

The ACC systems may use RADAR sensors 660, LIDAR sensors 664 and/or cameras. The ACC systems may include a longitudinal ACC and/or a transverse ACC. The longitudinal ACC monitors and controls the distance to the vehicle immediately in front of the vehicle 600 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicles in front. The lateral ACC carries out the distance measurement and advises the vehicle 600 to change lanes if necessary. The transverse ACC is connected to other ADAS applications such as LCA and CWS.

CACC uses information from other vehicles that can be received via network interface 624 and/or wireless antenna(s) 626 from other vehicles via a wireless connection or indirectly via a network connection (e.g., via the Internet). Direct links may be provided through a vehicle-to-vehicle (V2V) communications link, while indirect links may be provided through an infrastructure-to-vehicle (I2V) communications link. The V2V communication concept usually provides information about the vehicles immediately in front (e.g. vehicles immediately in front of and on the same road as vehicle 600), while the I2V communication concept provides information about the other traffic in front. CACC systems can include either or both of !2V and V2V information sources. Given the information about the vehicles in front of vehicle 600, CACC could be more reliable and have the potential to improve traffic flows and reduce road congestion.

FCW systems are designed to alert the driver to a hazard so the driver can take corrective action. FCW systems use a forward-facing camera and/or RADAR sensors 660 coupled to a dedicated processor, DSP, FPGA and/or ASIC that is electrically coupled to driver feedback, e.g. B. a display, speakers and / or a vibrating component. FCW systems can provide an alert, e.g. B. in the form of a sound, a visual warning, a vibration and / or a quick braking pulse.

AEB systems detect an impending collision with another vehicle or object and can automatically apply the brakes if the driver does not take corrective action within a certain time or distance parameter. AEB systems may use forward-facing cameras and/or RADAR sensors 660 coupled to a dedicated processor, DSP, FPGA and/or ASIC. When the AEB system detects a danger, it typically first alerts the driver to take a corrective action to avoid a collision and, if the driver does not take a corrective action, the AEB system may automatically apply the brakes in the effort to prevent or at least mitigate the effects of the predicted collision. AEB systems may include techniques such as dynamic brake support and/or impending collision braking.

LDW systems provide visual, audible and/or tactile warnings, such as: B. Steering wheel or seat vibrations to alert the driver when the vehicle 600 crosses the road markings. An LDW system will not activate if the driver indicates intentional lane departure by using the turn signal. LDW systems may use forward-facing cameras coupled to a dedicated processor, DSP, FPGA and/or ASIC that are electrically coupled to driver feedback such as: B. a display, a speaker and / or a vibrating component.

LKA systems are a variant of the LDW systems. LKA systems provide steering inputs or braking to correct the vehicle 600 when the vehicle 600 leaves the roadway.

BSW systems detect and warn the driver of vehicles in an automobile's blind spot. BSW systems can provide a visual, audible and/or tactile alarm to indicate that merging or changing lanes is unsafe. The system can provide an additional warning when the driver uses a turn signal. BSW systems may use rear- and side-facing camera(s) and/or RADAR sensor(s) 660 coupled to a dedicated processor, DSP, FPGA and/or ASIC, i.e. electrically coupled to a Driver feedback coupled, such as a display, a speaker and / or a vibrating component.

RCTW systems can provide visual, audible and/or tactile notifications when an object is detected outside the range of the rearview camera when the vehicle 600 is reversing. Some RCTW systems include the AEB system to ensure that the vehicle brakes are applied to avoid a collision. RCTW systems may use one or more rear-facing RADAR sensors 660 coupled with a dedicated processor, DSP, FPGA and/or ASIC that is electrically coupled to driver feedback, e.g. B. a display, a speaker and / or a vibrating component.

Traditional ADAS systems can be prone to false positives, which can be disruptive and distracting to the driver, but are usually not catastrophic because the ADAS systems alert the driver and allow the driver to decide whether a safety condition actually exists exists and to act accordingly. However, in an autonomous vehicle 600, in the event of conflicting results, the vehicle 600 itself must decide whether the result of a primary or a secondary computer (e.g. a first controller 636 or a second controller 636) should be taken into account. For example, in some embodiments, the ADAS system 638 may be a backup and/or secondary computer to provide perception information to a rationality module of the backup computer. The backup computer's rationality monitor can run redundant, diverse software on hardware components to detect errors in perception and dynamic driving tasks. Outputs from the ADAS system 638 may be provided to a monitoring MCU. If the outputs of the primary and secondary computers conflict with each other, the monitoring MCU must determine how to resolve the conflict to ensure safe operation.

In some examples, the primary computer may be configured to provide the monitoring MCU with a confidence score that indicates a confidence of the primary computer for the selected result. If the confidence score exceeds a threshold, this monitoring MCU can follow the lead of the primary computer, regardless of whether this secondary computer provides a conflicting or inconsistent result. If the confidence score does not reach the threshold and the primary and secondary computers report different results (e.g., the contradiction), the monitoring MCU can mediate between the computers to determine the appropriate result.

The monitoring MCU may be configured to execute neural network(s) trained and configured to determine, based at least on outputs from the primary computer and the secondary computer, the conditions under which the secondary Computer provides false alarms. Consequently, the neural network(s) in the monitoring MCU can learn when the output of a secondary computer can and cannot be trusted. For example, if the secondary computer is a RADAR-based FCW system, neural network(s) in the surveillance MCU can learn when the FCW system identifies metallic objects that are not actually hazards, such as a drain grate or manhole cover, that triggers an alarm. Similarly, if the secondary computer is a camera-based LDW system, a neural network in the surveillance MCU can learn to override the LDW when cyclists or pedestrians are present and lane departure is actually the safest maneuver. In embodiments that include one or more neural networks running on the monitoring MCU, the monitoring MCU may include at least one DLA or GPU capable of executing the neural network(s) with associated memory. In preferred embodiments, the monitoring MCU may be included and/or incorporated as part of the SoC 604.

In other examples, the ADAS system 638 may include a secondary computer that performs the ADAS functionality using conventional computer vision rules. Thus, the secondary computer can use classical computer vision rules (if-then), and the presence of a neural network network(s) in the monitoring MCU can improve reliability, security and performance. For example, diverse implementation and intentional non-identity makes the overall system more fault-tolerant, especially to errors caused by software (or software-hardware interfaces) functionality. For example, if there is a software bug or error in the software running on the primary computer and the non-identical software code running on the secondary computer provides the same overall result, the monitoring MCU can have greater confidence that the overall result is correct and the bug in the software or hardware on the primary computer does not cause a significant error.

In some examples, the output of the ADAS system 638 may be fed into the primary computer's perception block and/or the primary computer's dynamic driving task block. For example, if the ADAS system 638 displays a collision warning due to an object immediately ahead, the perception block may use this information in identifying objects. In other examples, the secondary computer may have its own neural network trained, thereby reducing the risk of false positives, as described herein.

The vehicle 600 may also include the infotainment SoC 630 (e.g., an in-vehicle infotainment (IVI) system). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 630 may include a combination of hardware and software that may be used to provide the vehicle 600 with audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., . TV, movies, streaming, etc.), telephone (e.g. hands-free function), network connectivity (e.g. LTE, Wi-Fi, etc.) and/or information services (e.g. navigation systems, rear parking sensors, a radio data system , vehicle-related information such as fuel level, total distance traveled, brake fluid level, oil level, door open/close, air filter information, etc.). For example, the infotainment SoC 630 can support radios, CD players, navigation systems, video players, USB and Bluetooth connectivity, carputer, in-car entertainment, Wi-Fi, steering wheel audio controls, hands-free voice control, a heads-up display (HUD), an HMI display 634, a telematics device, a control panel (e.g., for controlling and/or interacting with various components, functions and/or systems), and/or other components. The infotainment SoC 630 may also be used to provide information (e.g., visual and/or audible) to users of the vehicle put, e.g. B. information from the ADAS system 638, autonomous driving information such as planned vehicle maneuvers, trajectories, environmental data (e.g. intersection information, vehicle information, road information, etc.) and / or other information.

The infotainment SoC 630 may include GPU functionality. The infotainment SoC 630 may communicate with other devices, systems, and/or components of the vehicle 600 via the bus 602 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 630 may be coupled with a monitoring MCU so that the infotainment system's GPU can perform some self-driving functions if the primary controller 636 (e.g., the primary and/or backup computer of the Vehicle 600) fails. In such an example, the infotainment SoC 630 may place the vehicle 600 into safe stop mode, as described herein.

The vehicle 600 may also include an instrument cluster 632 (e.g., a digital dashboard, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 632 may include a controller and/or supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 632 may include a set of instruments such as a speedometer, fuel level, oil pressure, speedometer, odometer, turn signal, shift position indicator, seat belt warning light(s), parking brake warning light(s), engine malfunction light(s), airbag (SRS) system information, lighting controls, Include security system controls, navigation information, etc. In some examples, information may be displayed and/or shared between the infotainment SoC 630 and the instrument cluster 632. In other words, the instrument cluster 632 may be included as part of the infotainment SoC 630 or vice versa.

6D is a system diagram for communication between cloud-based servers and the example autonomous vehicle 600 6A , according to some embodiments of the present disclosure. The system 676 may include servers 678, networks 690 and vehicles including vehicle 600. The servers 678 may include a variety of GPUs 684(A)-684(H) (collectively referred to as GPUs 684), PCIe switches 682(A)-682(H) (collectively referred to as PCIe switches 682), and/or CPUs 680(A)-680(B) (collectively referred to as CPUs 680). The GPUs 684, the CPUs 680 and the PCle switches can be connected with high-speed connections such as the NVLink interfaces 688 and/or PCIe Connections 686 developed by NVIDIA. In some examples, the GPUs 684 are connected via NVLink and/or NVSwitch SoC and the GPUs 684 and the PCIe switches 682 are connected via PCIe connections. Although eight 684 GPUs, two 680 CPUs and two PCIe switches are shown, this should not be construed as limiting. Depending on the embodiment, each server 678 may include any number of GPUs 684, CPUs 680, and/or PCIe switches. For example, the servers 678 may each contain eight, sixteen, thirty-two, and/or more GPUs 684.

The servers 678 may receive image data representative of images showing unexpected or changing road conditions, such as, over the networks 690 and from the vehicles. E.g. recently started road works. The servers 678 may transmit, over the networks 690 and to the vehicles, neural networks 692, updated neural networks 692, and/or map information 694, including information about traffic and road conditions. The updates to the map information 694 may include updates to the HD map 622, such as: B. Information about construction sites, potholes, detours, floods and/or other obstacles. In some examples, the neural networks 692, the updated neural networks 692, and/or the map information 694 may result from new training and/or experiences represented in data from any number of vehicles in the environment, and/or based on training provided be performed in a data center (e.g., using servers 678 and/or other servers).

The servers 678 can be used to train machine learning models (e.g., neural networks) based on training data. The training data may be generated by vehicles and/or may be generated in a simulation (e.g. using a game engine). In some examples, the training data is tagged (e.g., when the neural network benefits from supervised learning) and/or undergoes other preprocessing, while in other examples, the training data is not tagged and/or preprocessed (e.g., . if the neural network does not require supervised learning). Training may be according to one or more classes of machine learning techniques, including but not limited to classes such as: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analyses), multilinear Subspace learning, manifold learning, representation learning (including surrogate dictionary learning), rule based machine learning, anomaly detection and any variants or combinations thereof. After training the machine learning models, the machine learning models may be used by the vehicles (e.g., transmitted to the vehicles over the network 690) and/or the machine learning models may be used by the servers 678 to remotely monitor the vehicles.

In some examples, the server 678 may receive data from the vehicles and apply the data to current real-time neural networks to enable intelligent real-time inferences. The servers 678 may include deep learning supercomputers and/or dedicated AI computers with GPUs 684, such as: B. DGX and DGX station machines developed by NVIDIA. However, in some examples, servers 678 may include deep learning infrastructure that uses only CPU-powered data centers.

The deep learning infrastructure of the servers 678 may enable rapid, real-time inference and use this capability to assess and verify the health of the processors, software, and/or associated hardware in the vehicle 600. For example, the deep learning infrastructure may receive regular updates from the vehicle 600, such as: B. a sequence of images and/or objects that the vehicle 600 has located in that sequence of images (e.g., through computer vision and/or other machine learning object classification techniques). The deep learning infrastructure can operate its own neural network to identify the objects and compare them with the objects identified by the vehicle 600. If the results do not agree and the infrastructure concludes that the AI in the vehicle 600 is defective, the server 678 may send a signal to the vehicle 600 instructing a fail-safe computer of the vehicle 600 to take control of the passengers to notify and carry out a safe parking maneuver.

For inference, servers 678 may include GPUs 684 and one or more programmable inference accelerators (e.g., NVIDIA TensorRT). The combination of GPU-powered servers and inference acceleration can enable real-time responsiveness. In other examples, if e.g. For example, if performance is less critical, servers powered by CPUs, FPGAs, and other processors can be used for derivation.

EXEMPLARY COMPUTING DEVICE

7 is a block diagram of an example computing device(s) 700 suitable for use in implementing some embodiments of the present disclosure. The computing device 700 may include an interconnection system 702 that directly or indirectly couples the following devices: memory 704, one or more central processing units (CPUs) 706, one or more graphics processing units (GPUs) 708, a communications interface 710, input/output (E /A) ports 712, input/output components 714, a power supply 716, and one or more display components 718 (e.g., displays), and one or more logic units 720. In at least one embodiment, the computing device(s) 700 may be a or multiple virtual machines (VM) and/or each of their components may include virtual components (e.g. virtual hardware components). Non-limiting examples may include one or more GPUs 708, one or more vGPUs, one or more CPUs 706 may include one or more include multiple vCPUs and/or one or more logic units 720 may include one or more virtual logic units. The computing device(s) 700 may thus include discrete components (e.g., a complete GPU associated with the computing device 700), virtual components (e.g., a portion of a GPU associated with the computing device 700), or a combination thereof.

Although the different blocks from 7 As shown connected to wires via connection system 702, this is not intended to be limiting and is for convenience only. For example, in some embodiments, a presentation component 718, such as. B. a display device, can be considered an I/O component 714 (e.g. if the display is a touchscreen). As another example, CPUs 706 and/or GPUs 708 may include memory (e.g., memory 704 may be representative of a storage device in addition to the memory of GPUs 708, CPUs 706, and/or other components). In other words, the computing device is off 7 merely illustrative. It does not distinguish between categories such as workstation, server, laptop, desktop, tablet, client device, mobile device, handheld device, game console, electronic control device ( electronic control unit (ECU)”, “virtual reality system” and/or other types of devices or systems, as they are all within the scope of the computing device 7 lay.

The connection system 702 may represent one or more connections or buses, e.g. B. an address bus, a data bus, a control bus or a combination thereof. The connection system 702 may include one or more types of buses or connections, e.g. B. an ISA bus (Industry Standard Architecture), an EISA bus (Extended Industry Standard Architecture), a VESA bus (Video Electronics Standards Association), a PCI bus (Peripheral Component Interconnect), a PCle bus (Peripheral Component Interconnect Express) and/or another bus or connection type. In some embodiments, there are direct connections between components. As an example, CPU 706 can be connected directly to memory 704. In addition, the CPU 706 can be connected directly to the GPU 708. For direct or point-to-point connection between components, the connection system 702 may include a PCIe connection to perform the connection. In these examples, no PCI bus needs to be included in the computing device 700.

Memory 704 may include a variety of computer-readable media. The computer-readable media can be any available media that the computing device 700 can access. The computer-readable media may include both volatile and non-volatile media as well as removable and non-removable media. By way of example and not limitation, the computer-readable media may include computer storage media and communication media.

The computer storage medium may include both volatile and non-volatile media and/or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules and/or other types of data. For example, memory 704 may store computer-readable instructions (e.g., representing one or more programs and/or program elements, e.g., an operating system). Storage media for computers include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, Digital Versatile Disks (DVD) or other optical media, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and which is accessible by the computing device 700 As used herein, computer storage media does not include signals per se.

The computer storage media may embody computer-readable instructions, data structures, program modules, and/or other types of data in a modulated data signal such as a carrier wave or other transport mechanism, and include any information delivery media. The term “modulated data signal” may refer to a signal that has one or more of its properties altered in such a way that information is encoded in the signal. By way of example, and not as a limitation, computer storage media may include wired media, such as a wired network or wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of all of the foregoing are also intended to be included within the scope of computer-readable media.

The CPUs 706 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computer 700 to perform one or more of the methods and/or processes described herein. The CPUs 706 may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a plurality of software threads simultaneously. The CPUs 706 may include any type of processor and may include different types of processors depending on the type of computing device 700 being implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). Depending on the computing device 700, it can be, for example, an Advanced RISC Machine (ARM) processor with reduced instruction set computing (RISC) or an x86 processor with complex instruction set computing (CISC). The computing device 700 may in addition to one or more microprocessors or additional coprocessors, such as. B. mathematical coprocessors, one or more CPUs 706 included.

The GPUs 708 may further or alternatively be configured by the CPUs 706 to execute at least some computer-readable instructions to control one or more components of the computer 700 to perform one or more of the methods and/or processes described herein. One or more GPUs 708 may be an integrated GPU (e.g., with one or more CPUs 706) and/or one or more GPUs 708 may be a separate GPU. In embodiments, one or more GPUs 708 may be a coprocessor of one or more CPUs 706. The GPUs 708 may be used by the computing device 700 to render graphics (e.g., 3D graphics) or perform more general calculations are used. For example, the GPUs 708 can be used for general-purpose computing on GPUs (GPGPU). The GPUs 708 can include hundreds or thousands of cores that can process hundreds or thousands of software threads simultaneously. The GPUs 708 may generate pixel data for output images in response to rendering commands (e.g., rendering commands from the CPUs 706 received via a host interface). The GPUs 708 can have graphics memory such as. B. Display memory, for storing pixel data or other suitable data, such as GPGPU data. The display memory may be included as part of the memory 704. The GPUs 708 may include two or more GPUs operating in parallel (e.g., over a connection). The connection can connect the GPUs directly (e.g. using NVLINK) or can connect the GPUs via a switch (e.g. using NVSwitch). When combined together, each GPU can generate 708 pixel data or GPGPU data for different parts of an output or for different outputs (e.g. a first GPU for a first image and a second GPU for a second image). Each GPU can include its own memory or can share memory with other GPUs.

The logic units 720 may further or alternatively be configured by the CPUs 706 and/or the GPUs 708 to execute at least some computer-readable instructions to control one or more components of the computer 700 to perform one or more of the methods and/or described herein Execute processes. In embodiments, the CPUs 706, the GPUs 708, and/or the logic units 720 may discretely or collectively perform any combination of the methods, processes, and/or portions thereof. One or more of the logic units 720 may be part of one and/or more CPUs 706 and/or GPUs 708 and/or one or more of the logic units 720 may be discrete components or otherwise external to the CPUs 706 and/or the GPUs 708. In embodiments, one or more of the logic units 720 may be a coprocessor of one or more of the CPUs 706 and/or one or more of the GPUs 708.

Examples of the logic unit(s) 720 include one or more processing cores and/or components thereof, such as data processing units (DPUs), tensor cores (TC), tensor processing units (TPU), visual pixel cores Pixel Visual Cores (PVC), Vision Processing Unit (VPU), Graphics Processing Cluster (GPC), Texture Processing Cluster (TPC), Streaming Multiprocessors (SM), Tree Traversal Unit (TTU) , Artificial Intelligence Accelerator (AIA), Deep Learning Accelerator (DLA), Arithmetic Logic Units (ALU), Application Specific Integrated Circuits (ASIC), Floating Point Unit (FPU), Input/Output ( I/O) elements, elements for interconnecting peripheral components (PCI) or express interconnecting peripheral components (peripheral component interconnect express - PCIe) and/or the like.

The communications interface 710 may include one or more receivers, transmitters, and/or transceivers that enable the computing device 700 to communicate with other computing devices via an electronic communications network, including wired and/or wireless communications. The communication interface 710 may include components and functions that enable communication over a number of different networks, e.g. B. wireless networks (e.g. Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g. communication via Ethernet or InfiniBand), low-power wide-area networks (e.g. LoRaWAN, Sigfox, etc.) and/or the Internet. In one or more embodiments, the logic unit(s) 720 and/or the communications interface 710 may include one or more data processing units (DPUs) for transmitting data received over a network and/or through the interconnection system 702 directly to one or more GPU(s) 708 (e.g. a memory thereof).

The I/O ports 712 may enable the computing device 700 to be logically coupled to other devices, including the I/O components 714, the presentation component(s) 718, and/or other components, some of which are built into the computing device 700 (e.g. integrated). Illustrative I/O components 714 include a microphone, a mouse, a keyboard, a joystick, a gamepad, a game controller, a satellite dish, a scanner, a printer, a wireless device, etc. The I/O components 714 may be natural Provide a natural user interface (NUI) that processes air gestures, voice input, or other physiological input generated by a user. In some cases, inputs can be transferred to an appropriate network element for further processing. A NUI can do any combination of voice recognition, tactile recognition, face recognition, biometric recognition, both on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch detection Implementation (as described in more detail below) that is connected to a display of the computing device 700. The computing device 700 may include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. In addition, the computing device 700 may include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit (IMU)) that enable motion detection. In some examples, the output of the accelerometers or gyroscopes from the computing device 700 may be used to display immersive augmented reality or virtual reality.

The power supply 716 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 716 may provide power to the computing device 700 to enable the components of the computing device 700 to function.

The display components 718 may include a display (e.g., a monitor, a touch screen, a television screen, a heads-up display (HUD), other display types, or a combination thereof), speakers, and/or other display components. The presentation components 718 may receive data from other components (e.g., GPUs 708, CPUs 706, DPUs, etc.) and output the data (e.g., as image, video, sound, etc.).

EXEMPLARY DATA CENTER

8th illustrates an example data center 800 that may be used in at least one embodiment of the present disclosure. The data center 800 may include a data center infrastructure layer 810, a framework layer 820, a software layer 830, and/or an application layer 840.

As in 8th As shown, the data center infrastructure layer 810 may include a resource orchestrator 812, clustered computing resources 814, and node computing resources (“node CR”) 816(1)-816(N), where “N” represents any positive integer. In at least one embodiment, the node CRs 816(1)-816(N) may include any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc .), storage devices (e.g., dynamic read-only memory), data storage devices (e.g., solid-state or hard disk drives), network input/output devices (NW I/O), network switches, virtual machines (VMs ), power modules and/or cooling modules, etc. include, but are not limited to. In at least one embodiment, one or more node CRs among node CRs 816(1)-816(N) may correspond to a server having one or more of the aforementioned computing resources. Additionally, in some embodiments, the node CRs 816(1)-8161(N) may include one or more virtual components, such as vGPUs, vCPUs, and/or the like, and/or may include one or more of the node CRs 816(1 )-816(N) correspond to a virtual machine (VM).

In at least one embodiment, the grouped computing resources 814 may be separate groupings of node C.R. 816 housed within one or more racks (not shown) or many racks housed in data centers in different geographic locations (also not shown). Separate groupings of node C.R. 816 within the grouped computing resources 814 may include grouped computing, network, memory, or data storage resources that may be configured or assigned to support one or more workloads. In at least one embodiment, multiple node C.R. 816, including CPUs, GPUs, DPUs, and/or processors, may be grouped in one or more racks to provide computing resources to support one or more workloads. The one or more racks may also contain any number of power modules, cooling modules and/or network switches in any combination.

The resource orchestrator 812 may have one or more node C.R. Configure or otherwise control 816(1)-816(N) and/or grouped computing resources 814. In at least one embodiment, the resource orchestrator 812 may include a software design infrastructure (SDI) management entity for the data center 800. The resource orchestrator 812 may consist of hardware, software, or a combination thereof.

In at least one embodiment, as in 8th As shown, the framework layer 820 may include a task scheduler 833, a configuration manager 834, a resource manager 836, and/or a distributed file system 838. The framework layer 820 may include a framework for supporting software 832 of a software layer 830 and/or one or more applications 842 of an application layer 840. The software 832 or the application(s) 842 may each include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 820 may be any type of free and open source software web application framework, such as Apache Spark™ (hereinafter “Spark”), which provides the distributed file system 838 for large-scale data processing ( e.g. “Big Data”). In at least one embodiment, the task scheduler 833 may include a Spark driver to facilitate scheduling of workloads supported by different tiers of the data center 800. The configuration manager 834 may be able to configure different layers, such as the software layer 830 and the framework layer 820, which includes Spark and the distributed file system 838 to support large-scale data processing. The resource manager 836 may be capable of managing clustered or grouped computing resources allocated or assigned to support the distributed file system 838 and the task scheduler 833. In at least one embodiment, clustered or grouped computing resources may include the clustered computing resource 814 on the data center infrastructure layer 810. The resource manager 836 and the resource orchestrator 812 may coordinate to manage these associated or assigned computing resources.

In at least one embodiment, the software 832 included in the software layer 830 may include software that is represented by at least portions of the node C.R. 816(1)-816(N), the clustered computing resources 814 and/or the distributed file system 838 of the framework layer 820 is used. One or more types of software may include, but are not limited to, Internet web page scanning software, email virus scanning software, database software, and streaming video content software.

In at least one embodiment, the application(s) 842 included in the application layer 840 may include one or more types of applications that are supported by at least portions of the node C.R. 816(1)-816(N), the clustered computing resources 814 and/or the distributed file system 838 of the framework layer 820 can be used. One or more types of applications may include any number of a genomics application, a cognitive computing application, and a machine learning application that includes training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and/or or include, but is not limited to, other machine learning applications used in connection with one or more embodiments.

In at least one embodiment, any of the configuration manager 834, the resource manager 836, and the resource orchestrator 812 may implement any number and type of self-modifying actions based on any amount and type of data collected in any technically feasible manner became. Self-modifying actions can relieve a data center operator of the data center 800 from making potentially poor configuration decisions and potentially avoiding underutilized and/or poorly performing sections of a data center.

The data center 800 may include tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models in accordance with one or more embodiments described herein. For example, a machine learning model(s) may be trained by calculating weighting parameters according to a neural network architecture using software and/or computing resources described above with respect to data center 800. In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to the data center 800 by using weighting parameters specified by one or more multiple training techniques may be calculated, such as, but not limited to, those described herein.

In at least one embodiment, the data center 800 may use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and/or other hardware (or corresponding virtual computing resources) to perform training and/or inference using the resources described above. Additionally, one or more of the software and/or hardware resources described above may be configured as a service to enable users to train or perform inference of information, such as image recognition, speech recognition, or other artificial intelligence services.

EXAMPLE NETWORK ENVIRONMENTS

Network environments suitable for use in implementing embodiments of the disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and/or other device types. The client devices, servers, and/or other types of devices (e.g., any device) may reside on one or more instances of the computing device 700 in 7 be implemented - each device can e.g. B. include similar components, features and/or functions of the computing device(s) 700. Additionally, when implementing backend devices (e.g., servers, NAS, etc.), the backend devices may also be part of a data center 800, an example of which is shown in 8th is described in more detail.

Components of a network environment may communicate with each other over one or more networks, which may be wired, wireless, or both. The network may include multiple networks or a network of networks. For example, the network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet and/or a public switched telephone network (PSTN), and/or one or more private networks. If the network includes a wireless telecommunications network, components such as a base station, a communications tower, or even access points (as well as other components) can provide wireless connectivity.

Compatible network environments may include one or more peer-to-peer network environments - in which case a server may not be included in a network environment - and one or more client-server network environments - in which case one or more servers may be included in a network environment be. In peer-to-peer network environments, the functionality described herein with respect to one or more servers may be implemented on any number of client devices.

In at least one embodiment, a network environment may include one or more cloud-based network environments, a distributed computing environment, a combination thereof, etc. A cloud-based network environment may include a framework layer, a task scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and/or edge servers. A framework layer may include a framework for supporting software of a software layer and/or one or more applications of an application layer. The software or application(s) may each include web-based service software or applications. In embodiments, one or more of the client devices may use the web-based service software or applications (e.g., by accessing the service software and/or applications via one or more application programming interfaces (API)). The framework layer may be, but is not limited to, a type of free and open source software web application framework, such as using a distributed file system to process large amounts of data (e.g., "big data").

A cloud-based network environment may provide cloud computing and/or cloud storage that perform any combination of computing and/or data storage functions described herein (or one or more portions thereof). Each of these different functions may be distributed across multiple locations of central or core servers (e.g., one or more data centers that may be distributed across a state, region, country, globe, etc.). When a connection to a user (e.g., a client device) is relatively close to one or more edge servers, one or more core servers may assign at least some functionality to the edge server(s). A cloud-based network environment may be private (e.g., limited to a single organization), may be public (e.g., available to many organizations), and/or a combination thereof (e.g., a hybrid cloud environment).

The client devices may include at least some of the components, features, and functionality of the example computing device(s) 700 described herein 7 include. By way of example and not limitation, a client device may be a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality Headset, global positioning system or device (GPS), video player, video camera, surveillance device or system, vehicle, boat, aircraft, virtual machine, drone, robot, portable communication device, hospital device, gaming device or system, entertainment system, vehicle computer system , embedded system controller, remote control, home appliance, consumer electronics device, workstation, edge device, any combination of these outlined devices, or any other suitable device.

The disclosure may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program modules that are executed by a computer or another computer, such as a personal data assistant or other handheld device. In general, program modules including routines, programs, objects, components, data structures, etc. refer to code that performs specific tasks or implements specific abstract data types. The disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general purpose computers, more specialized computing devices, etc. The disclosure may also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network .

As used herein, the use of “and/or” in reference to two or more elements is to be construed as just one element or a combination of elements. For example, "Element A, Element B and/or Element C" can only contain Element A, only Element B, only Element C, Element A and Element B, Element A and Element C, Element B and Element C, or Elements A, B and C included. Additionally, “at least one of Element A or Element B” may include at least one of Element A, at least one of Element B, or at least one of Element A and at least one of Element B. Further, “at least one of element A and element B” may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

The subject matter of the present disclosure is detailed herein to meet legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have contemplated that the claimed subject matter could be embodied in other ways to include other steps or combinations of steps similar to those described in this document in connection with other current or future technologies. Although the terms "step" and/or "block" may be used in this document to refer to different elements of the methods used, the terms should not be interpreted as implying any particular order among or between the various steps mentioned herein. unless the sequence of the individual steps is expressly described.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 16101232 [0082]

Claims (20)

Processor comprising: one or more circuits to calculate one or more control operations for an ego machine using a machine learning model, wherein one or more parameters of the machine learning model are updated during training based at least in part on a data set that is synthetic Data generated using a simulator, wherein one or more locations of one or more objects in an environment represented by the synthetic data are sampled based at least in part on one or more spatial priors corresponding to the environment. Processor after Claim 1 , wherein the one or more spatial priors are generated by sampling the one or more locations proportional to a longitudinal distance of the one or more objects to a synthetic ego machine represented by the synthetic data. Processor after Claim 1 , where the data set includes a combination of the synthetic data and real data. Processor after Claim 3 , wherein the machine learning model is trained at least in part using a discriminator that classifies the outputs of the machine learning model as corresponding to the synthetic data or the real data. Processor after Claim 1 , wherein the machine-learned model is trained at least in part using pseudo-labels, wherein one or more of the pseudo-labels are generated during training of the machine-learned model using outputs of the machine-learned model that have a confidence above a confidence threshold. Processor after Claim 1 , wherein the one or more spatial priors are sampled independently of a structure of a synthetic ego machine path through the environment. Processor after Claim 1 , wherein one or more simulator parameters are sampled to generate the synthetic data, wherein the one or more simulator parameters include at least a number of objects of the one or more objects, one or more poses of the one or more objects, one or more colors of the one or more objects, colors of one or more environmental features or one or more weather conditions of a virtual environment. Processor after Claim 1 , where the spatial prior corresponds to a target prior generated based at least in part on a distribution of a real data set. Processor after Claim 1 , wherein the processor comprises at least one of the following elements: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing operations for simulations; a system for performing digital twin operations; a system for performing simulations for light transportation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system implemented at least in part using cloud computing resources. System comprising: one or more processing units comprising processing circuitry for: sampling one or more object locations from a probability distribution containing lower probabilities for the one or more object locations when the longitudinal distance from an ego location machine increases; Generate, using a neural network network and based at least in part on the one or more object locations, synthetic data containing one or more objects at the one or more object locations; and updating, based at least in part on the synthetic data and the real data, one or more parameters of a machine learning model using a discriminator that classifies outputs of the machine learning model as corresponding to the synthetic data or the real data. System after Claim 10 , wherein the one or more parameters are updated at least in part based on the use of one or more pseudo-labels determined at least in part based on one or more outputs of the machine learning model that have a confidence above a confidence threshold. System after Claim 10 , where the probability distribution corresponds to a spatial prior that is independent of the structure of an ego machine path. System after Claim 10 , where one or more probabilities of the probability distribution decrease in proportion to the longitudinal distance. System after Claim 10 , wherein a value of a loss function used by the neural network is higher when the discriminator classifies an output of the outputs as corresponding to the synthetic data. System after Claim 10 , wherein the system comprises at least one of the following systems: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing operations for simulations; a system for performing digital twin operations; a system for performing simulations for light transportation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system implemented at least in part using cloud computing resources. A method comprising: computing one or more control operations for an ego machine based at least in part on a machine learning model, wherein one or more parameters of the machine learning model are updated during training based at least in part on synthetic data generated using a simulator wherein one or more locations of one or more objects represented by the synthetic data are sampled at least in part based on a spatial prior. Procedure according to Claim 16 , wherein the spatial prior is generated by sampling the one or more locations proportional to a longitudinal distance of the one or more objects to a synthetic ego machine represented by the synthetic data. Procedure according to Claim 16 , where the machine learning model is at least partially trained using a discriminator. Procedure according to Claim 16 , wherein the machine-learned model is trained at least in part using one or more pseudo-labels, the one or more pseudo-labels being generated during training of the machine-learned model using outputs of the machine-learned model that have a confidence above a confidence threshold. Procedure according to Claim 16 , where the prior corresponds to a target prior generated at least in part based on a distribution of a real data set.

Images