JP2023082647A - Deep learning based operational domain verification using camera-based inputs for autonomous systems and applications - Google Patents

Abstract

To provide systems and methods for deep learning based operational design domain verification using camera-based inputs for autonomous machine applications.SOLUTION: Methods and systems provided in various examples are, by using a machine learning model, to determine operational domain conditions related to an autonomous and/or semi-autonomous machine such as one or more of the following: the amount of camera blindness, the blindness classification, the illumination level, the path surface condition, the visibility distance, the scene type classification, and the distance to a scene. Once one or more of these conditions are determined, an operational level of the machine may be determined, and the machine may be controlled according to the operational level.SELECTED DRAWING: Figure 1A

Description

This application is related to U.S. Nonprovisional Application No. 16/570,187 filed September 13, 2019 and U.S. Nonprovisional Application No. 17/449,306 filed September 29, 2021, which the entirety of each is incorporated herein by reference.

Autonomous and semi-autonomous driving systems (e.g. advanced driver assistance systems (ADAS)) can increase, decrease or maintain a certain speed, blind spot monitoring, automatic emergency braking, lane keeping, Sensors (eg, cameras, LiDAR sensors, RADAR sensors, etc.) may be used to make decisions regarding performance of various tasks such as object detection, obstacle avoidance, lane change, lane assignment, camera calibration, and localization. In order to determine whether, and to what extent, these autonomous and semi-autonomous systems should operate autonomously, it would be beneficial to accurately understand the impact of the surrounding environment on these systems. However, sensors are compromised in their ability to perceive their surroundings for a variety of reasons, such as weather (e.g., rain, fog, snow, hail, smoke, etc.), traffic conditions, blockage (e.g., due to foreign objects, rainwater, etc.), or blurring. This can lead to sensor occlusion and/or loss of viewing distance. Potential sources of sensor obscuration and/or loss of viewing range may include snow, rain, glare, solar flares, mud, water, signal failure, and the like.

Conventional systems utilize feature-level approaches to address sensor visibility occlusion and loss of visibility distance, which detect pieces of physical evidence of sensor visibility occlusion or visibility loss and subsequently These features are then stitched together to determine the presence of events such as sensor obscuration and/or reduced visibility. These conventional methods primarily use analysis of missing sharp edge features (e.g. abrupt changes in gradient, color, brightness), color-based pixel analysis, or other low-level feature analysis in regions of the image. It relies on computer vision techniques, such as binary support vector machine classification, to detect visibility and sensor occlusion problems with visual field occlusion, and/or binary support vector machine classification with visual occlusion presence/absence outputs. However, such feature-based computer vision techniques require separate analysis of each feature, e.g. An analysis of how sensors are coupled to reduced visibility or visibility occlusion conditions is required, whereby the scalability of such approaches is limited by the complexity inherent in a wide variety of conditions and occurrences, and the real world will corrupt the data observed using the sensor in . For example, these conventional approaches are not effective for real-time or near real-time deployments due to the computational cost of running them.

Additionally, conventional systems may rely on classifying sources such as rain, snow, fog, and glare that result in reduced sensor viewing distance, but may not provide an accurate indication of sensor data usefulness. . For example, conventional systems may not utilize the identification of rain in an image to determine whether the corresponding image, or portions thereof, can be used for various autonomous or semi-autonomous tasks. In such instances, conventional systems may render images that clearly represent the environment within 100 meters of the vehicle unusable if it is raining. As such, without relying on the image for one or more tasks within the viewing area, the image may be erroneously rejected, disabling one or more tasks. Conventional systems may also be unable to distinguish between different types of sensor view occlusion, such as whether the image is blurred or obscured. Treating each type of sensor view range loss and sensor view occlusion with equal weights or with overall equal weighting results in less significant or detrimental types of sensor view range loss and sensor view occlusion resulting in more The unavailability of an instance of the data may even lead to decisions that are not entirely accurate (e.g. an image of an environment with a light drizzle may be available for one or more actions). However, images of environments with heavy fog cannot be used, and blurred images can be used for some operations, but obscured images cannot.)

Furthermore, by relying on hard-coded computer vision techniques, conventional systems are unable to learn from historical data or learn over time in deployment, thereby allowing new types or sensors to be developed. It limits your ability to adapt to the appearance of blindness. In addition, conventional systems for detecting ambient lighting levels and track surface conditions may rely on dedicated sensors, such as LUX sensors, and/or sensors next to tires to measure light intensity. It may also rely on a road wetness sensor located in the These dedicated sensors do not have a rich data input available, so the data output of the ambient environment is neither precise nor rich. As a result, these dedicated sensors are able to distinguish between lighting instances (e.g. street light lighting versus indoor parking lot lighting) and different track surface conditions (e.g. road with snow cover). It may not be possible to distinguish between when there is no snow on the road but snow on the machine's sensors).

U.S. Patent Application No. 16/101,232

Embodiments of the present disclosure relate to deep learning based on motion domain verification using camera-based input for autonomous systems and applications. One or more machine learning models, such as deep neural networks (DNNs), are used to represent one or more of the following behavioral design domain states for autonomous and/or semi-autonomous machines: Systems and methods are disclosed for calculating outputs: amount of camera view occlusion, class of view occlusion corresponding to camera view occlusion, lighting level, path surface condition, view distance, scene type classification, and scene type Distance to scene corresponding to type classification. These outputs can then be used to determine the level of autonomy, e.g., the level of activity, of the autonomous and/or semi-autonomous machine, which can then be controlled by the environment using the determined level of autonomy. can be controlled.

The system and method of the present disclosure, in contrast to conventional systems such as those described above, may obtain the states of the behavioral design domain through sensor data, thereby increasing the human perception of these states. It provides accurate and rich information similar to Moreover, because the systems and methods of the present disclosure apply multitasking deep learning, they may require less computational resources than conventional approaches.

The present system and method for validating a deep learning-based motion design domain using camera-based input for autonomous machine applications is described in detail below with reference to the accompanying drawings.

FIG. 10 is an exemplary data flow illustrating an exemplary process for determining lighting levels, track surface conditions, viewing distance, scene classification, distance to scene, and sensor visibility occlusion and its causes, according to some embodiments of the present disclosure; FIG. It is a diagram. 6 illustrates an example process for performing an example process for determining lighting levels, track surface conditions, viewing distance, scene classification, distance to scene, and sensor view occlusion and its causes, according to some embodiments of the present disclosure; 1 is a diagram of a target machine learning model; FIG. FIG. 11 includes an exemplary visualization of sensor data including varying visibility distance levels, in accordance with some embodiments of the present disclosure; FIG. 4 includes an exemplary visualization of sensor data including varying visibility distance levels, in accordance with some embodiments of the present disclosure; FIG. 11 includes an exemplary visualization of sensor data including varying sensor obscuration types, in accordance with some embodiments of the present disclosure; FIG. 4 includes an exemplary visualization of sensor data including varying scene types, in accordance with some embodiments of the present disclosure; FIG. 4 includes an exemplary visualization of sensor data including varying scene types, in accordance with some embodiments of the present disclosure; Methods for determining lighting levels, path surface conditions, viewing distances, scene classification, distance to scenes, and sensor view occlusions and their causes using neural networks, according to some embodiments of the present disclosure It is a flow diagram showing . 1 is an illustration of an exemplary autonomous vehicle, according to some examples of the present disclosure; FIG. 6B is an illustration of camera positions and fields of view of the exemplary autonomous vehicle of FIG. 6A, according to some embodiments of the present disclosure; 6B is a block diagram of an example system architecture of the example autonomous vehicle of FIG. 6A, according to some embodiments of the present disclosure; FIG. 6B is a system diagram of communications between a cloud-based server and the exemplary autonomous vehicle of FIG. 6A, according to some embodiments of the present disclosure; FIG. 1 is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the disclosure; FIG. 1 is a block diagram of an exemplary data center suitable for use in implementing some embodiments of the present disclosure; FIG.

Systems and methods are disclosed for deep learning based on motion domain verification using camera-based input for autonomous systems and applications. The systems and methods described herein can be used for applications in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, medical imaging, autonomous or semi-autonomous machines, and/or lighting levels, track surface conditions, It can be used in other technical spaces where determination of viewing distance, scene classification, distance to scene, and sensor visibility occlusion and its causes can be performed. Although the present disclosure may be described with respect to an exemplary autonomous vehicle 600 (alternatively referred to herein as "vehicle 600" or "egomachine 600", examples of which are described with respect to FIGS. 6A-6D), This is not intended to be limiting. For example, the systems and methods described herein can be used in non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more advanced driver assistance systems (ADAS)), manned and unmanned robots or robotics platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, airships, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, submarines, drones, and/or , may be used by other vehicle types, but is not limited to this.

In the disclosed approach, a processor, including one or more circuits, uses a neural network to generate an image of an active region based, at least in part, on image data generated using one or more image sensors. Data representing the state may be calculated. In a particular embodiment, the state of the active area includes the amount of view occlusion of the camera, the classification of the view occlusion corresponding to the amount of view occlusion of the camera, the lighting level, the condition of the track surface, the viewing distance, the scene type classification, and distance to scene corresponding to the scene type classification.

The amount of camera view occlusion, the view occlusion classification corresponding to the camera view occlusion amount, the lighting level, the path surface condition, the viewing distance, the scene type classification, and the distance to the scene corresponding to the scene type classification. Deep neural network (DNN) processing may be used to determine. Sensor data (eg, representing images) may be received from one or more sensors (eg, cameras) disposed on or associated with the vehicle. The sensor data determines the amount of camera obscuration, lighting levels, path surface conditions, viewing distance, scene type classification, and distance to a scene corresponding to the scene type classification associated with the sensor data. It may be applied to neural networks (eg, DNNs such as convolutional neural networks (CNNs)) that can be trained in such a way. The amount of camera view occlusion, the classification of view occlusion corresponding to the amount of camera view occlusion, and methods and systems for determining viewing distance are hereby incorporated by reference in their entirety, September 2019. the same or similar to the methods and systems described in U.S. Nonprovisional Application No. 16/570,187, filed September 13, and U.S. Nonprovisional Application No. 17/449,306, filed September 29, 2021; It's okay.

An appropriate level of egomachine operation for the current area or environment may be determined based at least in part on this sensor data. In an embodiment, suitable operating levels are level 0 (no automation), level 1 (driver assistance), level 2 (partial automation), level 3 (conditional automation), level 4 (high automation), and level It may correspond to one or more of the automation levels defined by the Society of Automotive Engineers (SAE), such as 5 (full automation).

Examples disclosed include automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, marine systems, Smart area monitoring systems, systems for deep learning operations, systems for performing simulation operations, systems implemented using edge devices, one or more virtual machines (VMs) , implemented at least partially in a data center, implemented at least partially using cloud computing resources, and/or other types of systems. obtain. Although specific examples are provided, these examples may be generalized beyond implementation details.

Referring now to FIG. 1A, FIG. 1A illustrates lighting levels, path surface conditions, viewing distance, scene classification, distance to scene, and sensor visibility occlusion and its causes, according to some embodiments of the present disclosure. 1 is an exemplary dataflow diagram showing an exemplary process 100 for doing. It should be understood that this and other configurations described herein are provided as examples only. Other configurations and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or in place of those shown, and some elements may be completely May be omitted. Moreover, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components and in any suitable combination and location. Various functions described herein as being performed by an entity may be performed by hardware, firmware, and/or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some examples, the systems, methods, and processes described herein may be implemented on the exemplary autonomous vehicle 600 of FIGS. 6A-6D, the exemplary computing device 700 of FIG. 7, and/or the exemplary It may be implemented using components, features, and/or functionality similar to those of data center 800 .

Process 100 may include generating and/or receiving sensor data 102 from one or more sensors of egomachine 600 . The sensor data 102 is processed by the egomachine 600 in the process 100 to determine the amount of camera view occlusion, the classification of the view occlusion corresponding to the amount of camera view occlusion, lighting levels, track surface conditions, viewing distance, scene It may be used to determine in real time or near real time one or more of the type classification and the distance to the scene corresponding to the scene type classification. The sensor data 102 may be, without limitation, sensors from any of the sensors of the egomachine 600 (and/or in some instances other vehicles or objects such as robotics devices, VR systems, AR systems, etc.). • may include data 102; For example, referring to FIGS. 6A-6C, the sensor data 102 may include global navigation satellite system (GNSS) sensors 658 (eg, global positioning system sensors), RADAR sensors 660, ultrasonic sensors, 662, LIDAR sensor 664, inertial measurement unit (IMU) sensor 666 (e.g., accelerometer, gyroscope, magnetic compass, magnetometer, etc.), microphone 676, stereo camera 668, wide view camera 670 ( a fisheye camera), an infrared camera 672, a surround camera 674 (e.g. a 360 degree camera), a long and/or medium range camera 678, a speed sensor 644 (e.g. for measuring the speed of the vehicle 600), and/or or other sensor types, including but not limited to. As another illustration, sensor data 102 may include virtual sensor data generated from any number of sensors of a virtual vehicle or other virtual object. In such instances, the virtual sensors correspond to virtual vehicles or other virtual objects within the simulated environment (e.g., used to test, train, and/or validate neural network performance). and the virtual sensor data can represent sensor data that is simulated or captured by a virtual sensor within a virtual environment. As such, by using virtual sensor data, the machine learning model 104 described herein can be tested, trained, and/or validated using simulation data within a simulated environment. can be done, allowing testing of more extreme scenarios outside of a real-world environment where such testing can be less secure.

The sensor data 102 may be image data representing an image, image data representing a video (e.g., a snapshot of a video), and/or sensor data representing a representation of the sensor's cognitive field (e.g., a LiDAR sensor depth map, ultra graphs of sonic sensor values, etc.). If the sensor data 102 includes image data, for example, compressed images such as, but not limited to, JPEG (Joint Photographic Experts Group) or luminance/chrominance (YUV) formats; 264/AVC (Advanced Video Coding) or H.264/AVC. 265/High Efficiency Video Coding (HEVC) compressed images as frames from compressed video formats such as Red Clear Clear Blue (RCCB), Red Clear Clear (RCCC) or Any type of image data format may be used, such as raw images, such as those from other types of image sensors, and/or other formats. Additionally, in some instances sensor data 102 may be used within process 100 without preprocessing (eg, in raw or captured format), while in other instances sensor data 102 may be the data 102 undergoes preprocessing (e.g., noise balancing, demosaicing, scaling, cropping, dilation, white balancing, tone curve adjustment, etc., e.g., using a sensor data preprocessor (not shown)); Can receive. As used herein, sensor data 102 may refer to raw sensor data, preprocessed sensor data, or a combination thereof.

The sensor data pre-processor uses image data representing one or more images (or other data representations) to form a multi-dimensional array/matrix (or tensor in some instances, or more specifically The sensor data can be loaded into memory in the form of input tensors). The array size may be calculated and/or expressed as W×H×C, where W represents the image width in pixels, H represents the height in pixels, and C is the color channel. represents the number of Other types and orderings of input image components are possible without loss of generality. Additionally, when batching is used, batch size B can be used as a dimension (eg, an additional fourth dimension). Batching may be used for training and/or inference. An input tensor may therefore represent an array of dimensions W×H×C×B. Any ordering of the dimensions may be possible, depending on the particular hardware and software used to implement the sensor data preprocessor. This ordering may be chosen to maximize training and/or inference performance of the machine learning model 104 .

In some embodiments, the sensor data preprocessor processes raw images captured by sensors (e.g., cameras) and included in the image data 102 to provide input images for the input layer of the machine learning model 104. A pre-processing image pipeline may be employed to generate pre-processed image data representing . An example of a suitable pre-processing image pipeline would take a raw RCCB Bayer (e.g. 1-channel) type image from the sensor and transform it into an RCB (e.g. 3-channel) Fixed Precision ) (eg, 16 bits per channel) format into a planar image. The preprocessing image pipeline may include decompanding, noise reduction, demosaicing, white balancing, histogram computation, and/or adaptive global tone mapping (eg, in that order or an alternate order).

If noise reduction is employed by the sensor data preprocessor, this processing may include bi-directional denoising in the Bayer domain. If demosaicing is employed by the sensor data preprocessor, this processing may include bilinear interpolation. If histogram computation is employed by the sensor data preprocessor, this processing may include computing the histogram of the C channel, and in some instances may be combined with decompanding or noise reduction. If adaptive global tone mapping is employed by the sensor data preprocessor, this processing may include performing an adaptive gamma-log transform. This may involve computing a histogram, obtaining a center tone level, and/or using the center tone level to estimate the maximum luminance.

Machine learning model 104 uses as input one or more images or other data representations such as those represented by sensor data 102 (e.g., LIDAR point clouds, RADAR representations, etc.) to generate output 106. can. In a non-limiting example, the machine learning model 104 can, for example, detect the camera's view occlusion 108, view occlusion classification 110, lighting level 112, track surface condition 114, viewing distance, scene type classification 118, and/or distance to scene. An image represented by the sensor data 102 (eg, after preprocessing) may be taken as input to produce an output 106 such as 120 . Although examples are described herein with respect to using a neural network, specifically a DNN, as the machine learning model 104, this is not intended to be limiting. For example, without limitation, the machine learning model 104 described herein can be any type of machine learning model, e.g., linear regression, logistic regression, decision tree, support vector machine (SVM) , Naive Bayes, k-nearest neighbor (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, recurrences, perceptrons, Long/Short Term Memory (LSTM), Hopfield, Boltzmann, Deep Belief, Deconvolution, Adversarial Generation, Liquid State Machines, etc.) and/or other types of machine learning May contain models.

With respect to calculating camera view occlusions 108 and view occlusion classifications 110, the machine learning model 104 may identify regions of interest in the sensor data 102 for sensor view occlusions and their causes (e.g., blurring, occlusion, etc.). More specifically, the machine learning model 104 infers view occlusion markers to identify potential sensor view occlusion locations and causes in the sensor data, as well as whether the sensor data, or portions thereof, are relevant to the system. can be designed to output a classification that identifies whether it can be used by In some instances, the neural network determines that sensor data is at least partially available (e.g., True, yes, 0) or that sensor data is not available (e.g., false). A binary decision (eg, True/False, Yes/No, 0/1) indicating (False, no, 1) may be output. If the data is indicated to be unavailable, the sensor data may be skipped, ignored, and/or the operating level of the autonomous or semi-autonomous application may be reduced to level 0, e.g. It can be used to make a decision to return.

View occlusion classification 110 may use detected view occlusion to express the cause of view occlusion associated with one or more pixels of sensor data 101 . Sensor data 102 may include any number of different visibility occlusion classifications. For example, one view-occluded region or pixel may have one, two, or more associated view-occluded classifications without departing from the scope of this disclosure. In some instances, the machine learning model 104 may be trained to predict multiple classifications and output multiple channels corresponding to these classifications (e.g., each channel corresponding to one visibility occlusion classification). obtain). For example, the visibility occlusion classification 110 may include one or more of occluded, blurred, reflected, open, ego vehicle, sky, frame label, etc., and the number of output channels may correspond to the number of desired classifications. . The visibility occlusion category 110 may include subcategories including, but not limited to, rain, glare, cracked lenses, light, mud, paper, people, and the like.

Illustratively, the visibility occlusion category "obstruction" is one or more of the following: sun, fog, water, mist, snow, frozen window glass, day, night, cracked lens, self-glare, mud, paper, foliage, etc. can be associated with the visibility occlusion subclass of In this way, the visibility occlusion sub-category may further delineate the cause of visibility occlusion in sensor data 102 . In some instances, the visibility occlusion subcategory may include further subcategories of visibility occlusion causes, such as those that indicate the level of visibility degradation (eg, severe, moderate, mild, etc.). The machine learning model 104 may be trained to predict view occlusion classes and/or corresponding view occlusion subclasses and may have output channel numbers comparable to these numbers.

The machine learning model 104 can be trained to predict the viewing distance 116 of the sensor data 102, eg, the maximum distance from the sensor at which an object or element can be identified. Based on the predicted view distance, the operating level of the machine may be adjusted. For example, if the machine learning model 104 determines that the viewing distance is short—eg, 20 meters or less—the corresponding sensor data can only be relied on for level 0 (no automation) or level 1 (driver assistance) tasks. or may be relied upon only for predictions within 20 meters of the machine (eg, predictions beyond 20 meters may be ignored or associated with lower confidence). Similarly, as another example, if the estimated viewing distance is large, e.g. It may be relied upon for overall performance, or for predictions corresponding to locations within 1000 meters of the machine.

In an example, the calculated visibility distances are very short visibility (eg, 10 meters to 100 meters), short visibility (eg, 10 meters to 100 meters), medium visibility (eg, 100 meters to 1000 meters). , long visibility (eg 1000 meters to 4000 meters), or clear visibility (eg over 4000 meters). In other examples, very short visibility distances may be associated with extreme fog or snowstorms, or heavy rain or snow, short visibility distances may be associated with heavy fog, and medium visibility distances may be associated with moderate visibility. It may be associated with fog, rain, or drizzle, long visibility distances may be associated with hazy weather conditions, or light rain or drizzle, and clear visibility distances may be associated with specific weather conditions that generate sensor data. It may be related to the maximum viewing distance of the sensor.

The predicted viewing distance may correspond to various tasks that the egomachine may perform. For example, for very short viewing distances, as long as the egomachine is traveling below a threshold speed (e.g. below 25 km/h), level 0, level 1 and level 2 low speed active safety features (e.g. lane assist, automatic lane keeping, automatic cruise control, automatic emergency braking, etc.) may be available. However, if the vehicle is traveling above a threshold speed, these functionalities may be disabled with very short fault distances, at least for instances of sensor data. For short viewing distances, full performance of Level 0, Level 1, and Level 2 parking functions (e.g., proximity detection, automatic parallel parking orientation, etc.) and low speed active safety functions are maintained as long as the egomachine is traveling below the threshold speed. May be available. For moderate visibility distances, full performance of Level 0, Level 1, Level 2, and Level 2+ driving functions may be available. For long viewing distances, in addition to level 0, level 1, level 2 and level 2+ functions of the first, second and third bins, level 3 and level 4 parking functions (e.g. automatic parallel parking (APP : automatic parallel parking), MPP, VVP) may be available. For clear visibility, the full performance of Level 3 highway autonomous driving functions may be available in addition to Level 0, Level 1, Level 2 and Level 2+ functions.

Machine learning model 104 may also be trained to predict scene type classification 118 for sensor data 102 and distance to scene 120 corresponding to the scene type classification. For example, the machine learning model 104 assumes that the sensor data represents one or more of distances to tunnels, construction zones, closed lanes, or toll booths, and scene types such as affected road areas. It can be trained to predict. The machine learning model 104 may predict from the sensor data 102 the road surface conditions 114, such as dry, wet, snowy, icy, etc., as well as lighting levels 112, e.g., the amount of light perceived by road users. Can be trained further.

In embodiments where machine learning model 104 includes a DNN, machine learning model 104 may include any number of layers. One or more of the layers may include input layers. The input layer may hold values associated with various sensor data 402 . One or more layers may include convolutional layers. A convolutional layer can compute the output of neurons connected to local regions in the input layer, each neuron computing the dot product of their weights with the small region they are connected to in the input volume. The result of the convolutional layer may be another volume, with one of its dimensions based on the number of filters applied (e.g., width, height, and number of filters, e.g., 12 being the number of filters). , then 32x32x12).

One or more layers may include deconvolutional layers (or transposed convolutional layers). For example, the result of the deconvolution layer can be another volume with a higher dimensionality than the input dimensionality of the data the deconvolution layer received.

One or more of the layers may include a rectified linear unit (ReLU) layer. The ReLU layer may, for example, apply an element-wise activation function that thresholds at zero, eg, max(0,x). The ReLU layer's resulting volume may be the same as the ReLU layer's input volume.

One or more of the layers may include pooling layers. A pooling layer may result in a smaller volume than the pooling layer's input (eg, 16x16x12 from a 32x32x12 input volume) and may perform a downsampling operation along a spatial dimension (eg, height and width).

One or more of the layers may include one or more fully connected layers. Each neuron in a fully connected layer can be connected to each neuron in the previous volume. A fully connected layer can compute class scores and the resulting volume can be 1×1×n, where n is the number of classes. In some instances, a CNN may include fully connected layers such that the outputs of one or more of the layers of the CNN may be provided as inputs to the fully connected layers of the CNN. can contain. In some instances, one or more convolutional streams may be implemented by machine learning model 104, and some or all of the convolutional streams may include respective fully connected layers.

In some non-limiting examples, the machine learning model 104 includes convolutional and max-pooling layers to facilitate image feature extraction, and multi-scale extended convolutional layers to facilitate global context feature extraction. It may include a series of layers, followed by layers and upsampling layers.

Although input layers, convolutional layers, pooling layers, ReLU layers, and fully connected layers are discussed herein with respect to the machine learning model 104, this is not intended to be limiting. Additional or alternative layers may be used in machine learning model 104, such as, for example, normalization layers, SoftMax layers, and/or other layer types.

Additionally, some of the layers, such as convolutional layers and fully connected layers, may contain parameters (eg, weights and/or biases), while other layers, such as ReLU layers and pooling layers, may not. In some instances, machine learning model 104 may learn parameters during training. In addition, some of the layers, such as convolutional layers, fully connected layers, and pooling layers, may contain additional hyperparameters (e.g., learning rate, stride, epoch, etc.), while others, such as the ReLU layer, May not be included. In embodiments where the machine learning model 104 backtracks on viewing distance, the last layer activation function of the CNN may include a ReLU activation function. In embodiments where the machine learning model 104 classifies instances of sensor data into range bins, the last layer activation functions of the CNN may include SoftMax activation functions. Parameters and hyper-parameters may vary from implementation to implementation without limitation.

In embodiments where machine learning model 104 includes a CNN, the order and number of CNN layers may vary depending on the embodiment. In other words, the order and number of CNN layers are not limited to any one architecture.

In an embodiment, the EgoMachine's level of action, e.g., action 124, is determined based on a predefined set of camera view occlusion, view occlusion classification, lighting level, path surface condition, viewing distance, scene type classification, and distance to scene. can include weighting by the weight of In other embodiments, the camera view occlusion may be associated with the highest of the predefined weights.

Referring now to FIG. 1B in conjunction with FIG. 1A, FIG. 1B illustrates lighting levels, track surface conditions, viewing distance, scene classification, distance to scene, and/or sensor visibility occlusion, according to some embodiments of the present disclosure. and/or an example machine learning model 104A for implementing an example process for determining its cause. For example, in one or more embodiments, a CNN that machine learning model 104A may include may include one or more trunks, an encoder-decoder architecture, and/or one or more output heads (eg, a view occlusion mask • May include a head 142, a lighting head 144, a track surface head 146, a view occlusion cause head 148, and a scene classification head 150). In an example, a CNN may include one or more convolutional layers corresponding to a feature detection trunk, which may output intermediate data (e.g., representing feature maps), which may include one or more It can be processed using the output head. For example, using the intermediate data, a first output head, eg, view occlusion mask head 142, can be used to calculate the amount of view occlusion of the camera, and a second output, eg, view occlusion cause head 148 A third output head, for example lighting head 144, may be used to calculate lighting levels, and a fourth output head, for example track surface head 146. A head may be used to compute track surface conditions and/or a fifth output head, eg, scene head 150, may be used for scene type classification (eg, scene type probability 152) and/or scene • Can be used to calculate the distance to scene (eg, distance to scene 154) corresponding to the type classification. In another embodiment, a fifth output head can be used to compute the scene type classification and a sixth output head can be used to compute the distance to the scene. If more than one head is used, the two or more heads may process data from the trunk in parallel, each head trained to accurately predict the corresponding output of its output head. can be However, in other embodiments a single trunk may be used without separate heads.

In an illustrative example, one or more of the first, second, third, fourth, fifth, and/or sixth heads of the CNN of machine learning model 104 may include at least one fully connected layer. . For example, each of lighting head 144, track surface head 146, view occlusion cause head 148, and scene head 150 may include at least one fully connected layer. Each neuron in the fully connected layer can be connected to each neuron in the previous volume. A fully connected layer can compute class scores and the resulting volume can be 1×1×n, where n is the number of classes. In some non-limiting examples, one or more of the heads of the CNN of machine learning model 104 may include one or more downsampling layers as well as one or more upsampling layers. For example, the CNN view occlusion mask head 142 of the machine learning model 104 includes a downsampling layer (e.g., a pooling layer) to facilitate image feature extraction and an upsampling layer to facilitate global context feature extraction. may include a series of layers followed by . In an embodiment, one or more pooling layers may perform a downsampling operation along a spatial dimension (e.g., height and width) to extract a volume smaller than the pooling layer's input (e.g., a 32x32x12 input 16x16x12 volume from the volume). In an illustrative example, each of illumination head 144, track surface head 146, view occlusion cause head 148, and scene head 150 may include a pooling layer. In a further example, the scene head 150 may use a sigmoid function to output a probability 152 of scene type, such as tunnel, construction zone, closed lane, or toll booth. In other instances, the scene head 150 may also use the ReLU layer to output the distance 154 to the scene.

Referring back to FIG. 1A, in some embodiments, output 106 may be post-processed by post-processor 122 after being computed by machine learning model 104 . Post-processor 122 may determine drivability corresponding to sensor data 102 . For example, post-processor 122 may perform temporal smoothing of output 106 of machine learning model 104 . Temporal smoothing, in some embodiments, incorporates prior predictions of the machine learning model 104 corresponding to temporally adjacent frames to smooth and reduce noise in the output of the machine learning model 104. can be used to improve system stability by reducing false positives on a single frame basis. In some instances, the value calculated by machine learning model 104 for the current instance of sensor data 102 is the value calculated by machine learning model 104 for one or more previous instances of sensor data 102. can be compared against If the sensor data 102 is image data representing an image, for example, the output 106 computed by the machine learning model 104 for the current or most recent image may be one or more temporally adjacent images ( may be compared with the output 106 computed by the machine learning model 104 for the previous image and/or subsequent image). As such, closing values corresponding to instances of sensor data 102 (e.g., for camera view occlusion 108, view occlusion classification 110, lighting level 112, track surface condition 114, viewing distance 116, scene type classification 118, and scene distance to 120 ) may be determined by weighting previous values associated with one or more other instances of sensor data 102 against current values associated with the instance of sensor data 102 . The results of temporal smoothing can be analyzed with respect to sensor data 102 to determine drivability.

Action 124 may be a decision made in real time or near real time by the system using sensor data 102 based on output 106 . For example, some or all of the sensor data 102 may be skipped or ignored with respect to one or more of the control decisions 124 if they are not sufficiently clear or valid. In some instances, such as when sensor data 102 cannot be used for safe operation of vehicle 600, action 124 returns control to the driver (eg, terminates autonomous or semi-autonomous operation). or to perform an emergency or safety maneuver (eg, stop, pull over, or a combination thereof). As such, action 124 may include suggesting one or more corrective actions, such as ignoring certain instances of sensor data, for effective and safe driving.

For autonomous or semi-autonomous driving in any instance, the operations 124 include the driving stack 126 (e.g., the sensor data manager layer of the autonomous driving software stack), the perception layer of the driving stack 126, the world model management layer of the driving stack 126, the driving Any decisions corresponding to the planning layer of the driving stack 126 , the control layer of the driving stack 126 , the obstacle avoidance layer of the driving stack 126 , and/or the actuation layer of the driving stack 126 may be included. Thus, as described herein, the drivability of sensor data 102 may be determined separately for any number of different operations corresponding to one or more tiers of operational stack 126 .

As an example, a first drivability may be determined for object detection operations with respect to the perception layer of the driving stack 126 and a second drivability may be determined for path planning with respect to the planning layer of the driving stack 126 . Information representing sensor data 102 may be sent to operating stack 126 along with an indication that one or more features, functionality, and/or components of the operating stack are disabled, enabled, or unchanged. The driving stack 126 includes one or more such as a world situation manager layer that manages the world situation using one or more maps (eg, 3D maps), localization components, perception components, and/or the like. can include layers of Additionally, the autonomous driving software stack includes a planning component (eg, as part of the planning layer), a control component (eg, as part of the control layer), and an actuation component (eg, as part of the actuation layer). , obstacle avoidance components (eg, as part of an obstacle avoidance layer), and/or other components (eg, as part of one or more additional or alternative layers). As such, operation 124 may direct any downstream tasks in operational stack 126 that rely on sensor data 102 such that proper use of sensor data 102 is managed.

Referring back to FIG. 1A, in an embodiment, machine learning model 104 may be trained using both information provided by sensor data 102 and information provided by training engine 128 . In an illustrative example, training engine 128 may include ground truth data 130 . In some embodiments, ground truth data 130 may include labels or annotations corresponding to sensor data 102 . For example, for each instance of sensor data 102A, the label or annotation may include the amount of camera view occlusion, the view occlusion classification corresponding to the camera view occlusion, lighting level, path surface condition, viewing distance, scene type classification , and the distance to the scene corresponding to the scene type classification. Labels and annotations may be generated within a drawing program (e.g., an annotation program), a computer aided design (CAD) program, a labeling program, another type of program suitable for generating annotations, and/or or may be hand-drawn in some instances. In any instance, ground truth data 130 may be synthetically created (eg, generated from a computer model or rendering), actually created (eg, designed and created from real-world data). machine-automated (e.g., using feature analysis and learning to extract features from data and then generating labels); annotated by humans (e.g., labelers or annotation experts position), and/or a combination thereof (eg, human identification of visible objects and computer determination of distances to objects and corresponding viewing distances).

In some embodiments, the determination of ground truth labels for distances to scenes may be performed manually. In an example, frames with the vehicle in the scene may be selected, such as frames in which the vehicle is in a tunnel. Frames that repeat after this may be marked with a distance of 0 until the same scene is no longer seen. Frames that repeat before this scene may be marked with a distance of 0 until the same scene entrance is found or the same scene is no longer seen. IMU and/or GPS data can then be used to calculate the distance between the previous frame and the current frame. That distance can be added to the distance of the previous frame to set the distance of the current frame so that the total length to this scene can be determined.

In an embodiment, training engine 128 converts output 106 to ground truth data 130 (e.g., amount of camera view occlusion, view occlusion classification corresponding to camera view occlusion, lighting level, path surface condition, view distance , the scene type classification, and the distance to the scene corresponding to the scene type classification). In an embodiment, machine learning model 104 may be trained using sensor data 102 using multiple iterations until the value of loss function 132 is below a predetermined threshold. Any type of loss function may be used, such as cross entropy loss, mean squared error, mean absolute error, mean biased error, and/or other loss function types. In some instances, the slope of loss function 132 may be computed iteratively with respect to parameter training. An optimizer such as Adam optimizer, stochastic gradient descent, or other types of optimization algorithms may be used to optimize the loss function 132 during training of the machine learning model 104 . Machine learning model 104 may be trained iteratively until the training parameters converge to optimal, target, or acceptable values.

2A-2B, FIGS. 2A-2B are exemplary visualizations of sensor data representations of various viewing distances, according to some embodiments of the present disclosure. With respect to FIG. 2A, assuming that visualization 200A (including heavy rain) corresponds to an instance of sensor data 102A (eg, an image from a data collection vehicle's camera), then for vehicle 202A, depth values (eg, LiDAR, RADAR), output from machine learning models or neural networks, output from computer vision algorithms, and/or other output types. In such an illustration, the data collection vehicle had instantiated sensor data 102A that included vehicle 202A, thus forming the basis for determining depth or distance information to objects in the environment. process may have been running. As such, if vehicle 202A is the farthest visible object from the data collection vehicle and the depth to vehicle 202A is known, the visible distance for an instance of sensor data 102A may correspond to the depth or distance to vehicle 202A. (In an embodiment, for example, additional distance is added if the viewing distance of the environment exceeds vehicle 202A, and the distance is somewhat less if vehicle 202A appears weak or blurred). Similarly, for visualization 200B (which includes light rain conditions), vehicle 202B is visible, and thus the visibility distance can be determined using the depth or distance value corresponding to vehicle 202B. Although rain is shown and described with respect to FIGS. 2A-2B, this is not intended to be limiting and the viewing distance in other situations such as fog, snow, sleet, hail, lighting, occlusions, combinations thereof, etc. It can be determined by the examples of the present disclosure.

Referring now to FIG. 3, FIG. 3 includes exemplary visualizations of sensor data with varying levels of sensor visibility occlusion, according to some embodiments of the present disclosure. For example, machine learning model 104 may use image data representing input image 310A as input and output image mask 310B that includes view-occluded regions 312 and 314 of the image. In one or more examples, the top of image 310A may be obscured or obscured by glare or reflections from rainwater, foreign objects, or sunlight (as depicted in FIG. 3). Different classifications can be represented by different RGB colors or indicators. View-occluded region 312 may be classified as an occluded or blurred region, and view-occluded region 314 may also be classified as an occluded or blurred region. Pixels in view-obstructed regions 312 and 314 may similarly be classified as occluded or blurred pixels. Visualization 310C shows image 310A overlaid with image mask 310B. In this way, sensor visibility occlusion can be accurately determined and viewed in an area-specific manner. For example, with respect to image 310A, the drivability determination can be sophisticated such that image 310A is valid for autonomous or semi-autonomous driving behavior. This is because image 310A is only partly blurred, and this part is the sky of the environment and should not affect operation 124 .

4A-4B, FIGS. 4A-4B include illustrative example visualizations of sensor data corresponding to various scene types, according to some embodiments of the present disclosure. As shown in FIG. 4A, machine learning model 104 uses image data such as sensor data 102 and/or data provided by training engine 128 to output data representing a scene such as a tunnel. can be done. As shown in FIG. 4B, machine learning model 104 uses image data such as sensor data 102 and/or data provided by training engine 128 to output data representing a scene, such as a toll booth. be able to.

Referring now to FIG. 5, each block of method 500 described herein includes computational processes that may be performed using any combination of hardware, firmware, and/or software. For example, various functions may be performed by a processor executing instructions stored in memory. Method 500 may also be implemented as computer usable instructions stored on a computer storage medium. Method 500 may be provided by a standalone application, service or hosted service (either standalone or in combination with another hosted service), or plugged into another product, to name a few examples. Additionally, the method 500 is illustratively described with respect to the egomachine 600 of FIGS. 6A-6D. However, the method 500 may be used within any one process and/or any one system or combination of processes and systems, including but not limited to those described herein. may additionally or alternatively be performed by

FIG. 5 illustrates using a neural network to determine lighting levels, path surface conditions, viewing distance, scene classification, distance to scene, and sensor visibility occlusions and their causes, according to some embodiments of the present disclosure. 5 is a flow diagram illustrating a method 500 for doing.

At block B502, the method 500 uses a neural network to generate one or more motion regions corresponding to the environment based on at least image data of the environment generated using the one or more sensors. including calculating data representing the parameters of In an example, the one or more parameters are a camera view occlusion level, a view occlusion classification corresponding to the camera view occlusion, a lighting level, a path surface condition, a viewing distance, a scene type classification, or a scene type. including at least one of the distances to the scene corresponding to the classification; In a further example, the neural network uses the sensor data 102 and/or the ground truth data 130 and/or the loss function 132 to classify camera view occlusions, corresponding view occlusion classifications for camera view occlusions, illumination Data representing level, track surface condition, viewing distance, scene type classification, and distance to the scene corresponding to the scene type classification may be calculated.

The method 500 includes determining, based at least in part on the data, a level of operation of the egomachine corresponding to the environment at block B504. For example, the egomachine's activity level corresponds to the camera's view occlusion, the view occlusion classification corresponding to the camera's view occlusion, the lighting level, the path surface condition, the viewing distance, the scene type classification, and/or the scene type classification. may be determined based on one or more of the distances to the scene to be viewed.

The method 500 includes controlling operation of the egomachine according to the operation level at block B506. For example, if the scene type classification is determined to be a toll booth and/or the distance to the scene is determined to be 100 yards, the egomachine may slow down according to embodiments of the present disclosure. In other instances, if the viewing distance is determined to be short, the egomachine slows down or performs other safety maneuvers (such as stopping, moving to a safe or designated location, etc.). good too.

(exemplary autonomous vehicle)
FIG. 6A is a diagram of an exemplary autonomous vehicle 600, according to some embodiments of the disclosure. Autonomous vehicle 600 (alternatively referred to herein as "vehicle 600") is a passenger vehicle such as a passenger car, truck, bus, first responder vehicle, shuttle, electric or motorized bicycle, motorcycle, fire engine. , police vehicles, ambulances, boats, construction vehicles, submarines, drones, trailer-coupled vehicles, and/or other types of vehicles (e.g., unmanned and/or carrying one or more passengers). obtain, but are not limited to: Autonomous vehicles are generally administered by the National Highway Traffic Safety Administration (NHTSA), departments of the US Department of Transportation, and the Society of Automotive Engineers (SAE) under the Taxonomy and Definition ons for terms related to Driving Automation Systems for On-Road Motor Vehicle" (Standard No. J3016-201806 published on June 15, 2018, Standard No. J3016-201609 published on September 30, 2016, and previous versions of this standard. and future versions). Vehicle 600 may be capable of functioning according to one or more of levels 3-5 of autonomous driving levels. Vehicle 600 may be capable of functionality according to one or more of levels 1-5 of autonomous driving levels. For example, vehicle 600 may be driver assisted (level 1), partially automated (level 2), conditionally automated (level 3), highly automated (level 4), and/or fully automated (level 4), depending on the embodiment. May have level 5) abilities. As used herein, the term “autonomy” refers to any and/or all types of autonomy of vehicle 600 or other machines, e.g., fully autonomous, highly autonomous, conditional autonomy, partial autonomy, providing auxiliary autonomy, semi-autonomy, primarily autonomy, or other designations.

Vehicle 600 may include components such as a vehicle chassis, body, wheels (eg, 2, 4, 6, 8, 18, etc.), tires, axles, and other components. Vehicle 600 may include a propulsion system 650, such as an internal combustion engine, a hybrid power plant, an all-electric engine, and/or another propulsion system type. Propulsion system 650 may be connected to the drive train of vehicle 600 , which may include a transmission, to enable propulsion of vehicle 600 . Propulsion system 650 may be controlled in response to receiving signals from throttle/accelerator 652 .

Steering system 654, which may include a steering wheel, steers vehicle 600 (e.g., along a desired course or route) when propulsion system 650 is operating (e.g., when the vehicle is in motion). can be used for Steering system 654 may receive signals from steering actuator 656 . The handle may be an option for full automation (level 5) functionality.

Brake sensor system 646 may be used to operate the vehicle brakes in response to receiving signals from brake actuator 648 and/or brake sensors.

Controller 636, which may include one or more system on chip (SoC) 604 (FIG. 6C) and/or GPUs, signals one or more components and/or systems of vehicle 600. (eg, a representation of a command) can be provided. For example, the controller operates the vehicle brakes via one or more brake actuators 648, the steering system 654 via one or more steering actuators 656, and the one or more Signals may be sent to operate propulsion system 650 via throttle/accelerator 652 . Controller 636 processes sensor signals and outputs operational commands (eg, signals representing commands) to enable disciplined driving and/or to assist the driver in driving vehicle 600. It may include one or more onboard (eg, integrated) computing devices (eg, supercomputers). Controllers 636 include a first controller 636 for autonomous driving functions, a second controller 636 for functional safety functions, a third controller 636 for artificial intelligence functions (e.g., computer vision), an information It may include a fourth controller 636 for tainment functions, a fifth controller 636 for redundancy in emergency situations, and/or other controllers. In some instances, a single controller 636 may handle two or more of the functions described above, and two or more controllers 636 may perform a single function and/or any combination thereof. can be processed.

Controller 636 provides signals to control one or more components and/or systems of vehicle 600 in response to sensor data (e.g., sensor input) received from one or more sensors. can be done. Sensor data may include, by way of illustration and not limitation, global navigation satellite system sensors 658 (e.g., global positioning system sensors), RADAR sensors 660, ultrasonic sensors 662, LIDAR sensors 664, inertial measurement units (IMU : inertial measurement unit) sensor 666 (e.g., accelerometer, gyroscope, magnetic compass, magnetometer, etc.), microphone 696, stereo camera 668, wide view camera 670 (e.g., fisheye camera), infrared camera 672, Surround camera 674 (eg, 360 degree camera), long and/or medium range camera 698, speed sensor 644 (eg, for measuring speed of vehicle 600), vibration sensor 642, steering sensor 640, brake • may be received from sensors (eg, as part of brake sensor system 646), and/or other sensor types;

One or more of controllers 636 receive inputs (eg, represented by input data) from instrument cluster 632 of vehicle 600 and output (eg, represented by output data, display data, etc.). ) may be provided via a human-machine interface (HMI) display 634 , audible annunciators, loudspeakers, and/or other components of vehicle 600 . Outputs include vehicle speed, speed, time, map data (eg, HD map 622 in FIG. 6C), position data (eg, location of vehicle 600, such as on a map), direction, location of other vehicles (eg, occupancy grid), information about the object and the status of the object as ascertained by the controller 636 . For example, the HMI display 634 may indicate the presence of one or more objects (eg, road signs, warning signs, changes in traffic lights, etc.) and/or driving maneuvers that the vehicle has made, is making, or will make. (eg, that you are now changing lanes, exiting exit 34B within 2 miles, etc.) can be displayed.

Vehicle 600 further includes a network interface 624 that can communicate over one or more networks using one or more wireless antennas 626 and/or a modem. For example, network interface 624 may be capable of communication via LTE, WCDMA, UMTS, GSM, CDMA2000, and the like. The wireless antenna 626 also supports local area networks such as Bluetooth®, Bluetooth LE, Z-Wave, ZigBee and/or low power wide area networks (LPWAN) such as LoRaWAN, SigFox. Wide-area networks) can be used to enable communication between objects in the environment (eg, vehicles, mobile devices, etc.).

FIG. 6B is an illustration of camera positions and fields of view of exemplary autonomous vehicle 600 of FIG. 6A, according to some embodiments of the present disclosure. The cameras and respective fields of view are one illustrative example and are not intended to be limiting. For example, additional and/or alternate cameras may be included and/or cameras may be placed at different locations on vehicle 600 .

Camera types of cameras may include, but are not limited to, digital cameras that may be adapted for use with vehicle 600 components and/or systems. The camera may operate at automotive safety integrity level (ASIL) B and/or at another ASIL. A camera type may be capable of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the implementation. A camera may have the ability to use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some instances, the color filter array is a red clear clear clear (RCCC) color filter array, a red clear clear blue (RCCB) color filter array, a red blue green clear (RBGC) color filter array. array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and/or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras with RCCC, RCCB, and/or RBGC color filter arrays, may be used in efforts to increase light sensitivity.

In some instances, one or more of the cameras may be used to perform advanced driver assistance system (ADAS) functions (eg, as part of a redundant or failsafe design). For example, a multi-function mono camera can be installed to provide functions including lane departure warning, traffic sign assist and intelligent headlamp control. One or more of the cameras (eg, all cameras) can simultaneously record and provide image data (eg, video).

One or more of the cameras are used to filter out stray light and reflections from within the vehicle (e.g. reflections from the dashboard reflected in the windshield mirrors) that can interfere with the camera's ability to capture image data. , in a mounting component such as a custom designed (3D printed) part. With reference to the side mirror mounting component, the side mirror component can be custom 3D printed such that the camera mounting plate matches the shape of the side mirror. In some instances, the camera may be integrated within the side mirror. For side-view cameras, the cameras can also be integrated into four stanchions at each corner of the cabin.

A camera with a field of view that includes a portion of the environment in front of vehicle 600 (e.g., a forward-facing camera) helps identify forward-facing paths and obstacles with the aid of one or more controllers 636 and/or control SoCs. can be used for surround view to help provide information essential to generating an occupancy grid and/or determining a preferred vehicle course. A forward-facing camera can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. Forward-facing cameras are also used for ADAS functions and systems, including other functions such as Lane Departure Warning (LDW), Autonomous Cruise Control (ACC), and/or traffic sign recognition. can be

Various cameras may be used in front-facing configurations, including, for example, a monocular camera platform that includes a complementary metal oxide semiconductor (CMOS) color imager. Another example may be a wide view camera 670 that may be used to capture objects within view from the surroundings (eg, pedestrians, crossing traffic, or bicyclists). Although only one wide-view camera is shown in FIG. 6B, any number of wide-view cameras 670 may be present on vehicle 600 . Additionally, a long-range camera 698 (eg, a long-view stereo camera pair) can be used for depth-based object detection, especially for objects for which the neural network has not yet been trained. Long range camera 698 can also be used for object detection and classification, and basic object tracking.

One or more stereo cameras 668 may also be included in the front-facing configuration. The stereo camera 668 is integrated with an expandable processing unit that can provide programmable logic (FPGA) and multi-core microprocessors with integrated CAN or Ethernet interfaces on a single chip. may include a controlled control unit. Such a unit can be used to generate a 3D map of the vehicle's environment, including range estimates for every point in the image. An alternative stereo camera 668 uses two camera lenses (one left and one right) and measures the distance from the vehicle to the target object and the generated information (e.g., metadata). and an image processing chip that can activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 668 may be used in addition to or instead of those described herein.

A camera with a field of view that includes portions of the environment to the sides of the vehicle 600 (e.g., a side-view camera) provides information used to create and update occupancy grids and to generate side impact collision alerts. can be used for surround view. For example, surround cameras 674 (eg, four surround cameras 674 as shown in FIG. 6B) may be positioned on vehicle 600 . Surround cameras 674 may include wide view cameras 670, fisheye cameras, 360 degree cameras, and/or the like. For example, four fisheye cameras can be placed in the front, back and sides of the vehicle. In an alternative arrangement, the vehicle may use three surround cameras 674 (eg, left, right, and rear) and one or more other cameras (eg, , front-facing cameras).

A camera with a field of view that includes a portion of the environment behind the vehicle 600 (eg, a rear-view camera) may be used for parking assistance, surround view, rear collision warning, and occupancy grid creation and updating. As described herein, a variety of cameras including, but not limited to, cameras also suitable as forward facing cameras (e.g., long and/or medium range camera 698, stereo camera 668, infrared camera 672, etc.) A wide variety of cameras can be used.

FIG. 6C is a block diagram of an exemplary system architecture for exemplary autonomous vehicle 600 of FIG. 6A, according to some embodiments of the present disclosure. It should be understood that this arrangement and other arrangements described herein are merely illustrative. Other arrangements and elements (eg, machines, interfaces, functions, ordering, groupings of functions, etc.) may be used in addition to or in place of those shown, and some elements may be excluded together. may Moreover, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components and in any suitable combination and location. Various functions described herein as being performed by an entity may be performed by hardware, firmware, and/or software. For example, various functions may be performed by a processor executing instructions stored in memory.

Each of the components, features, and systems of vehicle 600 of FIG. 6C are illustrated as being connected via bus 602 . Bus 602 may include a controller area network (CAN) data interface (alternatively referred to herein as a "CAN bus"). CAN may be a network within vehicle 600 that is used to help control various features and functions of vehicle 600, such as braking, acceleration, braking, steering, actuation of windshield wipers, and the like. A CAN bus may be configured to have dozens or hundreds of nodes, each with its own unique identifier (eg, CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, engine revolutions per minute (RPM), button positions, and/or other vehicle status indicators. The CAN bus may be ASIL B compliant.

Although bus 602 is described herein as being a CAN bus, this is not intended to be limiting. For example, FlexRay and/or Ethernet may be used in addition to or as an alternative to the CAN bus. Additionally, although a single line is used to represent bus 602, this is not intended to be limiting. For example, one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and/or one or more other types of buses using different protocols. There may be any number of buses 602 that may be included. In some instances, two or more buses 602 may be used to perform different functions and/or may be used for redundancy. For example, a first bus 602 may be used for collision avoidance functions and a second bus 602 may be used for motion control. In any example, each bus 602 may communicate with any component of vehicle 600, and two or more buses 602 may communicate with the same component. In some instances, each SoC 604, each controller 636, and/or each computer in the vehicle may have access to the same input data (e.g., inputs from sensors in vehicle 600) and a common interface such as a CAN bus. It can be connected to a bus.

Vehicle 600 may include one or more controllers 636, such as those described herein with respect to FIG. 6A. Controller 636 may be used for various functions. Controller 636 may be coupled to any of the various other components and systems of vehicle 600 to control vehicle 600, artificial intelligence in vehicle 600, infotainment for vehicle 600, and/or the like. can be used for

Vehicle 600 may include a system-on-chip (SoC) 604 . SoC 604 may include CPU 606, GPU 608, processor 610, cache 612, accelerator 614, data store 616, and/or other components and features not shown. SoC 604 may be used to control vehicle 600 in various platforms and systems. For example, SoC 604 is a system with HD map 622 that can obtain map refreshes and/or updates from one or more servers (eg, server 678 in FIG. system of vehicle 700).

CPU 606 may include a CPU cluster or CPU complex (alternatively referred to herein as a "CCPLEX"). CPU 606 may include multiple cores and/or L2 caches. For example, in some embodiments CPU 606 may include eight cores in a coherent multiprocessor configuration. In some embodiments, CPU 606 may include four dual-core clusters, each cluster having a dedicated L2 cache (eg, 2MBL2 cache). CPUs 606 (eg, CCPLEX) may be configured to support simultaneous cluster operation, allowing any combination of clusters of CPUs 606 to be active at any given time.

CPU 606 may implement power management capabilities that include one or more of the following features: Individual hardware blocks automatically restart clocks when idle to conserve dynamic power. Each core clock may be gated when the core is not actively executing instructions due to execution of WFI/WFE instructions Each core may be power gated independently Core clusters can be independently clock gated when all cores are clock gated or power gated, and/or each core cluster can be clock gated when all cores are power gated. • Can be independently power gated when gated. The CPU 606 may further implement enhanced algorithms for managing power states, in which allowed power states and expected wake-up times are specified, and hardware/microcode is , cluster, and CCPLEX. The processing core can support a simplified power state entry sequence in software with the work offloaded to microcode.

GPU 608 may include an integrated GPU (alternatively referred to herein as an "iGPU"). GPU 608 can be programmable and efficient for concurrent workloads. In some instances, GPU 608 may use an enhanced tensor instruction set. GPU 608 may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (eg, an L1 cache with at least 96 KB of storage), Two or more may share an L2 cache (eg, an L2 cache with 512 KB storage capacity). In some embodiments, GPU 608 may include at least eight streaming microprocessors. GPU 608 may use computational application programming interfaces (APIs). Additionally, GPU 608 may use one or more parallel computing platforms and/or programming models (eg, NVIDIA's CUDA).

GPU 608 may be power optimized for best performance in automotive and embedded use cases. For example, GPU 608 may be fabricated on FinFETs (Fin field-effect transistors). However, this is not intended to be limiting and GPU 608 may be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor may incorporate several mixed-precision processing cores partitioned into multiple blocks. For example and without limitation, 64 PF32 cores and 32 PF64 cores may be partitioned into four processing blocks. In such an example, each processing block consists of 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, 2 mixed precision NVIDIA TENSOR COREs for deep learning matrix operations, L0 instruction cache, warp scheduler, dispatch unit , and/or a 64KB register file. In addition, streaming microprocessors may include independent parallel integer and floating point data paths to provide efficient execution of workloads with a mixture of computational and addressing operations. Streaming microprocessors may include independent thread scheduling capabilities to enable finer-grained synchronization and coordination among concurrent threads. A streaming microprocessor may include a combined L1 data cache and shared memory unit to simplify programming and improve performance.

GPU 608 may include a high bandwidth memory (HBM) and/or 16 GB HBM2 memory subsystem to provide for peak memory bandwidth of 900 GB/sec in some instances. In some instances, graphics double data rate type five synchronous random-access memory (GDDR5) in addition to or instead of HBM memory A synchronous graphics random-access memory (SGRAM), such as a synchronous graphics random-access memory (SGRAM), may be used.

GPU 608 may include unified memory techniques, including access counters, to allow more accurate movement of memory pages to the processors that access them most frequently, thereby To improve the efficiency of storage ranges shared between processors. In some instances, address translation service (ATS) support may be used to allow GPU 608 to directly access CPU 606 page tables. In such instances, address translation requests may be sent to CPU 606 when GPU 608 memory management unit (MMU) experiences a miss. In response, CPU 606 can consult its page table for a virtual-to-real mapping of addresses and send translations back to GPU 608 . As such, unified memory technology can enable a single unified virtual address space for both CPU 606 and GPU 608 memory, thereby simplifying GPU 608 programming and porting of applications to GPU 608. do.

Additionally, GPU 608 may include access counters that may record the frequency of GPU 608 accesses to other processors' memories. Access counters can help ensure that memory pages are moved to the physical memory of the processors that are accessing them most frequently.

SoC 604 may include any number of caches 612, including those described herein. For example, cache 612 may include an L3 cache available to both CPU 606 and GPU 608 (eg, connected to both CPU 606 and GPU 608). Cache 612 may include a write-back cache that can record the state of the line, such as by using a cache coherence protocol (eg, MEI, MESI, MSI, etc.). The L3 cache may contain 4 MB or more, depending on the implementation, although smaller cache sizes may be used.

SoC 604 may include an arithmetic logic unit (ALU) that may be utilized in performing processing for any of the various tasks or operations of vehicle 600 (eg, process DNN). Additionally, SoC 604 may include a floating point unit (FPU) (or other math or math coprocessor type) for performing math operations within the system. For example, SoC 104 may include one or more FPUs integrated as execution units within CPU 606 and/or GPU 608 .

SoC 604 may include one or more accelerators 614 (eg, hardware accelerators, software accelerators, or combinations thereof). For example, SoC 604 may include a hardware acceleration cluster, which may include optimized hardware accelerators and/or large on-chip memory. A large on-chip memory (eg, 4MB of SRAM) can enable a hardware acceleration cluster to accelerate neural networks and other operations. The hardware acceleration cluster is used to complement the GPU 608 and to offload some of the tasks of the GPU 608 (e.g., to free up more cycles of the GPU 608 to perform other tasks). obtain. As an example, accelerator 614 may be used for target workloads that are sufficiently stable to be suitable for acceleration (eg, perception, convolutional neural networks (CNNs), etc.). As used herein, the term "CNN" includes region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (e.g., as used for object detection). , can include all types of CNNs.

Accelerator 614 (eg, hardware acceleration cluster) may include a deep learning accelerator (DLA). The DLA may include one or more Tensor Processing Units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured and optimized to perform image processing functions (eg, CNN, RCNN, etc.). The DLA can be further optimized for a particular set of neural network types and floating point operations and inference. DLA designs can deliver more performance per millimeter than general-purpose GPUs, greatly exceeding the performance of CPUs. The TPU can perform several functions, including, for example, single-instance convolution functions, supporting INT8, INT16, and FP16 data types for both features and weights, as well as post-processor functions.

DLAs can quickly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including but not limited to: cameras - CNN for object identification and detection using data from sensors, CNN for range estimation using data from camera sensors, emergency vehicle detection and identification and detection using data from microphones CNN, CNN for facial recognition and vehicle owner identification using data from camera sensors, and/or CNN for security and/or safety related events.

The DLA can perform any function of the GPU 608, and by using an inference accelerator, for example, a designer can target either the DLA or the GPU 608 for any function. can. For example, a designer may focus on processing CNN and floating point arithmetic on the DLA and offload other functions to GPU 608 and/or other accelerators 614 .

Accelerator 614 (eg, hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. PVA is designed to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and/or augmented reality (AR) and/or virtual reality (VR) applications. can be designed and constructed to PVA can provide a balance between performance and flexibility. For example, each PVA may include, for example, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and/or any number of vector processors. may include, but are not limited to.

The RISC core can interact with an image sensor (eg, the image sensor of any of the cameras described herein), an image signal processor, and/or the like. Each RISC core can contain any amount of memory. The RISC core can use any of several protocols, depending on the implementation. In some instances, the RISC core may run a real-time operating system (RTOS). A RISC core may be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and/or memory devices. For example, a RISC core may include an instruction cache and/or tightly coupled RAM.

DMA may allow PVA components to access system memory independent of CPU 606 . DMA can support any number of features used to provide optimizations to PVA, including but not limited to supporting multi-dimensional addressing and/or circular addressing. In some instances, the DMA supports addressing up to 6 dimensions or more, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and/or depth stepping. be able to.

A vector processor may be a programmable processor that can be designed to efficiently and flexibly perform programming of computer vision algorithms and provide signal processing capabilities. In some instances, a PVA may include a PVA core and two vector processing subsystem partitions. A PVA core may include a processor subsystem, a DMA engine (eg, two DMA engines), and/or other peripherals. The vector processing subsystem may operate as the PVA's primary processing engine and may include a vector processing unit (VPU), an instruction cache, and/or a vector memory (eg, VMEM). The VPU core may include a digital signal processor, such as, for example, a single instruction, multiple data (SIMD), very long instruction word (VLIW) digital signal processor. Combining SIMD and VLIW can increase throughput and speed.

Each vector processor may include an instruction cache and may be coupled to dedicated memory. As a result, in some instances each vector processor may be configured to execute independently of other vector processors. In other examples, vector processors included in a particular PVA can be configured to use data parallelism. For example, in some embodiments, multiple vector processors contained in a single PVA can execute the same computer vision algorithm, but on different regions of the image. In other examples, vector processors included in a particular PVA can run different computer vision algorithms simultaneously on the same image, or run different algorithms on successive images or portions of images. can even In particular, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. Additionally, the PVA may include additional error correcting code (ECC) memory to enhance overall system security.

Accelerator 614 (eg, hardware acceleration cluster) may include a computer vision network-on-chip and SRAM to provide high-bandwidth, low-latency SRAM for accelerator 614 . In some instances, the on-chip memory may include, by way of illustration and not limitation, at least 4MB of SRAM consisting of eight field configurable memory blocks that may be accessible by both PVA and DLA. . Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. PVAs and DLAs can access memory through a backbone that provides PVAs and DLAs with fast access to memory. The backbone may include a computer vision network-on-chip interconnecting PVAs and DLAs to memory (eg, using APBs).

The computer vision network-on-chip may include an interface that determines, prior to sending any control signals/addresses/data, that both the PVA and DLA provide ready and valid signals. Such an interface can provide separate phases and separate channels for transmitting control signals/addresses/data, as well as burst-type communication for continuous data transfer. This type of interface may follow the ISO26262 or IEC61508 standards, although other standards and protocols may be used.

In some instances, SoC 604 may include a real-time ray tracing hardware accelerator, such as described in US Pat. Real-time ray tracing hardware accelerator for RADAR signal interpretation, for sound propagation synthesis and/or analysis, for simulation of SONAR systems, for general wave propagation simulation, for localization and/or other Rapidly determine the position and size of objects (e.g., in world models) to generate real-time visualization simulations for comparison against LIDAR data and/or for other uses aimed at the function of can be used to make decisions efficiently. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing related operations.

Accelerators 614 (eg, hardware accelerator clusters) have a variety of applications for autonomous driving. PVA may be a programmable vision accelerator that can be used for critical processing steps in ADAS and autonomous vehicles. PVA's capabilities are well suited to areas of algorithms that require predictable processing at low power and low latency. In other words, PVA performs well in semi-dense or dense regular computations even on small data sets that require predictable execution times with low latency and low power. Therefore, in the context of platforms for autonomous vehicles, PVA is designed to run classic computer vision algorithms, as it is efficient in object detection and working with integer arithmetic. be.

For example, according to one embodiment of the present technology, PVA is used to perform computer stereo vision. A semi-global matching-based algorithm may be used in some instances, but this is not intended to be limiting. Many applications for level 3-5 autonomous driving require motion estimation/stereo matching on-the-fly (eg, structure from motion (SFM), pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions with input from two monocular cameras.

In some instances, PVA may be used to perform high density optical flow. By processing the raw RADAR data (eg, using a 4D Fast Fourier Transform) to provide processed RADAR. In another example, PVA is used for time-of-flight depth processing, for example, by processing raw time-of-flight data to provide processed time-of-flight data.

The DLA can be used to implement any type of network to enhance control and driving safety, including, for example, neural networks that output a measure of confidence in each object detection. Such confidence values can be interpreted as probabilities or as providing a relative "weight" for each detection compared to other detections. This confidence value allows the system to make further decisions regarding which detections should be considered true positive detections rather than false positive detections. For example, the system can set a confidence threshold and consider only detections above the threshold as true positive detections. In an automatic emergency braking (AEB) system, a false positive detection would cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. The DLA may run a neural network that regresses confidence values. The neural network may include bounding box dimensions, ground plane estimates obtained (e.g., from another subsystem), neural networks and/or other sensors (e.g., LIDAR sensor 664 or RADAR sensor 660). At least some subset of parameters may be received as its inputs, such as vehicle 600 heading, range, inertial measurement unit (IMU) sensor 666 output that correlates with 3D position estimates of objects, and others.

SoC 604 may include data store 616 (eg, memory). Data store 616 can be an on-chip memory of SoC 604 and can store neural networks to be run on the GPU and/or DLA. In some instances, data store 616 may have sufficient capacity to store multiple instances of neural networks for redundancy and security. Data store 612 may comprise L2 or L3 cache 612 . References to data store 616 may include references to memory associated with PVA, DLA, and/or other accelerators 614 as described herein.

SoC 604 may include one or more processors 610 (eg, embedded processors). Processor 610 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management capabilities and associated security enforcement. A boot and power management processor can be part of the SoC 604 boot sequence and can provide runtime power management services. A boot power and management processor may provide clock and voltage programming, support for system low power state transitions, management of SoC 604 thermal and temperature sensors, and/or management of SoC 604 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and SoC 604 may use the ring oscillator to detect the temperature of CPU 606, GPU 608, and/or accelerator 614. If the temperature is determined to exceed the threshold, the boot and power management processor enters a temperature fault routine to place the SoC 604 in a lower power state and/or put the vehicle 600 into surrogate driving mode for a safe stop. For example, the vehicle 600 can be brought to a safe stop).

Processor 610 may further include a set of embedded processors that may act as an audio processing engine. The audio processing engine may be an audio subsystem that allows full hardware support for multi-channel audio over multiple interfaces and a wide and flexible range of audio I/O interfaces. In some instances, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

Processor 610 may further include an always-on processor engine that can provide the necessary hardware features to support low power sensor management and wake use cases. An always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (eg, timers and interrupt controllers), various I/O controller peripherals, and routing logic.

Processor 610 may further include a safety cluster engine that includes a dedicated processor subsystem to handle safety management for automotive applications. A security cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (eg, timers, interrupt controllers, etc.), and/or routing logic. In safety mode, two or more cores can operate in lockstep mode and function as a single core with comparison logic to detect any difference between their operations.

Processor 610 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

Processor 610 may further include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

Processor 610 may be a processing block (e.g., microprocessor-implemented) that implements the video post-processing functions required by the video playback application to produce the final image for the player window. can include The video image compositor may perform lens distortion correction at the wide view camera 670, at the surround camera 674, and/or at the in-cabin surveillance camera sensor. The in-cabin surveillance camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC configured to identify in-cabin events and respond appropriately. The in-cabin system activates cellular service and makes a call, dictates an email, changes the vehicle's destination, activates or changes the vehicle's infotainment system and settings, or Lip reading can be performed to provide voice-activated web surfing. Certain features are only available to the driver when operating in autonomous mode and are disabled otherwise.

A video image synthesizer may include enhanced temporal noise reduction for both spatial noise reduction and temporal noise reduction. For example, if motion occurs in the video, noise reduction de-weights the information provided by adjacent frames and weights the spatial information appropriately. If an image or part of an image does not contain motion, temporal noise reduction performed by a video image synthesizer can use information from previous images to reduce noise in the current image.

The video image synthesizer may also be configured to perform stereo rectification on the input stereo lens frame. The video image compositor can also be used for user interface compositing when the operating system desktop is in use, and GPU 608 is not required to continuously render new surfaces. Even when GPU 608 is powered on and actively doing 3D rendering, the video image compositor can be used to offload GPU 608 to improve performance and responsiveness.

SoC 604 may include a mobile industry processor interface (MIPI) camera serial interface for receiving video and input from the camera, a high speed interface, and/or for the camera and associated pixel input functions. may further include video input blocks that may be used for SoC 604 may further include an input/output controller, which may be controlled by software and used to receive I/O signals not committed to a particular role.

SoC 604 may further include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management, and/or other devices. SoC 604 can connect data from cameras (e.g., via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LIDAR sensors 664 that can be connected via Ethernet). , RADAR sensors 660, etc.), data from bus 602 (e.g., vehicle 600 speed, steering wheel position, etc.), data from GNSS sensors 658 (e.g., connected via Ethernet or CAN bus). can be used for processing. SoC 604 may include its own DMA engine and may further include a dedicated high performance mass storage controller that may be used to offload CPU 606 from routine data management tasks.

SoC 604 may be an end-to-end platform with a flexible architecture that spans automation levels 3-5, thereby leveraging and efficiently using computer vision and ADAS techniques for diversity and redundancy, and deep learning Together with the tools, it provides a holistic functional safety architecture that provides a platform for a flexible and reliable driving software stack. SoC 604 can be faster, more reliable, more energy efficient, and more space efficient than conventional systems. For example, when accelerator 614 is combined with CPU 606, GPU 608 and data store 616 can provide a fast and efficient platform for Level 3-5 autonomous vehicles.

Thus, the technology provides capabilities and functionality unattainable by conventional systems. For example, computer vision algorithms can be executed on a CPU that can be configured using a high level programming language such as the C programming language to perform a wide variety of processing algorithms over a wide variety of visual data. . However, CPUs often fail to meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. Specifically, multiple CPUs are unable to run complex object detection algorithms in real-time, which is a requirement for in-vehicle ADAS applications and practical Level 3-5 autonomous vehicles.

By providing CPU complexes, GPU complexes, and hardware acceleration clusters, in contrast to conventional systems, the techniques described herein enable multiple neural networks to run simultaneously and/or in series. be executed and the results combined to enable level 3-5 autonomous driving functions. For example, a CNN running on a DLA or dGPU (e.g., GPU 620) enables a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained, text and word can include recognition. The DLA further includes a neural network capable of identifying, interpreting, and providing semantic understanding of the signs and passing the semantic understanding to a course planning module running on the CPU complex. can contain.

As another example, multiple neural networks can be run simultaneously, as required for level 3, 4, or 5 operation. For example, a warning sign consisting of "CAUTION: Flashing light indicates a frozen condition" along with a lightning can be independently or collectively interpreted by several neural networks. The sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a neural network that has been trained), and the text "Flashing light indicates icy conditions" indicates that the flashing light indicates an icy condition. It can be interpreted by a second deployed neural network that, when detected, informs the vehicle's route planning software (preferably running on the CPU complex) that an icy condition exists. Flashing lights can be identified by informing the vehicle's route planning software of the presence (or absence) of the flashing lights and running a third deployed neural network over multiple frames. All three neural networks can run concurrently, such as in the DLA and/or on GPU 608 .

In some instances, CNNs for facial recognition and vehicle owner identification may use data from camera sensors to identify the presence of a legitimate driver and/or owner of vehicle 600 . An always-on sensor processing engine is used to unlock and turn on the vehicle when the owner approaches the driver's door, and in security mode to operate the vehicle when the owner leaves the vehicle. can be used to stop In this manner, SoC 604 provides security against theft and/or hijacking.

In another example, a CNN for emergency vehicle detection and identification may use data from microphone 696 to detect and identify emergency vehicle sirens. In contrast to conventional systems that use general classifiers to detect sirens and extract features manually, SoC604 uses CNNs for classification of environmental and urban sounds, as well as classification of visual data. use. In one preferred embodiment, a CNN running on the DLA is trained (eg, by using the Doppler effect) to identify relative terminal velocities of emergency vehicles. The CNN may also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by GNSS sensors 658 . Thus, for example, when operating in Europe, CNN would attempt to detect European sirens, and when in the US, CNN would attempt to identify only North American sirens. . After the emergency vehicle is detected, the control program, with the assistance of the ultrasonic sensor 662, slows down the vehicle, stops it on the edge of the road, parks the vehicle, and/or until the emergency vehicle has passed. It can be used to run emergency vehicle safety routines, causing the vehicle to idle.

The vehicle may include a CPU 618 (eg, discrete CPU, or dCPU) that may be coupled to SoC 604 via a high-speed interconnect (eg, PCIe). CPU 618 may include, for example, an X86 processor. CPU 618 performs various functions including, for example, reconciling potentially inconsistent results between ADAS sensors and SoC 604 and/or monitoring the status and health of controller 636 and/or infotainment SoC 630 . It can be used to perform any of the functions.

Vehicle 600 may include a GPU 620 (eg, a discrete GPU, or dGPU) that may be coupled to SoC 604 via a high-speed interconnect (eg, NVIDIA's NVLINK). GPU 620 may provide additional artificial intelligence functionality, such as by running redundant and/or different neural networks, which may be configured based on input from sensors of vehicle 600 (eg, sensor data). can be used to train and/or update the

Vehicle 600 further includes a network interface 624 that may include one or more wireless antennas 626 (eg, one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). obtain. The network interface 624 may be connected to the cloud via the Internet (e.g., server 678 and/or other network devices), other vehicles, and/or computing devices (e.g., passenger client devices). can be used to enable wireless connectivity of A direct link may be established between two vehicles and/or an indirect link may be established (eg, over a network and over the Internet) to communicate with other vehicles. A direct link may be provided using a vehicle-to-vehicle communication link. A vehicle-to-vehicle communication link may provide vehicle 600 information regarding vehicles in proximity to vehicle 600 (eg, vehicles in front of, beside, and/or behind vehicle 600). This function may be part of the cooperative adaptive cruise control function of vehicle 600 .

Network interface 624 may include a SoC that provides modulation and demodulation functionality and allows controller 636 to communicate over a wireless network. Network interface 624 may include a radio frequency front end for baseband to radio frequency upconversion and radio frequency to baseband downconversion. Frequency conversion can be performed through well-known processes and/or can be performed using a superheterodyne process. In some instances, radio frequency front end functionality may be provided by a separate chip. Network interface via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and/or other wireless protocols may include wireless capabilities for communicating with

Vehicle 600 may further include data store 628, which may include off-chip (eg, off-SoC 604) storage. Data store 628 may be one or more storage devices including RAM, SRAM, DRAM, VRAM, flash, hard disk, and/or other components and/or devices capable of storing at least one bit of data. element.

Vehicle 600 may further include a GNSS sensor 658 . GNSS sensors 658 (eg, GPS, assisted GPS sensors, differential GPS (DGPS) sensors, etc.) assist with mapping, perception, occupancy grid generation, and/or route planning functions. For example, any number of GNSS sensors 658 may be used including, but not limited to, GPS using a USB connector with an Ethernet to serial (RS-232) bridge.

Vehicle 600 may further include a RADAR sensor 660 . RADAR sensor 660 may be used by vehicle 600 for long range vehicle detection, even in darkness and/or severe weather conditions. The RADAR functional safety level may be ASIL B. In some instances, the RADAR sensor 660 uses Ethernet access to access raw data, for control, and to access object tracking data (e.g., RADAR sensor 660) and/or bus 602 may be used to transmit data generated by 660). A wide variety of RADAR sensor types may be used. For example and without limitation, RADAR sensor 660 may be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

RADAR sensor 660 may include different configurations, such as long range with narrow field of view, short range with wide field of view, and short range side coverage. In some instances, long range RADAR may be used for adaptive cruise control functions. A long-range RADAR system can provide a wide field of view, such as within a range of 250m, achieved by two or more independent scans. The RADAR sensor 660 can help distinguish between static and moving objects and can be used by ADAS systems for emergency brake assist and forward collision warning. Long range RADAR sensors may include monostatic multimodal RADARs with multiple (eg, six or more) fixed RADAR antennas and high speed CAN and FlexRay interfaces. In one example with six antennas, the center four antennas provide a focused beam pattern designed to record around the vehicle 600 at high speeds with minimal interference from traffic in adjacent lanes. can create. The other two antennas can increase the field of view and allow rapid detection of vehicles entering or leaving the lane of vehicle 600 .

As an example, a medium-range RADAR system may include a range of up to 660m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 650 degrees (rear). A short-range RADAR system may include, but is not limited to, RADAR sensors designed to be installed at each end of the rear bumper. When installed at either end of the rear bumper, such a RADAR sensor system can create two beams that constantly monitor the blind spots behind and next to the vehicle.

Short-range RADAR systems can be used in ADAS systems for blind spot detection and/or lane change assistance.

Vehicle 600 may further include an ultrasonic sensor 662 . Ultrasonic sensors 662, which may be positioned at the front, rear, and/or sides of vehicle 600, may be used for parking assist and/or for creating and updating occupancy grids. A wide variety of ultrasonic sensors 662 may be used, and different ultrasonic sensors 662 may be used for different ranges of detection (eg, 2.5m, 4m). The ultrasonic sensor 662 is capable of operating at the ASIL B functional safety level.

Vehicle 600 may include a LIDAR sensor 664 . LIDAR sensors 664 may be used for object and pedestrian detection, emergency braking, collision avoidance, and/or other functions. The LIDAR sensor 664 may be at functional safety level ASIL B. In some instances, vehicle 600 may have multiple (e.g., 2, 4 , six, etc.) LIDAR sensors 664 .

In some instances, LIDAR sensor 664 may be capable of providing a list of objects and their distances in a 360 degree field of view. Commercially available LIDAR sensors 664 may, for example, have an accuracy of 2 cm to 3 cm and an advertised range of approximately 600 m with support for 600 Mbps Ethernet connections. In some instances, one or more non-protruding LIDAR sensors 664 may be used. In such instances, LIDAR sensors 664 may be implemented as small devices that may be incorporated into the front, rear, sides, and/or corners of vehicle 600 . In such an example, the LIDAR sensor 664 has a range of 200m even for low reflective objects and can provide up to 120 degrees horizontal and 35 degrees vertical field of view. The front mounted LIDAR sensor 664 may be configured for a horizontal field of view between 45 and 135 degrees.

In some instances, LIDAR technology such as 3D flash LIDAR may also be used. 3D flash LIDAR uses a laser flash as a transmitter to illuminate the vehicle's surroundings up to about 200m. A flash LIDAR unit contains a receptor that records the laser pulse transit time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LIDAR may allow highly accurate and distortion-free images of the surroundings to be produced with any laser flash. In some instances, four flash LIDAR sensors may be deployed, one on each side of vehicle 600 . Available 3D flash LIDAR systems include solid-state 3D steering array LIDAR cameras (eg, non-scanning LIDAR devices) that have no moving parts other than the blower. Flash LIDAR devices can use 5 ns Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of 3D range point clouds and co-listed intensity data. . By using flash LIDAR, and because flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 664 can be less susceptible to motion blur, vibration, and/or shock. .

The vehicle may further include IMU sensors 666 . In some instances, IMU sensor 666 may be positioned in the middle of the rear axle of vehicle 600 . IMU sensors 666 may include, for example, without limitation, accelerometers, magnetometers, gyroscopes, magnetic compasses, and/or other sensor types. In some instances, such as in 6-axis applications, IMU sensors 666 may include accelerometers and gyroscopes, while in 9-axis applications IMU sensors 666 may include accelerometers, gyroscopes, and magnetometers.

In some embodiments, the IMU sensor 666 combines a micro-electro-mechanical system (MEMS) inertial sensor, a sensitive GPS receiver, and an advanced Kalman filtering algorithm to detect position, velocity, and It can be implemented as a compact high-performance GPS-Aided Inertial Navigation System (GPS/INS) that provides attitude estimates. As such, in some instances, the IMU sensor 666 can determine heading without requiring input from a magnetic sensor by directly observing and correlating changes in velocity from the GPS to the IMU sensor 666. Vehicle 600 may be allowed to estimate. In some instances, IMU sensor 666 and GNSS sensor 658 may be combined in a single integrated unit.

A vehicle may include a microphone 696 located in and/or around the vehicle 600 . Microphone 696 may be used, among other things, for emergency vehicle detection and identification.

The vehicle may have any number of cameras, including stereo cameras 668, wide view cameras 670, infrared cameras 672, surround cameras 674, long and/or medium range cameras 698, and/or other camera types. • It may further include a type. A camera may be used to capture image data around the vehicle 600 . The type of camera used depends on the implementation and requirements of vehicle 600 , and any combination of camera types can be used to achieve the required coverage around vehicle 600 . Additionally, the number of cameras may vary depending on the implementation. For example, a vehicle may include 6 cameras, 7 cameras, 10 cameras, 12 cameras, and/or another number of cameras. The camera may support Gigabit Multimedia Serial Link (GMSL) and/or Gigabit Ethernet by way of example, but not limitation. Each camera is described in further detail herein in connection with FIGS. 6A and 6B.

Vehicle 600 may further include vibration sensor 642 . Vibration sensors 642 may measure vibrations of vehicle components, such as axles. For example, changes in vibration may indicate changes in the surface of the road. In another example, when two or more vibration sensors 642 are used, the difference in vibration can be used to determine friction or slippage of the road surface (e.g., the difference in vibration can be used to determine whether a power driven axle and a freewheeling between axles).

Vehicle 600 may include ADAS system 638 . In some instances, ADAS system 638 may include SoC. The ADAS system 638 includes autonomous/adaptive/automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW). warning), automatic Emergency braking (AEB: automatic emergency braking), lane departure warning (LDW: lane departure warning), lane keep assist (LKA: lane keep assist), blind spot warning (BSW: blind spot warning), rear cross traffic warning ( may include rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and/or other features and functions.

The ACC system may use RADAR sensors 660, LIDAR sensors 664, and/or cameras. An ACC system may include vertical ACC and/or horizontal ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately ahead of vehicle 600 and automatically adjusts vehicle speed to maintain a safe distance from the vehicle ahead. Lateral ACC enforces distance keeping and advises vehicle 600 to change lanes when necessary. Lateral ACC is relevant to other ADAS applications such as LCA and CWS.

CACC may be received from other vehicles via wireless links via network interface 624 and/or wireless antenna 626, or indirectly via network connections (eg, via the Internet). Use information from other vehicles where possible. A direct link may be provided by a vehicle-to-vehicle (V2V) communication link, while an indirect link may be an infrastructure-to-vehicle (I2V) communication link. In general, the V2V communication concept provides information about the immediately preceding vehicle (e.g., the vehicle directly in front of vehicle 600 that is in the same lane as vehicle 600), while the I2V communication concept provides information about further traffic ahead. offer. A CACC system may include either or both I2V and V2V information sources. Given the information of vehicles in front of vehicle 600, CACC can be more reliable, and CACC has the potential to make traffic flow smoother and reduce road congestion.

FCW systems are designed to warn drivers of hazards so that they can take corrective action. FCW systems are forward-facing cameras and/or RADARs coupled to dedicated processors, DSPs, FPGAs, and/or ASICs that are electrically coupled to driver feedback, such as displays, speakers, and/or vibration components. Sensor 660 is used. FCW systems can provide warnings in the form of audible, visual warnings, vibrations and/or quick brake pulses, and the like.

An AEB system detects an impending frontal collision with another vehicle or other object and automatically applies the brakes if the driver does not take corrective action within specified time or distance parameters. can be done. The AEB system may use a forward facing camera and/or RADAR sensor 660 coupled to a dedicated processor, DSP, FPGA and/or ASIC. When the AEB system detects a danger, the AEB system normally first warns the driver to take corrective action to avoid a collision; if the driver does not take corrective action, the AEB system will , the brakes can be automatically applied as part of an effort to prevent, or at least mitigate, the effects of an anticipated crash. AEB systems may include techniques such as dynamic brake support and/or crash emergency braking.

The LDW system provides visual, audible, and/or tactile alarms, such as steering wheel or seat vibrations, to warn the driver when the vehicle 600 crosses a lane marking. The LDW system will not activate when the driver indicates an intentional lane departure by activating the turn signals. LDW systems use front-facing cameras coupled to dedicated processors, DSPs, FPGAs, and/or ASICs that are electrically coupled to driver feedback, such as displays, speakers, and/or vibration components. can do.

The LKA system is a modification of the LDW system. The LKA system provides steering input or braking to correct the vehicle 600 if the vehicle 600 begins to drift out of its lane.

The BSW system detects and warns the vehicle operator in the vehicle's blind spots. The BSW system may provide visual, audible, and/or tactile warnings to indicate that merging or lane changes are unsafe. The system can provide additional warnings when the driver uses the turn signals. BSW systems are rear-facing cameras coupled to dedicated processors, DSPs, FPGAs, and/or ASICs electrically coupled to driver feedback, e.g., displays, speakers, and/or vibrating components. and/or a RADAR sensor 660 can be used.

The RCTW system can provide visual, audible, and/or tactile notification when an object is detected out of range of the rear camera when the vehicle 600 is backing up. Some RCTW systems include an AEB to ensure that vehicle braking is applied to avoid a collision. The RCTW system includes one or more rear panel sensors coupled to a dedicated processor, DSP, FPGA, and/or ASIC electrically coupled to driver feedback, e.g., displays, speakers, and/or vibrating components. A RADAR sensor 660 pointing toward the can be used.

Because conventional ADAS systems alert the driver and allow the driver to determine whether a safe condition really exists and act accordingly, conventional ADAS systems are typically not catastrophic. No, but tended to produce false positive results that could be irritating and distracting to the driver. However, in an autonomous vehicle 600, the vehicle 600 itself hears the results from the primary or secondary computer (eg, the first controller 636 or the second controller 636) when the results conflict. have to decide whether For example, in some embodiments ADAS system 638 may be a backup and/or secondary computer for providing perceptual information to a backup computer rationale module. Backup computer rationality monitors can run redundant and diverse software on hardware components to detect faults in perceptual and dynamic driving tasks. Output from ADAS system 638 may be provided to the supervisory MCU. If the outputs from the primary and secondary computers conflict, the supervisory MCU must decide how to reconcile the conflict to ensure safe operation.

In some instances, the primary computer may be configured to provide the supervisory MCU with a confidence score that indicates the primary computer's confidence in the selected outcome. If the confidence score exceeds the threshold, the supervisory MCU may follow the instructions of the primary computer regardless of whether the secondary computer gives contradictory or inconsistent results. If the confidence score does not meet the threshold, and if the primary and secondary computers give different results (e.g. conflicting), the supervisory MCU will mediate between the computers to determine the appropriate result. can be done.

The supervisory MCU executes a neural network trained and configured to determine a condition in which the secondary computer provides a false alarm based on outputs from the primary and secondary computers. can be configured. Thus, a neural network within the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot be trusted. For example, when the secondary computer is a RADAR-based FCW system, a neural network within the supervisory MCU may detect metal objects that are not actually dangerous, such as sewer grate or manhole covers, that trigger alarms. It can learn when the FCW is discerning. Similarly, when the secondary computer is a camera-based LDW system, the neural network within the supervisory MCU will predict that there are cyclists or pedestrians present and that lane departure is, in fact, the safest maneuver. We can learn to ignore the LDW when . In embodiments involving a neural network running on a supervisory MCU, the supervisory MCU may include at least one of a DLA or GPU suitable for running the neural network with associated memory. In preferred embodiments, a supervisory MCU may comprise and/or be included as a component of SoC 604 .

In other examples, ADAS system 638 may include a secondary computer that performs ADAS functions using conventional rules of computer vision. As such, the secondary computer can use classical computer vision rules (if-then), and the presence of neural networks within the supervisory MCU improves reliability, safety and performance. can be improved. For example, diverse implementations and intentional non-identity make the overall system more fault-tolerant, especially against failures caused by software (or software-hardware interface) functions. For example, if a software bug or error exists in the software running on the primary computer and non-identical software code running on the secondary computer provides the same overall result, the supervisory MCU will You can have greater confidence that the target results are correct and that bugs in the software or hardware on the primary computer have not caused serious errors.

In some instances, the output of the ADAS system 638 may be fed to the primary computer perception block and/or the primary computer dynamic driving task block. For example, if the ADAS system 638 indicated a forward collision warning due to an object immediately in front, the sensory block can use this information when identifying the object. In other instances, the secondary computer may have its own neural network trained as described herein, thus reducing the risk of false positives.

Vehicle 600 may further include an infotainment SoC 630 (eg, an in-vehicle infotainment system (IVI)). Although shown and described as a SoC, infotainment systems need not be SoCs and may include two or more discrete components. The infotainment SoC 630 provides audio (e.g., music, personal digital assistants, navigation instructions, news, wireless, etc.), video (e.g., TV, movies, streaming, etc.), telephony (e.g., hands-free calling), network connectivity (e.g., , LTE, Wi-Fi, etc.) and/or information services (e.g., navigation system, rear parking assistance, wireless data system, fuel level, total distance traveled, brake fuel level, oil level, door open/close, It may include a combination of hardware and software that may be used to provide vehicle 600 with vehicle related information (such as air filter information). For example, the infotainment SoC 630 includes radios, disc players, navigation systems, video players, USB and Bluetooth connectivity, car computers, in-car entertainment, Wi-Fi, steering wheel audio controls, hands-free voice control, head heads-up display (HUD), HMI display 634, telematics device, control panel (e.g., to control and/or interact with various components, features, and/or systems) for ), and/or other components. The infotainment SoC 630 may receive information from the ADAS system 638, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and/or other information. , may further be used to provide information (eg, visual and/or audible) to the user of the vehicle.

Infotainment SoC 630 may include GPU functionality. Infotainment SoC 630 may communicate with other devices, systems, and/or components of vehicle 600 via bus 602 (eg, CAN bus, Ethernet, etc.). In some instances, the infotainment system's GPU performs some self-driving functions if the primary controller 636 (e.g., primary and/or backup computer of vehicle 600) fails. As can be done, the infotainment SoC 630 may be coupled to the supervisory MCU. In such instances, the infotainment SoC 630 may put the vehicle 600 into a substitute driving mode for a safe stop, as described herein.

Vehicle 600 may further include instrument cluster 632 (eg, digital dash, electronic instrument cluster, digital instrument panel, etc.). Instrument cluster 632 may include a controller and/or supercomputer (eg, a discrete controller or supercomputer). Instrument cluster 632 includes speedometer, fuel level, oil pressure, tachometer, odometer, direction indicators, gear shift position indicator, seat belt warning light, parking brake warning light, engine fault light, airbag (SRS) system information, lighting. It may include a set of instrumentation such as controllers, safety system controllers, navigation information. In some instances, information may be displayed and/or shared between infotainment SoC 630 and instrument cluster 632 . In other words, instrument cluster 632 may be included as part of infotainment SoC 630 and vice versa.

FIG. 6D is a system diagram of communications between a cloud-based server and exemplary autonomous vehicle 600 of FIG. 6A, according to some embodiments of the present disclosure. System 676 may include server 678 , network 690 , and vehicles, including vehicle 600 . Server 678 includes a plurality of GPUs 684(A)-684(H) (collectively referred to herein as GPUs 684), PCIe switches 682(A)-682(H) (collectively referred to herein as PCIe switches 682). ), and/or CPUs 680(A)-680(B) (collectively referred to herein as CPUs 680). GPU 684, CPU 680, and PCIe switch may be interconnected with a high speed interconnect such as, for example, but not limited to, NVLink interface 688 and/or PCIe connection 686 developed by NVIDIA. In some instances, GPU 684 is connected via NVLink and/or NVSwitch SoC, and GPU 684 and PCIe switch 682 are connected via PCIe interconnect. Although eight GPUs 684, two CPUs 680, and two PCIe switches are shown, this is not intended to be limiting. Depending on the embodiment, each server 678 may include any number of GPUs 684, CPUs 680, and/or PCIe switches. For example, servers 678 may each include 8, 16, 32, and/or more GPUs 684 .

The server 678 may receive image data from the vehicle via the network 690 representing an image showing unexpected or changed road conditions, such as recently started road works. Server 678 may transmit neural network 692 , updated neural network 692 , and/or map information 694 , including information about traffic and road conditions, to vehicles via network 690 . Updates to map information 694 may include updates to HD map 622, such as information regarding construction sites, potholes, detours, floods, and/or other obstacles. In some examples, neural network 692, updated neural network 692, and/or map information 694 may be derived from new training and/or experience represented in data received from any number of vehicles in the environment. , and/or based on training performed at the data center (eg, using server 678 and/or other servers).

Server 678 may be used to train machine learning models (eg, neural networks) based on the training data. Training data may be generated by the vehicle and/or generated in a simulation (eg, using a game engine). In some instances, the training data is tagged (e.g., if the neural network benefits from supervised learning) and/or undergoes other preprocessing, while in other instances the training data - The data is not tagged and/or pre-processed (eg, if the neural network does not require supervised learning). Training may be performed according to any one or more classes of machine learning techniques, including, but not limited to, the following classes: supervised training, semi-supervised training, unsupervised training, Self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multi-linear subspace learning, manifold learning, representation learning (including preliminary dictionary learning), rule-based machine Learning, anomaly detection, and variations or combinations thereof. After the machine learning model is trained, the machine learning model can be used by the vehicle (eg, transmitted to the vehicle via network 690) and/or the machine learning model can be used to remotely monitor the vehicle. can be used by server 678 to

In some instances, server 678 may receive data from the vehicle and apply the data to state-of-the-art real-time neural networks for real-time intelligent inference. Server 678 may include a deep learning supercomputer powered by GPU 684 and/or a dedicated AI computer, such as the DGX and DGX station machines developed by NVIDIA. However, in some instances, server 678 may include a deep learning infrastructure using only CPU-powered data centers.

The deep learning infrastructure of server 678 may be capable of fast real-time inference, which may be used to assess and verify the health of processors, software, and/or related hardware within vehicle 600. . For example, the deep learning infrastructure can receive periodic updates from the vehicle 600, such as the sequence of images and/or objects that the vehicle 600 has positioned within the sequence of images (e.g., computer vision and/or via other machine learning object classification techniques). The deep learning infrastructure may run its own neural networks to identify objects and compare them to objects identified by vehicle 600, and if the results do not match, the infrastructure can identify objects and compare them to objects identified by vehicle 600. If the AI in the vehicle concludes that it is not functioning properly, the server 678 infers control, notifies the passenger, and instructs the fail-safe computer of the vehicle 600 to complete the safe parking maneuver. 600 can be sent.

For inference, server 678 may include GPU 684 and one or more programmable inference accelerators (eg, NVIDIA's TensorRT). The combination of GPU-powered servers and inference acceleration can enable real-time responsiveness. In other instances, such as when performance is less required, servers powered by CPUs, FPGAs, and other processors may be used for inference.

(exemplary computing device)
FIG. 7 is a block diagram of an example computing device 700 suitable for use in implementing some embodiments of the present disclosure. Computing device 700 may include an interconnection system 702 that indirectly or directly couples the following devices: memory 704, one or more central processing units (CPUs) 706, one or more graphics processing units ( GPU) 708, a communication interface 710, an input/output (I/O) port 712, an input/output component 714, a power supply 716, one or more presentation components 718 (eg, a display), and one or more A plurality of logical units 720; In at least one embodiment, computing device 700 may include one or more virtual machines (VMs) and/or any of its components may be virtual components (eg, virtual hardware components). can include As non-limiting examples, one or more of GPUs 708 may include one or more vGPUs, one or more of CPUs 706 may include one or more vCPUs, and/or One or more of logical units 720 may include one or more virtual logical units. As such, computing device 700 may include discrete components (eg, an entire GPU dedicated to computing device 700), virtual components (eg, a portion of a GPU dedicated to computing device 700), or combinations thereof. .

Although the various blocks in FIG. 7 are shown connected by lines via interconnection system 702, this is not intended to be limiting, but merely for clarity. For example, in some implementations, a presentation component 718 such as a display device may be considered an I/O component 714 (eg, if the display is a touch screen). As another example, CPU 706 and/or GPU 708 may include memory (eg, memory 704 may represent storage devices in addition to memory of GPU 708, CPU 706, and/or other components). In other words, the computing device of FIG. 7 is merely exemplary. "workstation", "server", "laptop", "desktop", "tablet", "client device", "mobile device", "handheld device", "game console", "electronic control unit ( Categories such as "electronic control unit (ECU)", "virtual reality system", and/or other device or system types are all intended to be within the scope of the computing device of FIG. indistinguishable.

Interconnection system 702 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnection system 702 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, video electronics standards (VESA). association) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and/or another type of bus or link. In some examples, there are direct connections between components. As one example, CPU 706 may be directly connected to memory 704 . Additionally, CPU 706 may be directly connected to GPU 708 . Where direct or point-to-point connections exist between components, interconnection system 702 may include PCIe links to implement the connections. In these examples, a PCI bus need not be included in computing device 700 .

Memory 704 may include any of a variety of computer readable media. Computer readable media can be any available media that can be accessed by computing device 700 . Computer readable media may include both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media.

Computer storage media includes volatile and nonvolatile media and/or implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules and/or other data types. or may include both removable and non-removable media. For example, memory 704 may store computer readable instructions (eg, representing programs and/or program elements), such as an operating system. Computer storage media may include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disks, etc. It may include, but is not limited to, a storage or other magnetic storage device or any other medium that may be used to store desired information and that may be accessed by computing device 700 . As used herein, computer storage media does not include the signal itself.

Computer storage media can embody computer readable instructions, data structures, program modules and/or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and can comprise any information delivery media. include. The term "modulated data signal" may refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.

CPU 706 executes at least some of the computer readable instructions to control one or more components of computing device 700 to perform one or more of the methods and/or processes described herein. can be configured to run CPUs 706 may each include one or more (eg, 1, 2, 4, 8, 28, 72, etc.) cores capable of processing multiple software threads simultaneously. CPU 706 may include any type of processor, and may include different types of processors depending on the type of computing device 700 implemented (e.g., processors with fewer cores for mobile devices, and processors with higher number of cores for servers). For example, depending on the type of computing device 700, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or a Complex Instruction Set Computing (CISC) processor. Instruction Set Computing). Computing device 700 may include one or more CPUs 706 in addition to one or more microprocessors or auxiliary co-processors, such as a computing co-processor.

In addition to or instead of CPU 706, GPU 708 executes at least some of the computer-readable instructions to control one or more components of computing device 700 to implement the methods and/or described herein. It may be configured to perform one or more of the processes. One or more of GPUs 708 may be integrated GPUs (eg, with one or more of CPUs 706) and/or one or more of GPUs 708 may be discrete GPUs. In embodiments, one or more of GPUs 708 may be co-processors of one or more of CPUs 706 . GPU 708 may be used by computing device 700 to render graphics (eg, 3D graphics) or perform general purpose computations. For example, GPU 708 may be used for general-purpose computing on GPU (GPGPU). GPU 708 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPU 708 can generate pixel data for an output image in response to rendering commands (eg, rendering commands from CPU 706 received via a host interface). GPU 708 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. A display memory may be included as part of memory 704 . GPU 708 may include two or more GPUs operating in parallel (eg, via links). A link may connect directly to a GPU (eg, using NVLINK) or connect the GPU through a switch (eg, using NVSwitch). When coupled together, each GPU 708 produces pixel data or GPGPU data for a different portion of the output or for different outputs (eg, the first GPU for the first image and the second GPU for the second image). be able to. Each GPU may contain its own memory or may share memory with other GPUs.

In addition to or instead of CPU 706 and/or GPU 708, logic unit 720 executes at least some of the computer readable instructions to control one or more of computing devices 700 as described herein. may be configured to perform one or more of the methods and/or processes of In embodiments, CPU 706, GPU 708, and/or logic unit 720 may discretely or jointly perform any combination of methods, processes, and/or portions thereof. One or more of logical units 720 may be part of and/or integrated in one or more of CPU 706 and/or GPU 708 and/or one of logical units 720 One or more may be discrete components to CPU 706 and/or GPU 708 or otherwise external thereto. In embodiments, one or more of logical units 720 may be one or more of CPUs 706 and/or one or more co-processors of GPU 708 .

Examples of logic unit 720 include one or more processing cores and/or components thereof, e.g., Data Processing Unit (DPU), Tensor Core (TC), Tensor Processing Unit (TPU). Tensor Processing Unit), Pixel Visual Core (PVC), Vision Processing Unit (VPU), Graphics Processing Cluster (GPC), Texture Processing Cluster (TPC) , Streaming Multiprocessor (SM), Tree Traversal Unit (TTU), Artificial Intelligence Accelerator (AIA), Deep Learning Accelerator (DLA) ), logical Arithmetic Unit (ALU), Application Specific Integrated Circuit (ASIC), Floating Point Unit (FPU), Input/Output (I/O) Element, Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) ) elements, and/or the like.

Communication interface 710 includes one or more receivers, transmitters, and/or transceivers that enable computing device 700 to communicate with other computing devices over electronic communication networks, including wired and/or wireless communications. can include Communication interface 710 communicates over a wireless network (eg, Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), a wired network (eg, Ethernet, or InfiniBand). ), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and/or the Internet. can contain. In one or more embodiments, logic unit 720 and/or communication interface 710 communicate data received over a network and/or through interconnection system 702 to one or more GPUs 708 (eg, memory thereof). It may include one or more data processing units (DPUs) for direct transmission.

I/O ports 712 may be connected to other devices, including I/O components 714, presentation components 718, and/or other components, some of which may be built into (eg, integrated with) computing device 700. can be enabled for computing device 700 to be logically coupled to . Exemplary I/O components 714 include microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dishes, scanners, printers, wireless devices, and the like. The I/O component 714 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some cases, the input may be sent to the appropriate network element for further processing. The NUI includes voice recognition, stylus recognition, face recognition, biometric recognition, on-screen and next-to-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 700. Any combination of (as described in more detail below) may be implemented. Computing device 700 may include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touch screen technology, and combinations thereof, for gesture detection and recognition. Additionally, computing device 700 may include an accelerometer or gyroscope that enables motion detection (eg, as part of an inertia measurement unit (IMU)). In some instances, accelerometer or gyroscope output may be used by computing device 700 to render immersive augmented or virtual reality.

Power supply 716 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 716 may provide power to computing device 700 to enable the components of computing device 700 to operate.

Presentation component 718 may be a display (eg, monitor, touch screen, television screen, heads-up display device (HUD), other display type, or combination thereof), speaker, and/or other presentation may contain components. The presentation component 718 can receive data from other components (eg, GPU 708, CPU 706, DPU, etc.) and output data (eg, as images, video, audio, etc.).

(exemplary data center)
FIG. 8 illustrates an exemplary data center 800 that may be used in at least one embodiment of this disclosure. Data center 800 may include data center infrastructure layer 810 , framework layer 820 , software layer 830 and/or application layer 840 .

As shown in FIG. 8, the data center infrastructure layer 810 includes a resource orchestrator 812, grouped computational resources 814, and node computational resources (“node C.R.”) 816(1)-816 ( N), where "N" represents any positive integer. In at least one embodiment, node C. R. 816(1)-816(N) may be any number of central processing units (CPUs) or other processors (DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processors). (including processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input/output (NW I/O) devices , network switches, virtual machines (VMs), power modules, and/or cooling modules, and the like. In some embodiments, node C. R. One or more of nodes C.816(1)-816(N). R. may correspond to a server having one or more of the aforementioned computing resources. Additionally, in some embodiments, node C. R. 816(1)-8161(N) may include one or more virtual components, eg, vGPUs, vCPUs, and/or the like, and/or nodes C. R. One or more of 816(1)-816(N) may correspond to virtual machines (VMs).

In at least one embodiment, the grouped computational resources 814 are separated into separate groups of nodes C.E.C.A. R. 816, or may include multiple racks housed in data centers at various geographic locations (also not shown). A separate group of nodes C . R. 816 may include grouped computing, network, memory or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several nodes C.E.C., including CPUs, GPUs, DPUs, and/or other processors. R. 816 may be grouped into one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and/or network switches in any combination.

A resource orchestrator 812 is one or more nodes C.I. R. 816(1)-816(N) and/or grouped computational resources 814 may be configured or otherwise controlled. In at least one embodiment, resource orchestrator 812 may include a software design infrastructure (“SDI”) management entity for data center 800 . Resource orchestrator 812 may include hardware, software, or some combination thereof.

In at least one embodiment, framework layer 820 may include job scheduler 833, configuration manager 834, resource manager 836, and/or distributed file system 838, as shown in FIG. Framework layer 820 may include frameworks to support software 832 of software layer 830 and/or one or more applications 842 of application layer 840 . Software 832 or application 842 may include web-based service software or applications, such as those provided by Amazon Web Services®, Google Cloud, and Microsoft Azure, respectively. Framework layer 820 includes free and open source software such as Apache Spark™ (“Spark”) that can use distributed file system 838 for large-scale data processing (eg, “big data”). • Can be of the type, but not limited to, software web application framework. In at least one embodiment, job scheduler 833 may include Spark drivers to facilitate scheduling of workloads supported by various tiers of data center 800 . The configuration manager 834 may have the ability to configure different layers, such as the software layer 830 and the framework layer 820, which includes Spark and distributed file systems 838 to support large-scale data processing. Resource manager 836 may have the ability to manage clustered or grouped computational resources mapped or allocated for support of distributed file system 838 and job scheduler 833 . In at least one embodiment, clustered or grouped computing resources may include computing resources 814 grouped in data center infrastructure layer 810 . Resource manager 836 can coordinate with resource orchestrator 812 to manage these mapped or assigned computational resources.

In at least one embodiment, software 832 included in software layer 830 is node C. R. At least portions of 816 ( 1 )- 816 (N) may include software used by grouped computational resources 814 and/or distributed file system 838 of framework layer 820 . The one or more types of software may include, but are not limited to, Internet web page search software, email virus scanning software, database software, and streaming video content software.

In at least one embodiment, application 842 included in application layer 840 is node C. R. 816(1)-816(N), grouped computational resources 814, and/or one or more types of applications used by distributed file system 838 of framework layer 820. . The one or more types of applications may include any number of genomics applications, cognitive computing, and training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and/or one or other machine learning applications used in conjunction with embodiments, including but not limited to machine learning applications.

In at least one embodiment, any one of configuration manager 834, resource manager 836, and resource orchestrator 812 may perform any task based on any amount and type of data obtained in any technically possible manner. number and types of self-modifying actions can be implemented. Self-modifying actions reduce the data center 800 data center 800 from making potentially poor configuration decisions and possibly avoiding underutilized and/or underperforming portions of the data center. It can free the center operator.

Data center 800 may train one or more machine learning models or use one or more machine learning models to predict or infer information according to one or more embodiments described herein. It may include tools, services, software or other resources to do so. For example, a machine learning model may be trained by computation of weight parameters through a neural network architecture using the software and/or computational resources described above with respect to data center 800 . In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be trained by one or more training techniques, such as but not limited to those described herein. can be used to infer or predict information using the resources described above with respect to data center 800 by using weight parameters calculated via .

In at least one embodiment, data center 800 includes CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and/or other devices for performing training and/or inference using the aforementioned resources. Hardware (or corresponding virtual computing resources) can be used. Additionally, one or more of the software and/or hardware resources described above may be used as a service, e.g., image recognition, speech recognition, or other artificial intelligence, to enable users to train or perform reasoning on information. services.

(Exemplary network environment)
A network environment suitable for use in implementing embodiments of the present disclosure includes one or more client devices, servers, network attached storage (NAS), other backend devices, and/or or may include other device types. Client devices, servers, and/or other device types (eg, each device) may be implemented with one or more instances of computing device 700 in FIG. may contain similar components, features, and/or functionality. Additionally, if backend devices (eg, servers, NAS, etc.) are implemented, the backend devices may be included as part of data center 800, an illustration of which is provided herein with respect to FIG. is further detailed in

Components of a network environment may communicate with each other via a network, which may be wired, wireless, or both. A network may include multiple networks or one of multiple networks. Illustratively, the network may include one or more wide area networks (WAN), one or more local area networks (LAN), one or more public networks, such as the Internet and/or public It may include a switched telephone network (PSTN) and/or one or more private networks. When the network comprises a wireless telecommunications network, components such as base stations, communication towers or access points (as well as other components) may provide wireless connectivity.

Compatible network environments include one or more peer-to-peer network environments (where servers may not be included in the network environment) and one or more client-server network environments. (where one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein with respect to servers may be implemented by any number of client devices.

In at least one embodiment, the network environment can include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment is implemented on one or more of the servers, which may include a framework layer, a job scheduler, a resource manager, and one or more core network servers and/or edge servers. may include a distributed file system that is A framework layer may include a framework to support software in the software layer and/or one or more applications in the application layer. Software or applications may include web-based service software or applications, respectively. In an embodiment, one or more of the client devices may use web-based service software or applications (e.g., access the service software via one or more application programming interfaces (APIs)). and/or by accessing the application). The framework layer can be of the type free and open source software web application frameworks that can use distributed file systems, for example, for large-scale data processing (e.g., "big data"). but not limited to this.

A cloud-based network environment may provide cloud computing and/or cloud storage that implements any combination of the computing and/or data storage functions (or one or more portions thereof) described herein. . Any of these various functions may be distributed to multiple locations from a central or core server (eg, one or more data centers that may be distributed across states, regions, countries, and the world). If the connection to the user (eg, client device) is relatively close to the edge server, the core server may assign at least a portion of the functionality to the edge server. A cloud-based network environment may be private (eg, restricted to a single organization), public (eg, available to many organizations), and/or a combination thereof (eg, a hybrid cloud environment). .

A client device may include at least some of the components, features, and functionality of exemplary computing device 700 described herein with respect to FIG. By way of illustration and not limitation, client devices include personal computers (PCs), laptop computers, mobile devices, smart phones, tablet computers, smart watches, wearable computers, personal digital assistants (PDAs), MP3 players, virtual reality headsets, global positioning systems (GPS) or devices, video players, video cameras, surveillance devices or systems, vehicles, boats, airships, virtual machines, drones, robots, handheld communication devices, hospital devices, gaming devices or systems, entertainment systems, vehicle computer systems, embedded system controllers, remote controls, appliances, consumer electronic devices, workstations, edge devices, any combination or any of these depicted devices can be implemented as other suitable devices.

This disclosure relates to computer code or machine-usable instructions, including computer-executable instructions, such as program modules, being executed by a computer or other machine, such as a personal digital assistant or other handheld device. It may be explained in general terms. Generally, program modules including routines, programs, objects, components, data structures, etc. refer to code that performs particular tasks or implements particular abstract data types. The present disclosure may be implemented in various system configurations, including handheld devices, consumer electronics, general purpose computers, more specialized computing devices, and the like. The disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

In this specification, the word "and/or" referring to two or more elements should be construed to mean only one element or any combination of elements. For example, "element A, element B, and/or element C" means only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C may be included. Additionally, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B . Furthermore, "at least one of element A and element B" means at least one of element A, at least one of element B, or at least one of element A and element B at least one of

The invention of this disclosure has been described with specificity to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors believe that the claimed invention, in conjunction with other present or future technology, may include different steps or combinations of steps similar to those described in this document. However, it is intended that it may be embodied in other ways. Additionally, although the terms "step" and/or "block" may be used herein to imply different elements of the method used, these terms are used to clarify the order of the individual steps. It should not be construed as implying any particular order among the various steps disclosed herein, unless and except when so stated.

Claims (20)

A processor comprising one or more circuits,
the one or more circuits comprising:
calculating, using a neural network, data representing one or more parameters of an operating region corresponding to said environment based at least on image data of the environment generated using one or more sensors;
determining, based at least in part on the data, a level of egomachine performance corresponding to the environment;
A processor that controls the operation of the egomachine according to the operation level. The one or more parameters are
camera visibility level,
Visual occlusion classification,
lighting level,
condition of the track surface,
viewing distance,
2. The processor of claim 1, comprising at least one of: a scene type classification; or a distance to a scene corresponding to the scene type classification. A processor comprising one or more circuits,
the one or more circuits comprising:
generating intermediate data using the trunk of said neural network;
calculating a subset of the data using at least one head of the neural network based on the intermediate data, the subset of the data comprising:
camera visibility level,
Visual occlusion classification,
lighting level,
condition of the track surface,
viewing distance,
2. The processor of claim 1, representing at least one of: a scene type classification; or a distance to a scene corresponding to the scene type classification. 4. The processor of claim 3, wherein the at least one head includes one or more downsampling layers followed by one or more upsampling layers. 4. The processor of claim 3, wherein the trunk includes one or more convolutional layers and the intermediate data represents one or more feature maps. 4. The processor of claim 3, wherein said at least one head includes at least one fully connected layer. 4. The processor of claim 3, wherein said at least one head includes a sigmoid function. wherein the determination of the operating level of the egomachine comprises: the camera view occlusion level, the view occlusion classification, the lighting level, the path surface condition, the viewing distance, the scene type classification, or the distance to a scene; 3. The processor of claim 2, including weighting at least one of with a predetermined weight. 9. The processor of claim 8, wherein the camera's view occlusion level has the highest weight among the predetermined weights. The operating levels correspond to levels of vehicle autonomy including level 0 (L0), level 1 (L1), level 2 (L2), level 3 (L3), level 4 (L4), or level 5 (L5). 2. The processor of claim 1, wherein: 3. The processor of claim 2, wherein the scene type classification includes at least one of tunnels, construction zones, toll booths, or closed lanes. wherein said data comprises camera-based input for said determination of said level of motion, said determination of said level of motion further being at least partially based on data generated using one or more other sensor modalities; 2. The processor of claim 1, wherein: the processor
control systems for autonomous or semi-autonomous machines,
cognitive systems for autonomous or semi-autonomous machines,
a system for performing simulation operations;
a system for performing deep learning operations;
Systems implemented using edge devices,
systems implemented using robots,
a system that incorporates one or more virtual machines (VMs);
2. The processor of claim 1, included in at least one of: a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources. one or more sensors for generating image data;
one or more processing units,
at least one of lighting level, path surface conditions, viewing distance, scene type classification, or distance to scene corresponding to said scene type classification, using a neural network, based at least in part on image data; and at least one of an amount of camera view occlusion or a cause of camera view occlusion;
determining a level of activity of the egomachine based at least in part on the data;
determining one or more control operations based at least in part on the activity level;
A system comprising one or more processing units. wherein said data comprises camera-based input for said determination of said level of motion, said determination of said level of motion further being at least partially based on data generated using one or more other sensor modalities; 15. The system of claim 14, wherein the system is based on wherein said determination of said level of operation of said egomachine is at least one of said lighting level, said path surface condition, said viewing distance, said scene type classification, or said distance to said scene corresponding to said scene type classification. and at least one of an amount of the camera's view occlusion or the source camera's view occlusion with a predetermined weight. 17. The system of claim 16, wherein the amount of camera view occlusion has the highest weight among the predetermined weights. said system
control systems for autonomous or semi-autonomous machines,
cognitive systems for autonomous or semi-autonomous machines,
a system for performing simulation operations;
a system for performing deep learning operations;
Systems implemented using edge devices,
systems implemented using robots,
a system that incorporates one or more virtual machines (VMs);
15. The system of claim 14, included in at least one of: a system that is at least partially implemented in a data center; or a system that is at least partially implemented using cloud computing resources. A processor including processing circuitry, said processing circuitry processing data computed using a neural network and image data generated using one or more sensors of said egomachine. determining a level of operation of said egomachine based at least in part on a network, wherein said data includes at least one of lighting level, surface conditions, viewing distance, scene type classification or said scene type classification; A processor representing at least one of a distance to a scene corresponding to the type classification and at least one of an amount of camera view occlusion or a cause of camera view occlusion. wherein said data comprises camera-based input for said determination of said level of motion, said determination of said level of motion further being at least partially based on data generated using one or more other sensor modalities; 20. The processor of claim 19, based on.

Images