DE102020124285B4 - Vision-based teleoperation of a moving robotic system - Google Patents

Abstract

A computer-implemented method comprising: determining (1406), from an image of a hand (102, 106, 110, 114, 802-817), a first pose (702) of the hand; translating (1408) the first pose (702) of the hand (102, 106, 110, 114, 802-817) into a corresponding second pose of a robot hand (104, 108, 112, 116, 204, 704, 818-833), determining (1410) a set of motions that the robot hand (104, 108, 112, 116, 204, 704, 818-833) transitions from a first pose of the robotic hand (104, 108, 112, 116, 204, 704, 818-833) to the second pose; causing (1412) the robotic hand (104, 108, 112, 116, 204, 704, 818-833) to perform the set of movements; andkinematically adjusting (314) the robotic hand (104, 108, 112, 116, 204, 704, 818-833) by minimizing a cost function based at least on vectors derived from primary fingers (406, 408, 410) of the hand (102, 106, 110, 114, 802-817) pointing to the thumb (412) of the hand (102, 106, 110, 114, 802-817), and vectors pointing between the primary fingers (406, 408, 410) of the hand (102, 106, 110, 114, 802-817).

Description

TECHNICAL PART

At least one embodiment relates to the use of robots to perform and facilitate tasks under the control of a human operator. For example, at least one embodiment relates to controlling a robotic arm by mimicking an action of a human hand according to various novel techniques described herein.

BACKGROUND

Controlling robots to perform tasks can be a difficult and challenging problem. One method of controlling a robot is direct human control. For example, some robotic systems are equipped with a joystick or programming interface that allows the operator to move the robot. However, such interfaces tend to be unintuitive and difficult to use, requiring significant training and practice to use. This is especially true when the operator is attempting to perform a complex task such as interacting with other objects.

In this context, the U.S. 2013/0211592 A1 a teleoperation system that allows a robotic arm to move by following a movement of a user's hand movement without an additional mechanical device. The teleoperation system includes a follower robot having a robot arm, and a main console configured to detect a user's gesture and control the follower robot from a remote location so that the follower robot moves by following the user's gesture.

the DE 11 2004 002 219 T5 relates to a method for driving a robot. The robot can perform a motion similar to that of a moving object or an imitation thereof by obtaining an image of the moving object or its imitation. The robot can therefore be easily driven using image data as input data. When the moving object is a human's hand, an accuracy of specifying the image data can be increased when the image data includes data created considering an individual difference in the human hand. The elements that determine individual differences include the curvature of positions of phalanges of four fingers except the thumb, the thrust of the phalanges of the thumb, an opening angle between two adjacent fingers, the difference of reference angles of two opening fingers, and the bulge of the thumb.

the U.S. 10,406,686 B2 discloses a system for generating instructions for operating a robot to perform operations on a workpiece. The system has a 3D model or models of a robot scene that provide robot scene data. The system also has data from one or both hands of a gesture made with one or both hands and a computing device containing program code that processes the robot scene data and the gesture data to generate an instruction for operating the robot . Position and orientation of the hand and a finger can be used to calculate a corresponding position and orientation of the robot tool in the robot scene. One way to calculate the robot target is to intersect the finger direction with the scene data. This can be done with an image of the scene or a CAD of the scene.

the JP 2015-071206 A provides a control device, a robot, a teaching data generation method, and a program capable of reducing training costs for a robot operation in a holding operation and an operation on a workpiece by a held tool. The control device includes: an image recognition unit for performing image recognition processing, and a processing unit for performing specific processing of posture data indicating the posture of a holding part and generation processing of training data of the robot. In addition, the image recognition unit performs image recognition processing on a captured image of the holding state in which a trainer's hand holds a tool. Based on the result of the image recognition processing of the holding state, the processing unit generates the training data including the position adjustment data of the holding part in the holding state.

the JP 2011-110620 A relates to a method for controlling an action of a robot. A monocular camera and a stereo camera capture a training image including a wrist and a hand of a trainer. A controller determines finger coordinates indicating the joints of the fingers and the positions of the fingertips based on the teaching image and calculates an angle of each joint of a robot hand based on the finger coordinates.

the CN 104589356A relates to a method for teleoperative control of an artificial hand based on hand movement detection. After first tracking an outer contour of the hand with a hybrid active contour model, a particle filter algorithm is applied to track the hand position. An active contour model combining depth information and YCbCr skin color detection space technology is used to solve the problem of getting a solution when the image contains strong noise or the target has weak boundaries. A particle filter position detection method based on template matching is used, which can detect and estimate the three-dimensional pose of a human hand with high-dimensional features.

The improvement of teleoperation interfaces is an important task in the field of robot control.

This object is solved according to the present invention by a computer-implemented method, a system and a computer-readable medium as defined in the independent claims. The dependent claims define embodiments of the invention.

character list

• 1 Figure 12 shows examples of teleoperations for various tasks, according to at least one embodiment;
• 2 shows an example of a robot arm and a robot hand, according to at least one embodiment;
• 3 12 shows an example of a system that tracks the movement of a human hand and translates the movements into corresponding movements of a robotic arm, according to at least one embodiment;
• 4 12 shows an example of a glove that may be used to assist in estimating a hand pose, in accordance with at least one embodiment;
• 5 12 shows an example of an architecture of a system that determines a hand pose, according to at least one embodiment;
• 6 12 shows an example of estimating a hand pose from a point cloud, according to at least one embodiment;
• 7 12 shows an example of a pose of a human hand and a robotic gripper performing a corresponding pose, according to at least one embodiment;
• 8th 12 shows an example of poses of a human hand and corresponding poses of a robot gripper, according to at least one embodiment;
• 9 Figure 12 shows an example of a task in which a robot is controlled to remove paper from a folded wallet, according to at least one embodiment;
• 10 Figure 12 shows an example of a task in which a robot is controlled to retrieve an item from a drawer, in accordance with at least one embodiment;
• 11 Figure 12 shows an example of a task in which a robot is controlled to open a vessel, according to at least one embodiment;
• 12 Figure 12 shows an example of the execution time of teleoperation tasks, according to one embodiment;
• 13 Figure 12 shows an example of the success rate of teleoperation tasks, according to one embodiment;
• 14 Figure 12 shows an example of a process that, as a result of execution by a computer system, directs a robotic arm to perform a task by emulating the movement of a human hand;
• 15A 12 shows an inference and/or training logic, according to at least one embodiment;
• 15B 12 shows an inference and/or training logic, according to at least one embodiment;
• 16 shows a training and the use of a neural network, according to at least one embodiment;
• 17 Figure 12 shows an example of a data center system, according to at least one embodiment;
• 18A shows an example of an autonomous vehicle, according to at least one embodiment;
• 18B shows an example of camera positions and fields of view for the autonomous vehicle 18A , according to at least one embodiment;
• 18C FIG. 12 is a block diagram showing an example system architecture for the autonomous vehicle of FIG 18A shows, according to at least one embodiment;
• 18D is a diagram showing a system for communication between cloud-based servers and the autonomous vehicle of the 18A shows, according to at least one embodiment;
• 19 Figure 12 is a block diagram showing a computer system according to at least one embodiment;
• 20 Figure 12 is a block diagram showing a computer system according to at least one embodiment;
• 21 Figure 12 shows a computer system, according to at least one embodiment;
• 22 Figure 12 shows a computer system, according to at least one embodiment;
• 23A Figure 12 shows a computer system, according to at least one embodiment;
• 23B Figure 12 shows a computer system, according to at least one embodiment;
• 23C Figure 12 shows a computer system, according to at least one embodiment;
• 23D Figure 12 shows a computer system, according to at least one embodiment;
• 23E and 23F show a common programming model, according to at least one embodiment;
• 24 Figure 12 shows example integrated circuits and associated graphics processors, according to at least one embodiment;
• 25A and 25B 12 show example integrated circuits and associated graphics processors, according to at least one embodiment;
• 26A and 26B Figure 12 shows further example graphics processor logic, in accordance with at least one embodiment;
• 27 Figure 12 shows a computer system, according to at least one embodiment;
• 28A Figure 12 shows a parallel processor, according to at least one embodiment;
• 28B Figure 12 shows a partition unit, according to at least one embodiment;
• 28C 12 shows a processing cluster, according to at least one embodiment;
• 28D 12 shows a graphics multiprocessor, according to at least one embodiment;
• 29 shows a multi-graphics processing unit
• GPU), according to at least one embodiment;
• 30 12 shows a graphics processor, according to at least one embodiment;
• 31 12 is a block diagram showing a processor microarchitecture for a processor, according to at least one embodiment;
• 32 12 shows a processor for deep learning applications, according to at least one embodiment;
• 33 12 is a block diagram showing an example neuromorphic processor, according to at least one embodiment;
• 34 12 shows at least parts of a graphics processor, according to one or more embodiments;
• 35 12 shows at least parts of a graphics processor, according to one or more embodiments;
• 36 12 shows at least parts of a graphics processor, according to one or more embodiments;
• 37 3710 shows a block diagram of the graphics processing engine 3710 of a graphics processor, according to at least one embodiment.
• 38 12 shows a block diagram of at least parts of a graphics processor core, according to at least one embodiment;
• 39A and 39B 12 show thread execution logic 3900 comprising an array of processing elements of a graphics processor core, according to at least one embodiment;
• 40 Figure 12 shows a parallel processing unit (“PPU”), according to at least one embodiment;
• 41 12 shows a general processing cluster ("GPC"), according to at least one embodiment;
• 42 Figure 12 shows a memory partition unit of a parallel processing unit ("PPU"), according to at least one embodiment; and
• 43 12 shows a streaming multiprocessor, according to at least one embodiment.

DETAILED DESCRIPTION

This document describes a system and method for controlling a robot using teleoperation. In at least one embodiment, the system estimates the pose of a human hand and then directs a robot into an appropriate pose. In one example, a human operator performs a hand motion that is captured with a depth camera, and the system uses the depth camera data to generate a point cloud. The point cloud is processed to determine a pose of the hand. The hand pose identifies the position of the fingertips of the hand and an approximation of the joint angles and bones of the hand. The system then translates the pose of the hand into a corresponding pose of a robot hand. In some examples, the system corrects for differences in the size of the robot hand and differences in the number of fingers between the human hand and the robot hand. In some examples, the pose of a human hand can be translated into a robotic hand with a greater or lesser number of fingers. In some examples, the pose of a human hand can be translated into the pose of an excavator, industrial robot, or vehicle. For example, a human can mimic the shape of a claw or shovel on earthmoving machines and the system can translate the pose of a human hand into a corresponding pose of the claw or shovel, thus providing intuitive control of the machine. The human operator can view the robot either directly or remotely via a video display and is able to command the robot to perform a task by duplicating the desired movement in real time.

In at least one embodiment, teleoperation provides the ability to provide robotic systems with sophisticated reasoning, intuition, and creativity to accomplish tasks. At least one example described here is a low-cost, vision-based teleoperation system that allows full control over the entire 23 autonomous (DoA) robotic system by observing a bare human hand. Some examples allow operators to perform a variety of complex manipulation tasks beyond simple pick and place operations. In at least one embodiment, high-dimensional, multimodal status data can be collected that can be used in the future to learn sensorimotor strategies for demanding manipulation tasks. The system performance of at least one embodiment is measured by speed and reliability metrics for two human demonstrators on a variety of tasks. In various embodiments, the system can be implemented by one or more systems, as in 15-42 described/illustrated.

Various example embodiments may be applied in the fields of search and rescue, space travel, medicine, prosthetics, and applied machine learning. In at least one embodiment, teleoperation enables a robotic system to perform complex tasks by itself that harnesses human cognition, creativity and responsiveness via a human-machine interface (“HMI”). In at least one embodiment, haptic feedback and enhanced human skeletal and finger tracking are incorporated. At least one embodiment is a low-cost, marker-less, glove-less teleoperation solution that leverages innovations in computer vision, optimization, motion generation, and GPU computing. At least one embodiment has the ability to sense and relay fine, articulated manipulations to drive a highly agile robotic system to perform a variety of gripping and manipulation tasks. In one example, four Intel RealSense depth cameras and two NVIDIA GPUs combined with deep learning and nonlinear optimization created an articulated teleoperation system with a minimal footprint. Despite the lack of tactile feedback, human cognition makes the system very powerful and effective. In at least one embodiment, this result supports studies of the human gaze that suggest that humans learn to use vision to plan, control, and predict state of hand actions before precisely controlling their hand. In some examples, the depth cameras may be RADAR or LIDAR based imaging systems, or medical imaging systems such as magnetic resonance imaging machines or X-ray imaging machines.

In various examples, the teleoperation system takes advantage of man's ability to plan and execute his physical actions and to predict the consequences of his physical actions by sight alone - a sufficient condition for the solution of a variety of tasks. Various embodiments have one or more of the following advantages: a marker-less, glove-less, and fully vision-based teleoperation system that dexterously positions a highly posable direct-mimic robotic hand-arm system, a novel cost function, and a projection scheme for the kinematic realignment of human wrists to robotic wrists that enhances the dexterity of the hand and preserving the feasibility of precision grips in the event of errors in tracking the joints of the hand, a demonstration of a teleoperation system for a variety of tasks requiring particular fine manipulation and dexterity, with system evaluation by two trained human operators (also called pilots, or demonstrators). ) showed that despite the lack of tactile feedback, high success rates can be achieved with the tasks.

In various embodiments, the developed system enables such a skilful robot manipulation through depth observations of the bare human hand with several cameras. In some examples, the system may be a fully vision-based, gloveless teleoperation system that dexterously moves a highly maneuverable robotic hand-arm system through direct imitation. The system can also perform a number of tasks that require fine manipulation and dexterity in particular (e.g., pulling paper money out of a wallet and picking up two dice with four fingers at the same time, as in 1 shown).

1 12 shows an example of a teleoperation for various tasks, according to at least one embodiment. In one example, a robot gripper 104 grips a cylinder using a gripping pose based on a pose of a human hand 102. In another example, a robot gripper 108 grips a cube with a gripping pose based on a human hand 106. In another example, a robot gripper 112 grips a cup using a gripping pose based on a human hand 110. In another example, a robot gripper 116 grips a wallet using a gripping pose based on a human hand 114.

The teleoperation setup may include a robotic system and an adjacent arena for human operators, as in 2 shown. 2 shows an example of a robot with tactile sensors according to at least one embodiment. In at least one embodiment, a robot 202 has a robot gripper 204 that is used to grip objects using tactile sensors. In at least one embodiment, a set of cameras 206, 208, 210, and 212 are used to view the robot's 202 workspace. In at least one embodiment, gripper 204 includes a set of tactile sensors 216, 218, 220, and 222 that provide sensory information to a control computer system. In at least one embodiment, the tactile sensors may be covered with a friction material to enhance the robot's ability to grasp an object.

In some embodiments, as in 2 shown, the robotic system can be a KUKA LBR iiwa7 R800 series arm with a Wonik Robotics Allegro hand equipped with four SynTouch BioTac tactile sensors on the fingertips and a 3M TB641 grip tape applied to the inner surfaces of the phalanges and palm, where the rubbery surfaces of both the BioTac sensors and the 3M tape can improve hand friction while the BioTacs themselves 23 can generate signals that can later be used to learn sensorimotor control using demonstrations. The human sphere of action can be a table dressed in black surrounded by four calibrated and time-synchronized cameras, such as Intel RealSense RGB D cameras, which can be spatially arranged to cover a working area of 80 cm × 55 cm × 38 cm cover. In some examples, the cameras can be right next to the robot to improve line of sight and visual proximity since teleoperation relies entirely on human vision and spatial reasoning. It should be noted that 2 is intended as an embodiment, and in various embodiments the system may include any robotic system using any robotic component (e.g., various types of robotic arms, hands, tactile sensors, handles, other sensors, cameras, and/or variations thereof) in any suitable environment can.

To create a natural-feeling teleoperation system, an imitation-type paradigm can be used. The mere movement of a human hand - pose and finger configuration - can be constantly observed and measured by a visual perception module. The movement of the human hand can then be forwarded to the robot system in such a way that the copied movement is self-explanatory. This approach can allow a human operator to curve and arrange their fingers, form grips, reorient their palms, and translate, with the robotic system following in a similar fashion. In at least one embodiment, the system relies heavily on Dense Articulated Real Time Tracking ("DART"), which can form the backbone of tracking the pose and joint angles of the human hand. The system architecture and component connections are in 3 shown in one embodiment.

3 12 shows an example of a system that tracks a human hand in real time and controls a robot to perform a corresponding movement, according to at least one embodiment. In at least one embodiment, the system operates using three threads, which are independent processes running on one or more processors of a computer system. In at least one embodiment, the learning thread provides hand pose and joint angle presets using a fused input point cloud sourced from four cameras in the studio. In at least one embodiment, the tracking thread executes DART for default hand tracking and kinematic retargeting required to map the human hand configuration to the Allegro hand. In at least one embodiment, the thread of control executes the Riemannian motion policies to provide the KUKA and Allegro hands with the common targeting commands given the hand pose and joint angles.

In at least one embodiment, one or more images of a hand are obtained from RGB depth ("RGB-D") cameras 302 . The images are processed by a Pointnet:Stage 1 304, a Pointnet:Stage 2 306, and a Jointnet 308 to generate a hand pose for the hand in the images. In at least one embodiment, an articulated hand model 310 and the hand pose are processed with DART 312 and kinematic realignment (tracking, repositioning, retargeting, kinematic retargeting) 314 to generate a corresponding hand pose for a robotic gripper. In at least one embodiment, a control thread applies the Riemannian motion guidelines 318 to the gripper's hand pose and the resulting information is used to control the robot 320 .

3 Figure 12 shows an embodiment that includes the following components: 1) DART for hand tracking, 2) Deep Neural Networks for estimating human hand state and robustness of DART, 3) Refinement of human hand state with DART and its transformation by nonlinear optimization in Allegro hand states; and 4) motion generation and control through Riemannian Motion Policies (RMPs) and torque impedance controllers. An example is given in 3 shown in which the above components are concatenated. In at least one embodiment, the system has an overall latency of about one second.

In at least one embodiment, DART is used to continuously track the pose and joint angle of a human hand. In at least one embodiment game, DART uses a moving model of the hand that is registered using an input point cloud. A human hand model can be obtained and converted into a single mesh model. Using CAD (Computer Aided Design) software, the fingers of the mesh model can be partitioned into their respective proximal, medial and distal limbs and restored as separate meshes along with an associated Extensible Markup Language (XML) file describing their kinematic arrangement be exported. In total, the human hand model may have 20 pivot joints: four joints per finger with one abduction joint and three flexion joints.

In at least one embodiment, DART is a model-based tracker that performs non-linear optimization and initialization (e.g., from the previous frame or a first guess). In some examples, if this initialization is not within the convergence basin, the tracker may not converge to the correct solution. In various embodiments, when tracking the human hand model with point cloud data, the hand model can often snap to incorrect local minima, resulting in tracking errors every few minutes. Therefore, in order to reliably track the human hand over long periods of time - which is useful for teleoperation - it may be desirable to have reliable presets for the hand pose of a human hand, clean segmentation of the hand, and a multi-camera studio to prevent the hand model from snapping to unexpected local minima. In various embodiments, one method for generating preset hand poses is to train a neural network on a large data set of human hand poses generated by camera images.

4 Figure 12 shows an example of a color glove used to acquire hand pose and segmentation. In at least one embodiment, a glove 402 includes five fingers 404, 406, 408, 410, and 412, each of which is colored with a different colored fabric. In at least one embodiment, unique colors are printed, so generating annotations with OpenCV color thresholds is trivial. In at least one embodiment, the colors on the back of the palm 414 uniquely determine the pose of the hand.

In at least one embodiment, a cloth glove (as in 4 shown) with colored blobs as an effective solution to obtain a hand pose using a deep neural network. In at least one embodiment, data collection occurs in two phases. In at least one embodiment, in the first phase, the user wears the glove to obtain hand pose specifications for DART that can robustly track the human hand. In at least one embodiment, this process generates hand pose and joint angle annotations for raw depth maps from the second phase RGB-D cameras. In at least one embodiment, the second phase uses these annotations and works with raw point clouds from corresponding depth maps and frees the operator from the need to wear the glove.

In at least one embodiment, the color glove is used for hand tracking. In at least one embodiment, the glove has colored blobs on the fingertips and three on the back of the hand. In at least one embodiment, hand tracking includes both hand pose and finger joint angles. In at least one embodiment, the user moves their hand across a table in a multi-camera studio with four Intel RealSense D415 RGB-D cameras facing down on the table. In at least one embodiment, the problem of estimating gloved hand pose is formulated via the location of key points. In at least one embodiment, ResNet-50 with Spatial-Softmax is used to back-calculate from an RGB image to the 2D locations of the centers of the colored dots on the glove, called GloveNet. In at least one embodiment, the colored points on the fingertips are also recalculated, but they have not been found to ultimately help in full hand tracking, and the predictions of the points on the back of the hand are used to estimate hand pose.

In at least one embodiment, hand pose may be estimated from three unique key points indicated by three different colored dots on the back of the glove palm. In at least one embodiment, to obtain annotations for the centers of the points, the HSV threshold in OpenCV is used to generate segmentations and calculate the centroids of these segmented colored points. In at least one embodiment, the user contributes to support the segmentation for high quality He wears a black glove with colored dots and moves his hand across a table that is also covered with black cloth. In at least one embodiment, the hand pose can be determined via predicted 2D positions of the points from all four cameras: the 2D key points are converted to their corresponding 3D positions using the depth values, so that each point has a total of four 3D predictions from four cameras Has. In at least one embodiment, these 3D positions are filtered and temporally smoothed to obtain the hand pose. In at least one embodiment, hand segmentation is also achieved by removing the 3D points that lie outside the bounding volume of the hand. In at least one embodiment, the dimensions of this volume are heuristically derived from the hand pose derived from the neural network predictions. In at least one embodiment, and critically, DART now optimizes only on the segmented hand points, preventing the hand model from slipping to points on the arm, as is often observed when using a full point cloud. In at least one embodiment, DART does not use RGB images - the glove only provides pose priorities and assisted hand segmentation - and therefore the result of DART with hand pose priorities and segmentation in the first phase is annotation generation for raw point clouds that were captured by the cameras for a second phase, which can operate using the bare human hand.

In at least one embodiment, it is desirable in a second phase that the user no longer has to wear gloves for every teleoperation in the future. In at least one embodiment, the first phase works with an RGB image, while the second phase works directly with a fused point cloud of the bare human hand obtained by back-projecting four depth maps from extrinsically calibrated cameras into a global frame of reference. In at least one embodiment, the annotations for this phase come from the data generated in the first phase. In at least one embodiment, since the camera also provides synchronized depth images, the first stage tracking results may provide point cloud annotations.

In at least one embodiment, the merged point cloud contains both the points on the table and the human body and arm, making it imperative to locate the hand first. In at least one embodiment, points on the table are removed by plane fitting and the remaining points - containing the arm and human body - are provided with an architecture that both locates the hand and provides the pose of a human hand. In at least one embodiment, the network estimates the hand pose using voting based regression on the 3D locations of specified key points on the hand. In at least one embodiment, the network is trained to predict the 3D coordinates of 23 key points on the hand - 4 joint key points on 5 fingers each and 3 on the back of the palm to estimate hand pose. In at least one embodiment, the loss function is a standard Euclidean loss between the predicted and ground truth key points along with the voting loss. In at least one embodiment, an auxiliary segmentation loss is also added to obtain hand segmentation. In at least one embodiment, for efficiency, any input point cloud of size Nx3 is uniformly subsampled to a fixed size of 8192x3 before feeding it into a mesh.

In at least one embodiment, a reasonable estimation and segmentation of the hand pose is achieved, but with this network it is still difficult to obtain high quality predictions for the 20 pivot key points on the fingers. In at least one embodiment, the uniform undersampling used in input means that the points on the fingers are not densely sampled. Therefore, a second stage refinement is required, in which the points on hand are resampled from the original raw point cloud, taking into account the first stage pose and segmentation. According to at least one embodiment, the entire network architecture is in 5 shown. In at least one embodiment, the second stage is only trained on the loss functions of the key points, no segmentation is required. In at least one embodiment, the points sampled on the hand are used instead and the 23 key points are accurately predicted. In at least one embodiment, additional randomization is added to the hand pose for the second stage to provide robustness to any inaccuracies in the hand pose from the first stage. According to at least one embodiment 7 , how the second stage refinement improves the system. In at least one embodiment, both stages of a network are traced to 100K point clouds ers accumulated over a game of 30-45 minutes each for a total of 7-8 hours by running darts with prompts from the glove. In at least one embodiment, together they provide annotations for key points, joint angles, and segmentation. The training lasts 15 hours in total on a single NVIDIA TitanXp GPU.

In at least one embodiment, keypoints are a natural representation for Euclidean space as used in PointNet++ architectures, while most moving models use joints as a natural parameterization. In at least one embodiment, it is desirable to have an output in joint space that can serve as a common template for DART. In at least one embodiment, a third neural network is trained that maps 23 key point positions predicted by the PointNet++-inspired architecture to corresponding joint angles. In at least one embodiment, this neural network, called JointNet, is a two-layer fully connected network that takes inputs of size 23×3 and predicts a 20-dimensional vector of joint angles for fingers.

In at least one embodiment, the neural networks are trained on data collected across multiple human hands within the confines of the studio's work volume, thereby ensuring accurate pose adjustments for this application and enabling reasonable targets for DART. In at least one embodiment, hand tracking performed qualitatively well for hands that are geometrically close to the human DART hand model. In at least one embodiment, for a validation set of 7000 images of different hand poses and finger configurations, the average key point error was 9.7 mm and the joint error was 1.33 degrees per joint.

In 5 the architecture inspired by PointNet++ works in two stages. The first stage segments the hand (as shown in pink) and provides a rough hand pose. The second stage refines the hand pose, taking into account the hand segmentation and the pose from the first stage. The loss functions include the segmentation loss, the Euclidean loss between the predicted keypoints and the ground truth keypoints, and the matching loss as used in [11]. Because the second stage refines the key points, the segmentation loss is not needed. The sentence abstraction takes an input of size N × (d + C) and outputs N' × (d + C ₄ ), while the feature propagation layer takes an input equal to N' × (d + C) and a tensor of size N × ( d + C ₃ ). Together they form the backbone of the network. MLPs are used to map the PointNet++ backbone integrations to the appropriate desired outputs. Further details of the network can be found in the appendix.

In 6 the input point cloud includes points from both the table and the human body and arm, according to one embodiment. In at least one embodiment, a plane has been fitted to remove points on the table and the remaining points are fed into the first stage of a network that restores the hand pose. In at least one embodiment, the second stage refines the pose and provides a more accurate result. In at least one embodiment, the hand shots on the right show the result of Stage 1 and Stage 2 in 5 .

In at least one embodiment, data collection is initiated with DART and without handpose constraints, thereby training a first network to produce handpose constraints. As a result, DART and the most recently trained neural network can generate increasing amounts of data. In at least one embodiment, the network is continually updated with the latest datasets to produce better and better specifications for DART, which may ultimately expand the range in which DART can function without failure. In some examples, the hand pose neural network may be a PointNet-based architecture that works directly with fused point cloud data obtained by back-projecting depth images from extrinsically calibrated depth cameras into a single global reference frame with annotations provided by DART. Since the merged point cloud contains both the points on the table and the human body and arm in various exemplary embodiments, it may be essential in various cases to first locate the hand. Points can be removed from the table by fitting a plane and passing the remaining points containing the arm and human body to PointNet, which can both locate the hand and provide the hand pose. PointNet can be based on estimating hand pose using a voting-based regression scheme on the 3D positions of specified key points on the hand, a technique that may involve spatial softmax commonly used in 2D key point localization. In various embodiments PointNet can be trained to predict 3D coordinates of 23 key points specified on the hand - four joint key points for each of the five fingers and three key points on the back of the hand to estimate hand pose. The loss function may be the Euclidean distance between the predicted points and the key points of ground truth. Additionally, an auxiliary segmentation loss may be included to obtain hand segmentation. For efficiency, any input point cloud can be uniformly subsampled down to a fixed size of 8192 × 3 before being fed into PointNet. In at least one embodiment, a reasonable estimate of hand pose and segmentation may be achieved, and high quality predictions may not yet be obtained for the 20 common keypoints on the fingers. In at least one embodiment, the uniform undersampling used in the input may indicate that the points on the fingers are not densely sampled, so refinement may be required in a second stage, where the points on the hand are subsampled from the original raw point cloud be resampled to account for the pose and segmentation of the first stage. In at least one embodiment, the second stage can be trained on the same loss functions, but can instead use only the hand sampled points to accurately predict the 23 key points. In at least one embodiment, random noise may be added to the second stage hand pose to provide robustness to any inaccuracies in the first stage hand pose. 4 Figure 12 illustrates the second level of refinement within the system, according to at least one embodiment. In at least one embodiment, both stages of PointNet can be trained on 100K point clouds collected over a batch of 30-45 minutes each for a total of 7-8 hours by running DART to provide annotation for key points, joint angles, and segmentation. In at least one embodiment, a third neural network may be trained that maps the key points predicted by PointNet to the corresponding joint angles to provide finger joint angle specifications. This neural network, which can be referred to as a JointNet, can be a two-layer fully connected network that takes inputs of size 23×3 and predicts a 20-dimensional vector of joint angles for fingers.

In at least one embodiment, the neural networks are trained on data collected from multiple human hands, thereby ensuring accurate pose matching for this system and allowing reasonable targets for DART. In some embodiments, hand tracking may work better for hands that are geometrically close to the human DART hand model.

wobei q_h, q_a die Winkel des Modells der menschlichen Hand bzw. der Allegro-Hand sein können, r_i ∈ R³ der Vektor, der vom Ursprung eines Koordinatensystems zu einem anderen zeigt, ausgedrückt im Ursprungskoordinatensystem sein kann (siehe 7). Darüber hinaus ist in mindestens einem Ausführungsbeispiel $d_i=\Vert r_i\left(q_h\right)\Vert \text{und}{r}_i^{}\left(q_h\right)=\frac{r_i\left(q_h\right)}{\Vert r_i\left(q_h\right)\Vert }.$

Die Umschaltgewichtsfunktion s(d_i) kann wie folgt definiert sein: $$s\left(d_i\right)=\{\begin{array}{c}\mathrm{1,}{d}_i>\epsilon \\ \mathrm{200,}{d}_i\le \epsilon \wedge r_i\left(q_h\right)\in S_1\\ \mathrm{400,}{d}_i\le \epsilon \wedge r_i\left(q_h\right)\in S_2\end{array}$$

wobei S₁ Vektoren sein können, die von einem Primärfinger (Zeige-, Mittel-, Ringfinger) ausgehen und auf den Daumen zeigen, und S₂ m Vektoren zwischen zwei Primärfingern sein können, wenn beide Primärfinger assoziierte Vektoren ∈ S₁ haben (z.B. beide Primärfinger werden mit dem Daumen projiziert). In mindestens einem Ausführungsbeispiel ist die Distanzfunktion ƒ(d_i) ∈ R definiert als: $$ƒ\left(d_i\right)=\{\begin{array}{c}\mathrm{\beta }{d}_i,d_i>\epsilon \\ {\mathrm{\eta }}_1,d_i\le \epsilon \wedge r_i\left(q_h\right)\in S_1\\ {\mathrm{\eta }}_2,d_i\le \epsilon \wedge r_i\left(q_h\right)\in S_2\end{array}$$

wobei β = 1.6 ein Skalierungsfaktor sein kann, η₁ = 1 × 10^-4 m ein Abstand zwischen einem primären Finger und dem Daumen sein kann, und η₂ = 3 × 10^-2 m kann ein Mindestabstand zwischen zwei primären Fingern sein kann, wenn beide primären Finger mit dem Daumen projiziert werden. Satz Beschreibung S₁ Vektoren, die von einem primären Finger (Zeige-, Mittel-, Ringfinger) ausgehen und auf den Daumen zeigen. S₂ Vektoren zwischen zwei primären Fingern, wenn beide primären Finger assoziierte Vektoren ∈ S1 haben, z.B. wenn beide primären Finger mit dem Daumen projiziert werden. In at least one embodiment, teleoperation of a robotic hand that is kinematically different than the human hand may require a module that can map the observed human wrists to the robotic wrists, which in some embodiments may be referred to as allegro wrists. 7 FIG. 7 shows an example of a pose 702 of a human hand and a robotic gripper 704 performing a corresponding pose, according to at least one embodiment. There can be many different approaches to kinematic tracking. For example, a module according to at least one embodiment may be used to match palm to fingertip and medial joint positions and directionality of the proximal phalanx and distal thumb phalanx. In at least one embodiment, the optimized mapping may be mapped using human depth images such that a depth network can take a depth image and output joint angles. In at least one embodiment, motion retargeting is also used. For example, a deep recurrent neural network can be trained unsupervised to track movements between skeletons. In at least one embodiment, the system uses the fingertip task space metric since distal areas may be of the highest priority in grasping and manipulation tasks as measured by their contact prevalence, level of innervation, and increased controllability for fine hand manipulation capability. In at least one embodiment, the joint axes and locations may differ between two hands, and therefore metrics that directly compare the joint angles between the two hands cannot be used. In at least one embodiment, both the distance and the direction between the fingertips are considered to detect and optimize the positioning of the fingertips. Specifically, in at least one exemplary embodiment, the cost function for the kinematic adjustment can be selected as follows: $$C\left(q_H,q_a\right)=\frac{1}{2}{\displaystyle {\sum }_{i=0}^{N}s\left({\mathrm{i.e}}_i\right){\Vert {\mathrm{right}}_i\left(q_a\right)-ƒ\left({\mathrm{i.e}}_i\right){\mathrm{right}}_i^{}\left(q_H\right)\Vert }^2}+g{\Vert q_a\Vert }^2$$

where q _h , q _a can be the angles of the model of the human hand and the Allegro hand, respectively, _ri ∈ R ³ can be the vector pointing from the origin of one coordinate system to another, expressed in the origin coordinate system (see 7 ). In addition, in at least one embodiment ${\mathrm{i.e}}_i=\Vert {\mathrm{right}}_i\left(q_H\right)\Vert \text{and}{\mathrm{right}}_i^{}\left(q_H\right)=\frac{{\mathrm{right}}_i\left(q_H\right)}{\Vert {\mathrm{right}}_i\left(q_H\right)\Vert }.$

The switching weight function s(d _i ) can be defined as follows: $$s\left({\mathrm{i.e}}_i\right)=\{\begin{array}{c}\mathrm{1,}{\mathrm{i.e}}_i>\mathrm{<figure-callout\; id="200"\; label="e"\; filenames="DE102020124285B4\_0020.png"\; state="\{\{state\}\}">e</figure-callout>}\\ \mathrm{200,}{\mathrm{i.e}}_i\le e\wedge {\mathrm{right}}_i\left(q_H\right)\in {\mathrm{<figure-callout\; id="1"\; label="S"\; filenames="DE102020124285B4\_0011.png,DE102020124285B4\_0013.png"\; state="\{\{state\}\}">S</figure-callout>}}_1\\ \mathrm{400,}{\mathrm{i.e}}_i\le e\wedge {\mathrm{right}}_i\left(q_H\right)\in {\mathrm{<figure-callout\; id="2"\; label="S"\; filenames="DE102020124285B4\_0011.png,DE102020124285B4\_0013.png"\; state="\{\{state\}\}">S</figure-callout>}}_2\end{array}$$

where S ₁ can be vectors emanating from a primary finger (index, middle, ring finger) and pointing to the thumb, and S ₂ can be m vectors between two primary fingers if both primary fingers have associated vectors ∈ S ₁ (e.g. both Primary fingers are projected with the thumb). In at least one embodiment, the distance function ƒ(d _i ) ∈ R is defined as: $$ƒ\left({\mathrm{i.e}}_i\right)=\{\begin{array}{c}\mathrm{\beta }{\mathrm{i.e}}_i,{\mathrm{i.e}}_i>e\\ {\mathrm{n}}_1,{\mathrm{i.e}}_i\le e\wedge {\mathrm{right}}_i\left(q_H\right)\in S_1\\ {\mathrm{<figure-callout\; id="2"\; label="n"\; filenames="DE102020124285B4\_0011.png,DE102020124285B4\_0013.png"\; state="\{\{state\}\}">n</figure-callout>}}_2,{\mathrm{i.e}}_i\le e\wedge {\mathrm{right}}_i\left(q_H\right)\in {\mathrm{<figure-callout\; id="2"\; label="S"\; filenames="DE102020124285B4\_0011.png,DE102020124285B4\_0013.png"\; state="\{\{state\}\}">S</figure-callout>}}_2\end{array}$$

where β = 1.6 can be a scaling factor, η ₁ = 1 × 10 ^-4 m can be a distance between a primary finger and the thumb, and η ₂ = 3 × 10 ^-2 m can be a minimum distance between two primary fingers, when both primary fingers are projected with the thumb. Sentence description p ₁ Vectors starting from a primary finger (index, middle, ring finger) pointing to the thumb. S ₂ Vectors between two primary fingers when both primary fingers have associated vectors ∈ S1, e.g. when both primary fingers are projected with the thumb.

In at least one embodiment, these projections ensure that the contacts between the primary fingers and the thumb are close together without collisions of the primary fingers in a precision grip. In at least one embodiment, this can be particularly useful for inaccuracies in visual finger tracking. In some examples, the vectors r _i can not only capture distance and direction from one task space to another, but their expression in local coordinates can further contain information about how the coordinate systems and thus the fingertips are oriented to each other. Thus, in at least one embodiment, the coordinate systems of the human hand model may have equivalent coordinate systems on the Allegro model with similarity in orientation and placement. In the 7 The vectors shown can form a minimal set that produces the desired retargeting behavior. In some embodiments, y=2.5×10 ^-3 may be a weight in regularizing the allegro angles to zero (equivalent to the fully open hand). In at least one embodiment, this term helps reduce redundancy in the solution and ensures that the hand never goes into unusual minima from which it is difficult to recover (eg, when fingers become embedded in the palm). In at least one embodiment, in 8th various mappings from the human hand 602-617 to an Allegro robotic hand 618-633 are shown as produced by the kinematic simulation. 8th shows a collection of poses of a human hand 802-817 and corresponding poses for a robot gripper 818-833.

In at least one embodiment, the above functionality is minimized in real-time using the Sequential Least Squares Quadratic Programming ("SLSQP") algorithm. In at least one embodiment, the routine is initiated with the Allegro joint angles set to zero, and any solution thereafter may be initiated with the previous solution. In at least one embodiment, the forward kinematic calculations are performed between the different coordinate systems of both the human hand model and the Allegro hand model. In at least one embodiment, a first-order low-pass filter is applied to the raw transmission joint angles to remove high-frequency noise associated with human hand tracking and to remove discrete Smooth events, such as the projection algorithm that induces step response changes in tracked angles.

In at least one embodiment, Riemannian Motion Rules ("RMPs") are real-time motion generation methods that compute acceleration fields from potential functional gradients and corresponding Riemannian metrics. In at least one embodiment, RMPs combine multi-priority Cartesian trajectory generation and collision avoidance behavior in a coherent frame of reference. In at least one embodiment, they are used to control the Cartesian pose of the Allegro palm using the observed pose of a human hand while avoiding arm or palm collisions with the table or operator using collision planes. In at least one embodiment, with these objectives in mind, the RMPs generated arm-to-joint target trajectories that were sent at 200 Hz to the arm's torque impedance controller. In at least one embodiment, the kinematically tracked Allegro angles were sent to the arm's torque impedance controller at 30 Hz. In at least one embodiment, a final detail of the calibration is to register the movements of the pose of a human hand with the robotic system. In at least one embodiment, this is performed using a transformation from the robot coordinate system to the camera coordinate system. In at least one embodiment, this transformation is computed using the initial view of the human hand and an assumed initial pose of the robot hand. In at least one embodiment, to facilitate the operator's spatial awareness, the desired initial hand pose of the operator is a fully open hand with the palm parallel to the table and fingers pointing forward. In at least one embodiment, the assumed initial position of the robot mimics this pose. In this manner, in at least one embodiment, the robot moves in the same direction as the operator's hand, allowing for intuitive spatial thinking.

Overall, the system can be used reliably to solve a variety of tasks with different levels of difficulty. In some examples, the ability to perform these tasks demonstrates that the system may have the dexterity, precision and power grips, multi-finger grip and non-grip manipulations, finger motions of the hand, and compound hand manipulations (e.g., two-finger grips with simultaneous manipulation with the remaining fingers).

9 FIG. 12 shows an example of a task in which a robot is controlled to remove paper from a folded wallet, according to at least one embodiment. In at least one embodiment, the operator must first open the wallet and take it to a designated convenient location to retrieve paper money. In at least one embodiment, the hand is able to hold the paper by pressing the fingers together.

10 12 shows an example of a task in which a robot is controlled to retrieve an object from a drawer, according to at least one embodiment. In at least one embodiment, this is a somewhat tedious task and requires dexterity in opening the drawer and holding the tea bag.

11 12 shows an example of a task in which a robot is controlled to open a vessel, according to at least one embodiment. In at least one embodiment, the task requires rotating the lid multiple times to open it while maintaining the contacts.

The table below describes examples of 15 different tasks of varying complexity, ranging from classic pick-and-place tasks to multi-step, long-term tasks. In at least one embodiment, each of these tasks is performed with five (5) consecutive attempts to avoid preferential selection, and the success rate is reported accordingly. In at least one embodiment, the attempt is considered a failure if the object falls out of the workspace volume. The last column represents the skills required for teleoperation, in which the hand changes state over time. task description Required Skills Pick and place Picking up an object on the table and placing it in a red bowl. grab, let go • Foam block • Pringles can • Spam box Stacking Blocks Stacking three blocks on top of each other. precise gripping, precise • Large (L) (6.3cm) Let go • Medium (M) (3.8cm) • Small (S) (2.3 cm) Pouring Pearls Pouring beads from a beaker into a bowl. grab, pour opening a vessel Opening a peanut jar and placing the lid on the table. Move fingers, grab, let go moving a block Picking up a block, rotating the block 180 degrees by hand, and putting it down again. Grasping, hand manipulations, releasing container Opening a plastic container, pulling out and opening a cardboard box. Twisting, pulling, pushing, grasping, hand manipulations Insertion of cups Insertion of concentric cups into each other. grab, let go tea drawer Opening a tea drawer, taking out a single tea bag and placing it on the table, closing the tea drawer. precise gripping, pulling, pushing, releasing pushing a card Slide a card along the box and pick it up with two fingers. Pushing, precise gripping, letting go wallet Opening the wallet and pulling out paper money. precise gripping, pulling, pushing, hand manipulations turning over a box Rotate the crate 90 degrees and place it on the given target. Squeeze, grab, release

In one experiment, operators went through a warm-up phase in which they attempted to complete the task in three to five (3-5) non-consecutive trials. Later, five consecutive test attempts were performed by each operator for each task to avoid preferential selection of results, and operators were ranked based on their performance. The performance metrics for these tasks include mean completion time (“CT”) and success rate, which capture the speed and reliability of the teleoperation system. The system has been tested with two operators and the performance metrics are in 12 and 13 listed.

In at least one embodiment, the system can be reliably used to solve a variety of tasks of varying difficulty. In at least one embodiment, differences in mean CT across tasks indicate the effects of task complexity and time horizon scale. In at least one embodiment, discrepancies in average CT between operators per task indicate that there is a dependency on operator behavior. In at least one embodiment, the effects include fatigue, exercise, creativity, and motivation. In at least one embodiment, the ability to perform these tasks demonstrates that the teleoperation system has the dexterity to perform precision and power grips, multi-finger grip and non-grip manipulation, in-hand finger manipulation with the fingers, and compound in-hand manipulation ( eg grasping with two fingers while manipulating with the other fingers). In at least one embodiment, it should be noted that certain tasks, eg, container and wallet, take a particularly long time to be teleoperated, due largely to the fact that these tasks are multi-stage. On the other hand, the task that the on taking small cubes can be particularly challenging, since the behavior of releasing the grips on these objects can be unpredictable with the projection scheme used in kinematic reenactment. Nonetheless, such an extensive demonstration of dexterity, implemented solely through mere human hand observation, is empirical evidence that the approach and architecture works well here, according to various embodiments. In at least one embodiment, an important aspect to emphasize is that although the full teleoperation process may not be perfect for a given task (e.g. the operator may lose an item in hand but retrieve it to complete the task) However, the data collected is just as valuable in helping the robot learn to recover from mistakes. In at least one embodiment, the data may be considered game data useful for learning long-term planning. In at least one embodiment, discrete events such as intermittent finger-to-object contacts can be observed in the tactile signals. In at least one embodiment, ripples in these state-action signals reveal the rich, complex behavior induced in the system by this adjustment being made. In at least one embodiment, force estimates can also be made. In at least one embodiment, this data can now be generated on-demand for a particular task, with the hope that functional sensorimotor patterns can be collected and imparted to the system in an autonomous setting.

In at least one embodiment, the system may enable a highly mobile hand-arm system to find a motorized solution to a variety of manipulation tasks by translating observed human hand and finger movements into robotic arm and finger movements. In at least one embodiment, multiple tasks, such as pulling paper money from a wallet and opening a cardboard box in a plastic container, can be so complex that developing a robotic solution by hand or directly applying learning methods is likely unsolvable. Solving these and other tasks by the executed robot can allow these solutions to be generated for many demonstrations as needed. In addition, creating these solutions on the system itself can read, access, and store the various tactile signals in the robot's fingertips, various commanded and measured joint position and velocity signals through the hand and arm, various torque commands throughout the system, and any camera feeds associated with the system. In at least one embodiment, this rich source of data may be used in conjunction with task demonstrations to solve complex, multi-step, long-term tasks.

The techniques described herein are, in at least one embodiment, a viable, low-cost solution for teleoperation of a high autonomy (DoA) robotic system. In at least one embodiment, the operator's observable work volume could be increased to enable tasks covering longer distances with better RGB-D cameras. In at least one embodiment, in kinematic reenactment, the projection schemes enable successful manipulation of small objects, but may interfere with finger movement tasks and timely release of grips on small objects. In at least one embodiment, this problem is fully addressed with hand tracking that can precisely resolve situations where human hand fingertips are in contact. In at least one embodiment, human hand tracking is further enhanced through improved robustness in the size and shape of the operator's hand. The lack of tactile feedback can make it difficult to perform precision tasks. In at least one embodiment, the incorporation of autonomous control functions reduces some of the control burden on the operator. In at least one embodiment, system latency is reduced and the responsiveness of the RMP motion generator is tuned for faster responses. In at least one embodiment, high-precision tasks, such as fitting slip-fit pin-in-hole insertions, present a challenge Significantly reduced visibility of the small parts and insertion points.

In at least one embodiment, the input to the system is a point cloud of the human demonstrator's hand. In at least one embodiment, a neural network, eg, a neural network based on the PointNet++ neural network, associates the point cloud with an estimate of the hand pose relative to the camera and the joint angles of the hand. In at least one embodiment, these estimates are combined with a moving hand model and the original Then pass the point cloud to DART, which performs the tracking by refining the neural network estimates. Finally, to perform kinematic re-enactment, an optimization problem is solved that finds the Allegro hand joint angles that result in fingertip poses that approximate those of the human hand, according to one embodiment.

In at least one embodiment, gloved hand tracking is performed by key point detection with neural networks. In at least one embodiment, the user wears a black glove with colored dots and moves the hand on a table covered with black cloth, i.e. the scene is designed to support hand tracking. Because in at least one embodiment the colors are different and the majority of the background is black, annotations are created for these colored points using OpenCV HSV color thresholds. In at least one embodiment, HSV thresholds vary with the time of day, and therefore data is collected over days to create a large data set of 50k images. At least one embodiment uses a neural network to fit this data, making the whole process robust to light changes and bad annotation, and avoiding the burden on the user of finding the appropriate thresholds at test time. In at least one embodiment, the network, called GloveNet, uses four (4) layers of ResNet-50 with a spatial softmax at the end to regress to the 2D positions of the fingertips. In at least one embodiment, anti-aliasing ResNet-50 is used for accurate and consistent predictions. The different stages of the pipeline are explained below.

At least one embodiment uses imgaug and applies various augmentations to data during training. At least one embodiment focuses on the hand moving on the table. For each trained image, at least one embodiment sets the color values of all pixels having a depth above a threshold to zero. At the time of training, at least one embodiment either fills in these zero values with one of the glove's colors or leaves the image unchanged based on a random number generator. At least one embodiment also replaces these zero values with random noise based on some probability. In at least one embodiment, the network learns to ignore background colors that resemble the colors on the glove.

At least one embodiment obtains confidence for each predicted fingertip position using test time augmentation ("TTA"). At least one embodiment creates new images by shifting the original image by random shifts and passing them all through the network in a batch. At least one embodiment then subtracts the applied random shifts from the predicted positions for each image to bring them into the frame of reference of the original image and averages them to obtain the mean and standard deviation. At least one embodiment uses standard deviation as a confidence measure. At least one embodiment also uses them to clean up the noisy ground truth data.

At test time, at least one embodiment generates four randomly shifted images per camera image and a total of 16 images from all four cameras. In at least one embodiment, the predicted fingertip positions and their confidence measures are calculated and those with low confidence are discarded. At least one embodiment of the confident ones calculates the Euclidean distances di between them and the previous fingertip positions and converts them into probabilities p _i using _Softmax : $${\text{p}}_i=\frac{\text{ex}\left(-a\left({\text{i.e}}_i-{\text{at least}}_i{\text{i.e}}_i\right)\right)}{{\displaystyle {\sum }_{i=0}^{\text{N}}\text{ex}\left(-a\left({\text{i.e}}_i-{\text{at least}}_i{\text{i.e}}_i\right)\right)}}$$

conv3 40×30 $\left[\begin{array}{c}1\times \mathrm{1,}128\\ 3\times \mathrm{3,}128\\ 1\times \mathrm{1,}512\end{array}\right]\times 4$

conv transpose 80×60 3 × 3, 8 spatial_softmax 8×2 β = 50 At least one embodiment shifts the predicted locations having a probability p _i >0.2 into a rolling buffer and calculates the geometric median to get the final predicted fingertip location in 3D. The hyperparameter is α = 500. layer name output size parameter input 320×240 - conv1 160×120 7×7, 64, stride 2 3 × 3 max pool, stride 2 conv2 80×60 $\left[\begin{array}{c}1\times \mathrm{1,}64\\ 3\times \mathrm{3,}64\\ 1\times \mathrm{1,}256\end{array}\right]\times 3$

conv3 40×30 $\left[\begin{array}{c}1\times \mathrm{1,}128\\ 3\times \mathrm{3,}128\\ 1\times \mathrm{1,}512\end{array}\right]\times 4$

conv transpose 80×60 3×3, 8 spatial_softmax 8×2 β = 50

In at least one embodiment, the predictions for the backhand points were stable, but the predictions for the fingertip points were quite inconsistent over time. Since the annotations were generated in at least one embodiment by calculating the center of mass (“CoM”) of the segmented point using the HSV color threshold in OpenCV, the CoM of the fingertip was somewhat inconsistent across frames due to occlusions . Therefore, at least one embodiment relies only on the hand pose estimate provided by the points on the back of the palm. layer name MLP features radius no. Points SAT ₁ [13, 64, 64, 128] 0.2 2048 SAT ₂ [128, 128, 128,256] 0.4 1024 SA ₃ [256, 128, 128, 256] 0.8 512 SAT ₄ [256, 128, 128, 256] 1.2 256 FP ₄ [256+256, 256, 256] 512 FP ₃ [256+256, 256, 256] 1024 FP ₂ [256+128, 256, 256] 2048 FP ₁ [256+3, 256, 256] 8192

In at least one embodiment, the architecture consists of four (4) sentence abstraction layers, SA _i , and four (4) feature propagation layers, FP _j . In at least one embodiment, the set abstraction layer undersamples the points while the feature expansion layer interpolates features at a higher resolution.

In at least one embodiment, a set abstraction level takes N×(d+C) as input N points with d-dim coordinates and C-dim point feature. In at least one embodiment, a tensor of N'×(d+C') is output, where N' subsampled points with d-dim coordinates and new C'-dim feature vectors summarize a local context.

In at least one embodiment, in a feature propagation plane, point features are propagated from N _i x (d + C) points to N _il points, where N _il and N _i (with N _i ≤ N _i -l) are the point set size of input and output of the set abstraction level i are. In at least one embodiment, this is accomplished by interpolating feature values of N _i points at the coordinates of the N _il points. In at least one embodiment, the interpolated features at _nil points are then concatenated with skipped linked point features from the set level of abstraction.

Name der Schicht Parameter mlp layer1 [256, 256, 256] voting [256, 23×3] seg [256, 23] cls [256, 2] In at least one embodiment, the framework for estimating the hand pose is an architecture that provides an enclosure, f, of size NxC. Using different MLPs, this inclusion is mapped to the appropriate desired outputs. $$\begin{array}{c}\mathrm{e.g}=\text{mlp}\_\text{layers}1\left(ƒ\right)\\ \\ {\delta }_{xy\mathrm{e.g}}=\text{voting}\left(\mathrm{e.g}\right)\\ \\ \text{cords}={\text{input}}_{\text{xyz}}+{\delta }_{xy\mathrm{e.g}}\\ \\ {\text{Joint Mask}}_{xy\mathrm{e.g}}=\text{sigmoid}\left(\text{bless}\left(\mathrm{e.g}\right)\right)\\ \\ {\text{hand seg}}_{xy\mathrm{e.g}}=\text{cls}\left(\mathrm{e.g}\right)\\ \\ {\text{HandSegProb}}_{xy\mathrm{e.g}}=\text{sigmoid}\left({\text{hand seg}}_{xy\mathrm{e.g}}\right)\\ \\ \text{weights}={\text{HandSegProb}}_{xy\mathrm{e.g}}{\text{Joint Mask}}_{xy\mathrm{e.g}}\\ \text{key points}=\frac{{\displaystyle \sum \text{weights}}\cdot \text{coords}}{{\displaystyle \sum weiGHts}}\end{array}$$

layer name parameter mlp layer1 [256, 256, 256] voting [256, 23×3] bless [256, 23] cls [256, 2]

In at least one embodiment, the voting layer maintains the relative positions, δ _xyz , of the 23 key points with respect to each point. The seg layer preserves the masks for each keypoint, ie the neighborhood of points that contribute to a keypoint's location. In at least one embodiment, the HandSeg layer segments the hand from the background. In at least one embodiment, Euclidean losses are used for both voting and keypoints, while sigmoidal cross entropy is used for HandSeg.

In at least one embodiment, the 23×3 key point locations are unrolled into a 69 dimensional vector before being fed into the JointNet, which provides a 20 dimensional vector of joint angles. In at least one embodiment of all the hand-designed architectures tried, this particular architecture proved to be an effective compromise between accuracy and efficiency. layer name parameter linear 1 69×128 linear2 128×256 linear3 256×20

Execution times for the five (5) consecutive attempts for each of the tasks are shown according to one embodiment. In at least one embodiment, the failed attempt is labeled F. In at least one embodiment, in most trials, the operator used only three to four (3-4) warm-up training trials. In at least one embodiment, these five (5) consecutive attempts allow both testing the ability to perform a particular task without fatigue and demonstrating that the tracking is functioning without error. In at least one embodiment, performance may vary depending on the operator and the condition of the day, but experiments have shown that performance is generally quite similar. tasks operator Execution times for 5 consecutive attempts Average hours Picking up and putting down: Block operator 1 19 16 17 11 18 16 3.11 Picking up and putting down: Klotz operator 2 22 22 19 16 14 19 3.57 Pickup and Drop: Spam operator 1 28 14 15 16 23 19 6.05 Pickup and Drop: Spam operator 2 23 23 28 29 20 25 3.78 slide card operator 1 27 26 32 38 35 32 5.12 slide card operator 2 18 12 18 15 17 16 2.54 Pick up and drop: Pringles operator 1 50 18 20 29 18 27 13.6 Pick up and drop: Pringles operator 2 25 53 29 36 63 41 16.22 moving a block operator 1 48 67 f f 58 58 9.50 moving a block operator 2 37 44 f f 28 36 8.02 pour operator 1 38 42 32 f 28 35 6.21 pour operator 2 73 56 62 50 57 60 8.61 turning over a box operator 1 51 39 45 f 77 53 16.73 turning over a box operator 2 174 26 90 30 67 77 60.18 Blocks (L) operator 1 41 49 54 45 165 71 52.87 Blocks (L) operator 2 53 93 79 43 61 66 20.12 peanut jar operator 1 89 66 79 77 75 77 8.25 peanut jar operator 2 68 105 84 87 57 80 18.45 Insertion of cups operator 1 64 94 70 f 71 75 13.2 Insertion of cups operator 2 125 f 124 124 112 121 6.18 tea operator 1 48 115 170 58 154 109 55.00 tea operator 2 54 48 99 105 213 104 66.22 Blocks (M) operator 1 179 278 64 80 298 180 108.3 7 Blocks (M) operator 2 99 48 82 75 152 91 38.63 wallet operator 1 105 66 195 96 63 105 61.82 wallet operator 2 321 92 328 100 218 212 114.3 6 Blocks (S) operator 1 136 371 169 f 484 290 165.8 8 Blocks (S) operator 2 113 89 69 117 67 91 23.57 container operator 1 442 271 375 297 405 358 72.18 container operator 2 189 212 258 238 243 228 27.39

In at least one embodiment, neural network tracking produces unsatisfactory results - it does not provide the level of accuracy equivalent to on-line sequential least squares optimization. In at least one embodiment, the projection threshold used in tracking may require some adjustment when detecting small objects, and therefore it becomes cumbersome to train a neural network for a new random task.

In at least one embodiment, a hand tracking system relies on a combination of model-based and model-free tracking. In at least one embodiment, the model based tracking systems to be more accurate as they optimize online for the observations fed into the model. However, since the optimization tends to be highly non-linear in at least one embodiment, they also need good initialization to find a meaningful solution. In at least one embodiment, at least one embodiment uses a model-free system that can provide good initialization. In at least one embodiment, a model-free system is a neural network that is trained on the data generated by a model-based system.

In at least one embodiment, model-based tracking is used in DART [27] and data is collected in the areas where it works reliably and this is repeated to cover a wide range of poses. In at least one embodiment, the performance of DART can be stochastic: it can sometimes work reliably for the same motion and other times fail catastrophically due to false local minima in the optimization due to the input point cloud. However, in at least one embodiment, if data is collected for the scenarios in which it performs reasonably well, a neural network can be used to adjust that data and ensure that it can provide good initialization for DART and prevent it from doing so in the future falls into the wrong local minima. In at least one embodiment, this is due to the fact that neural networks can generalize slightly outside of the training set. In at least one embodiment, this method of data acquisition and neural network adjustment is performed repeatedly and improves the performance of DART such that tracking works without dropouts over a long period of time. In at least one embodiment, a two-tier PointNet++ based architecture is trained using the annotations generated by DART and allows, in at least one embodiment, to make tracking both robust and accurate by providing good initialization.

14 Figure 12 shows an example of a process that, as a result of one embodiment, a computer system directs a robotic arm to perform a task by emulating the movement of a human hand. In at least one embodiment, the computer system is a computer system as in 21-23 shown. In at least one embodiment, the computer system includes a processor and a memory, the memory storing executable instructions that, when executed by the processor, cause the system to execute the 14 performs the operations shown and described in the description below. In at least one embodiment, the processor is a plurality of processors or a specialized processor such as a graphics processing unit (GPU), as in 15-43 shown and described below.

In at least one embodiment, in step 1402, the computer system obtains an image of a human hand from a depth camera. In various embodiments, the depth camera may be an RGB depth camera, a binocular camera, a radar or laser based imager, or a medical imaging system such as an MRI, X-ray, computed tomography, or computed axial tomography scanner. In at least one embodiment, in step 1404, the computer system creates a point cloud of the human hand from the image. The point cloud provides three-dimensional data describing the hand from which a pose can be determined. In at least one embodiment, in step 1406, the computer system analyzes the point cloud to determine a joint structure and joint angles for the hand. In various exemplary embodiments, this includes the pose of the hand in combination with the position of the hand in space.

In at least one embodiment, in step 1408, the computer system translates the pose of a human hand into a corresponding pose of a robot gripper. In at least one embodiment, the robotic gripper may be a flexible robotic hand similar to that of a human hand. The articulated robot hand may have fewer or more fingers than a human hand, and each finger of the articulated robot hand may have one or more movable segments. In at least one embodiment, the pose of a human hand includes a set of joint segments and joint angles and an overall position of the human hand, and the pose of the robot gripper is determined by duplicating the joint angles of the pose of a human hand using the robot hand. In at least one embodiment, the robotic hand has fewer fingers than the human hand, and poses associated with a subset of the human fingers are used to determine the corresponding pose of the robotic gripper. In at least one embodiment, the robotic gripper may be larger or smaller than the human hand, and a scaling factor is applied to the pose of the human hand in determining a corresponding pose of the robotic gripper. In at least one embodiment, the pose comprises a human Hand a position of the hand, and a corresponding position for the robot hand is determined by translating a coordinate system from a coordinate system for the human operator to that of the robot. In some examples, a scaling factor can be applied to the location aspect of the pose. For example, six inches of human hand movement may correspond to six feet of robot gripper movement.

In at least one embodiment, in step 1410, the computer system determines a set of motions for repositioning the robot gripper from a current position to the corresponding pose determined in step 1408. In at least one embodiment, the system determines a path of motion for the robot hand that will bring the robot hand from its current pose to the pose specified in step 1408 . In at least one embodiment, these motions are determined according to Riemann's motion rules. In at least one embodiment, in step 1412, the computer system causes the robot to perform the set of movements specified in step 1410. As a result, in at least one embodiment, the robot gripper assumes the pose that corresponds to the pose of a human hand. In various embodiments, using these techniques, an operator is able to intuitively control the movement of the robot through movements of the operator's hand. In at least one embodiment, no glove or worn apparatus is required.

15A 15 shows the inference and/or training logic 1515 used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with below 15A and or 15B provided.

In at least one embodiment, inference and/or training logic 1515 may include, without limitation, code and/or data storage 1501 for storing forward and/or output weight and/or input/output data and/or other neuron or layer configuration parameters of a neural network trained and/or used for inference, according to aspects of one or more embodiments. In at least one embodiment, training logic 1515 may include or be coupled to code and/or data storage 1501 to store graph code or other software for controlling the timing and/or order in which weight and/or other parameter information is loaded to configure logic, including integer and/or floating point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code, such as graph code, loads weight or other parametric information into processor ALUs based on a neural network architecture to which the code conforms. In at least one embodiment, the code and/or data store 1501 stores weight parameters and/or input/output data of each neural network layer used during forward propagation of input/output data and/or weight parameters during training and/or inference using trained or used by aspects of one or more embodiments. In at least one embodiment, any portion of code and/or data memory 1501 may be aggregated with other on-chip or off-chip data memory, including a processor's L1, L2, or L3 cache or system memory.

In at least one embodiment, any portion of code and/or data storage 1501 may be implemented internally or externally to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and/or code and/or data memory 1501 may be cache memory, dynamic randomly addressable memory (“DRAM”), static randomly addressable memory ( static randomly addressable memory (“SRAM”), non-volatile memory (e.g. flash memory) or other memory. In at least one embodiment, the choice of whether the code and/or code and/or data memory 1501 is, for example, on-processor or off-processor, or consists of DRAM, SRAM, Flash, or some other type of memory, may depend on the memory available on-chip versus that off-chip memory, latency requirements of the training and/or inference functions performed, the stack size of the data used in inferring and/or training a neural network, or a combination of these factors.

In at least one embodiment, the inference and/or training logic 1515 may include, without limitation, code and/or data storage 1505 to store backward and/or output weights and/or input/output data corresponding to neurons or layers of a neural network trained in aspects of one or more embodiments for inference and/or ver be turned. In at least one embodiment, the code and/or data store 1505 stores weight parameters and/or input/output data of each neural network layer used during back propagation of input/output data and/or weight parameters during training and/or inference using trained or used by aspects of one or more embodiments. In at least one embodiment, training logic 1515 may include or be coupled to code and/or data storage 1505 to store graph code or other software for controlling the timing and/or order in which weight and/or other parameter information is loaded to configure logic, including integer and/or floating point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code, such as graph code, loads weight or other parametric information into processor ALUs based on a neural network architecture to which the code conforms. In at least one embodiment, any portion of code and/or data memory 1505 may be aggregated with other on-chip or off-chip data memory, including a processor's L1, L2, or L3 cache memory or system memory. In at least one embodiment, any portion of code and/or data storage 1505 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and/or data storage 1505 may be cache memory, DRAM, SRAM, non-volatile memory (eg, flash memory), or other memory. In at least one embodiment, the choice of whether the code and/or data storage 1505 is, for example, on-processor or off-processor, or consists of DRAM, SRAM, Flash, or other memory type, may depend on the available memory on-chip or off-chip, latency requirements of the training and/or inference functions performed, the batch size of the data used in the inference and/or training of a neural network, or a combination of these factors.

In at least one embodiment, code and/or data storage 1501 and code and/or data storage 1505 may be separate storage structures. In at least one embodiment, code and/or data storage 1501 and code and/or data storage 1505 may be the same storage structure. In at least one embodiment, code and/or data storage 1501 and code and/or data storage 1505 may be partially the same memory structure and partially separate memory structures. In at least one embodiment, any portion of code and/or data memory 1501 and code and/or data memory 1505 may be aggregated with other on-chip or off-chip data memory, including L1, L2, or L3 cache -Memory or system memory of a processor.

In at least one embodiment, the inference and/or training logic 1515 may include, without limitation, one or more arithmetic logic unit(s) ("ALU(s)") 1510, including integer and/or floating point units, to provide logical and/or perform mathematical operations based at least in part on or specified by training and/or inference code (e.g. graph code) the result of which can produce activations stored in an activation memory 1520 (e.g. output values from layers or neurons within a neural network) which are functions of input/output and/or weight parameter data stored in code and/or data memory 1501 and/or code and/or data memory 1505. In at least one embodiment, activations stored in an activation memory 1520 are generated according to linear algebraic and/or matrix-based mathematics performed by ALU(s) 1510 in response to execution of instructions or other code, stored in code and/or data memory 1505 and/or data 1501 stored weight values as operands together with other values, such as bias values, gradient information, pulse values or other parameters or hyperparameters, which can be stored arbitrarily or completely in code and/or data memory 1505 or code and/or or data memory 1501 or another memory can be stored on or off-chip.

In at least one embodiment, the ALU(s) 1510 reside within one or more processors or other hardware logic device or circuitry, while in another embodiment, the ALU(s) 1510 reside external to a processor or other hardware logic device or circuitry who can use it (e.g. a co-processor). In at least one embodiment, the ALU(s) 1510 may be contained within the execution units of a processor or otherwise in a group of ALUs to which the execution units of a processor are either within the same processor or distributed among different processors of different types (e.g., central processing units, graphics processing units, units with fixed functionality, etc.). In at least one embodiment, the data memory 1501, the code and/or data memory 1505 and activation memory 1520 reside on the same processor or other hardware logic device or circuitry, while in another embodiment they reside in different processors or other hardware logic devices or circuitry, or in a combination of the same and different processors or other hardware logic devices or circuits may be located. In at least one embodiment, any portion of activation memory 1520 may be aggregated with other on-chip or off-chip data memory, including a processor's L1, L2, or L3 cache memory or system memory. Additionally, the inference and/or training code may be stored with other code accessible by a processor or other hardware logic or circuitry and using retrieval, decoding, scheduling, execution, shutdown and/or other logic circuits of a processor.

In at least one embodiment, enable memory 1520 may be cache memory, DRAM, SRAM, non-volatile memory (eg, flash memory), or other memory. In at least one embodiment, activation memory 1520 may reside in whole or in part internally or externally to one or more processors or other logic circuits. In at least one embodiment, the choice of whether the activation memory 1520 is, for example, on-processor or off-processor, or consists of DRAM, SRAM, Flash, or another type of memory, may depend on the available on-chip or off-chip memory, the latency requirements of the training and /or inference functions, the batch size of the data used in inference and/or training a neural network, or a combination of these factors. In at least one embodiment, the 15A The illustrated inference and/or training logic 1515 may be used in conjunction with an application specific integrated circuit ("ASIC"), such as Google's Tensorflow® Processing Unit, a Graphcore™ Inference Processing Unit (IPU), or a Nervana® (e.g., " Lake Crest") processor from Intel Corp. In at least one embodiment, the 15A The illustrated inference and/or training logic 1515 may be used in conjunction with central processing unit ("CPU") hardware, graphics processing unit ("GPU") hardware, or other hardware such as field programmable gate arrays ("FPGAs") .

15B 15 shows the inference and/or training logic 1515 according to at least one embodiment. In at least one embodiment, the inference and/or training logic 1515 may include, without limitation, hardware logic in which computational resources are dedicated or otherwise used solely in connection with weight values or other information corresponding to one or more layers of neurons within a neural network . In at least one embodiment, the 15B illustrated inference and/or training logic 1515 can be used in conjunction with an application-specific integrated circuit (ASIC), such as Google's Tensorflow® Processing Unit, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., "Lake Crest ") Processor by Intel Corp. In at least one embodiment, the 15B The illustrated inference and/or training logic 1515 may be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware such as field programmable gate arrays (FPGAs). In at least one embodiment, the inference and/or training logic 1515 includes, without limitation, code and/or data storage 1501 and code and/or data storage 1505 operable to store code (e.g., graph code), weight values, and/or other information, including default values, gradient information, momentum values and/or other parameter or hyperparameter information. In at least one embodiment described in 15B As shown, both code and/or data storage 1501 and code and/or data storage 1505 are associated with a dedicated computing resource, such as computing hardware 1502 and computing hardware 1506, respectively. In at least one embodiment, both computing hardware include 1502 and the computing hardware 1506 one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in the code and/or data memory 1501 or in the code and/or data memory 1505 and the result thereof is stored in activation memory 1520.

In at least one embodiment, each of the code and/or data stores 1501 and 1505 and the corresponding computational hardware 1502 and 1506, respectively, corresponds to different layers of a neural network such that the activation of the code resulting from a "storage/computation pair 1501/1502". - and/or data memory 1501 and computational hardware 1502 is provided as input to the next "memory/computation pair 1505/1506" of code and/or data memory 1505 and computational hardware 1506 to conceptualize the organization of a neural network to reflect. In at least one embodiment, each of the storage/computation pairs 1501/1502 and 1505/1506 correspond to more than one layer of a neural network. In at least one embodiment, additional storage/computation pairs (not shown) may be included in inference and/or training logic 1515 subsequent to or in parallel with storage/computation pairs 1501/1502 and 1505/1506.

16 shows the training and use of a deep neural network, according to at least one embodiment. In at least one embodiment, the untrained neural network 91606 is trained using a training data set 1602 . In at least one embodiment, the training framework (framework) 1604 is a PyTorch framework, while in other embodiments the training framework 1604 is a Tensorflow, Boost, Caffe, Microsoft Cognitive Toolkit/CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training framework. In at least one embodiment, the training framework 1604 trains an untrained neural network 1606 and enables it to be trained using the processing resources described herein to generate a trained neural network 1608 . In at least one embodiment, the weights may be chosen randomly or by pre-training using a deep belief network. In at least one embodiment, the training may be either supervised, partially supervised, or unsupervised.

In at least one embodiment, the untrained neural network 1606 is trained using supervised learning, where the training data set 1602 includes an input paired with a desired output for an input, or where the training data set 1602 includes an input with a known output and an output of the neural network 1606 is manually graded. In at least one embodiment, the untrained neural network 1606 is trained under supervision, processes inputs from the training data set 1602, and compares the resulting outputs to a set of expected or desired outputs. Errors are then propagated back through the untrained neural network 1606 in at least one embodiment. In at least one embodiment, the training framework 1604 adjusts weights that control the untrained neural network 1606 . In at least one embodiment, the training framework 1604 includes tools to monitor how well the untrained neural network 1606 is converging to a model, such as the trained neural network 1608, that is appropriate given known input data such as new data 1612 to generate correct answers as in the result 1614. In at least one embodiment, the training framework 1604 repeatedly trains the untrained neural network 1606 while adjusting the weights to refine an output of the untrained neural network 1606 using a loss function and an adjustment algorithm such as stochastic gradient descent. In at least one embodiment, the training framework 1604 trains the untrained neural network 1606 until the untrained neural network 1606 achieves a desired accuracy. In at least one embodiment, the trained neural network 1608 can then be used to implement any number of machine learning operations.

In at least one embodiment, the untrained neural network 1606 is trained using unsupervised learning, where the untrained neural network 1606 attempts to train itself using untagged data. In at least one embodiment, unsupervised learning dataset 1602 includes input data with no associated output data or ground truth data. In at least one embodiment, the untrained neural network 1606 can learn groupings within the training data set 1602 and determine how individual inputs relate to the untrained data set 1602 . In at least one embodiment, unsupervised training can produce a self-organizing map that is a type of trained neural network 1608 capable of performing operations useful in reducing the dimensionality of new data 1612 . In at least one embodiment, unsupervised training may also be used to perform anomaly detection, which allows for the identification of data points in a new data set 1612 that deviate from the new data set 1612's normal patterns.

In at least one embodiment, semi-supervised learning, a technique in which the data set 1602 under training comprises a mixture of tagged and untagged data, may be used. In at least one embodiment, the training framework 1604 may be used to perform incremental learning, e.g., through delegated learning techniques. In at least one embodiment, incremental learning allows the trained neural network 1608 to adapt to new data 1612 without forgetting the knowledge that entered the network during initial training.

17 17 shows an example of the data center 1700 in which at least one embodiment may be employed. In at least one embodiment, the data center 1700 includes a data center infrastructure layer 1710, a framework layer 1720, a software layer 1730, and an application layer 1740.

In at least one embodiment, as in 17 As illustrated, the data center infrastructure layer 1710 may include a resource orchestrator 1712, clustered computing resources 1714, and node computing resources ("CRs") 1716(1)-1716(N), where "N" is any whole, represents a positive integer. In at least one embodiment, node CRs 1716(1)-1716(N) may include any number of central processing units ("CPUs") or other processors (including accelerators, field programmable gate arrays (FPGAs), GPUs, etc.), storage devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input/output ("NW I/O") devices, network switches, virtual machines ("VMs"), power and cooling modules, etc. In at least one embodiment, one or more node CRs from among node CRs 1716(1)-1716(N) may be a server with one or more of the above computing resources.

In at least one embodiment, clustered compute resources 1714 may include separate clusters of node C.R.s housed in one or more racks (not shown) or many racks housed in data centers in different geographic locations (also not shown). Separate groupings of node C.R.s within grouped compute resources 1714 may include grouped compute, network, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, multiple node C.R.s including CPUs or processors may be grouped within one or more cabinets to provide computing resources in support of one or more workloads. In at least one embodiment, one or more cabinets may also include any number of power modules, cooling modules, and network switches in any combination.

In at least one embodiment, resource orchestrator 1712 may configure or otherwise control one or more node C.R.s 1716(1)-1716(N) and/or clustered computing resources 1714. In at least one embodiment, resource orchestrator 1712 may include a software design infrastructure ("SDI") manager for data center 1700 . In at least one embodiment, the resource orchestrator may include hardware, software, or a combination thereof.

In at least one embodiment, as in 17 shown, the framework layer 1720 includes a task scheduler 1732, a configuration manager 1734, a resource manager 1736 and a distributed file system 1738. In at least one embodiment, the framework layer 1720 can be a framework for supporting the software 1732 of the software layer 1730 and/or one or more application(s) 1742 of the application layer 1740. In at least one embodiment, software 1732 or application(s) 1742 may each include web-based service software or applications such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 1720 may be some type of free and open source software web application framework such as Apache Spark™ (hereinafter “Spark”), which implements the distributed file system 1738 for processing large amounts of data (e.g., “big data”). can use, but is not limited to. In at least one embodiment, the task scheduler may include a Spark driver to facilitate scheduling of workloads supported by different layers of the data center 1700. In at least one embodiment, the configuration manager 1734 may be able to configure various layers such as the software layer 1730 and the framework layer 1720 including Spark and the distributed file system 1738 to support processing large amounts of data. In at least one embodiment, resource manager 1736 may be capable of managing clustered or grouped computing resources associated with or allocated to support distributed file system 1738 and task scheduler 1732 . In at least one embodiment, clustered or clustered computing resources may include clustered computing resource 1714 at data center infrastructure layer 1710 . In at least one embodiment, the resource manager 1736 may coordinate with the resource orchestrator 1712 to manage these allocated or allocated computing resources.

In at least one embodiment, software 1732 included in software layer 1730 may include software implemented by at least portions of node CRs 1716(1)-1716(N), clustered computing resources 1714, and/or distributed file system 1738 of the framework Layer 1720 is used. One or more types of software may include, but are not limited to, Internet web site search software, e-mail virus scanning software, database software, and video streaming software.

In at least one embodiment, the application(s) 1742 contained in the application layer 1740 may include one or more types of applications hosted by at least portions of the node C.R.s 1716(1)-1716(N), clustered computing resources 1714 and/or the distributed file system 1738 of the framework layer 1720. One or more types of applications can include, but are not limited to, any number of genomics applications, cognitive computation, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc .) or other machine learning applications used in connection with one or more embodiments.

In at least one embodiment, each of configuration manager 1734, resource manager 1736, and resource orchestrator 1712 may perform any number and type of self-modifying actions based on any amount and type of data collected in any technically appropriate manner. In at least one embodiment, self-modifying actions may relieve a data center 1700 operator from potentially making poor configuration decisions and potentially avoiding underutilized and/or underperforming parts of a data center.

In at least one embodiment, data center 1700 may include tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, according to at least one embodiment, a machine learning model may be trained by computing weight parameters according to a neural network architecture using the software and computational resources described above with respect to data center 1700 . In at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1700 using weight parameters defined by one or more described herein training techniques are calculated.

In at least one embodiment, a data center may use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware to perform training and/or inference using the resources described above. Additionally, one or more of the software and/or hardware resources described above may be configured as a service to enable users to train or perform inference of information, such as image recognition, speech recognition, or other artificial intelligence services.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, the inference and/or training logic 1515 in the system 17 for inferring or predicting operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

18A 18 shows an example of an autonomous vehicle 1800, according to at least one embodiment. In at least one embodiment, autonomous vehicle 1800 (alternatively referred to herein as “vehicle 1800”) may be, without limitation, a passenger vehicle, such as an automobile, truck, bus, and/or other type of vehicle that accommodates one or more passengers. In at least one embodiment, vehicle 1800 may be an articulated lorry used to transport goods. In at least one embodiment, vehicle 1800 may be an airplane, robotic vehicle, or other type of vehicle.

Autonomous vehicles can be described in terms of the level of automation defined by the National Highway Traffic Safety Administration ("NHTSA"), a division of the US Department of Transportation, and the Society of Automotive Engineers ("SAE") "Taxonomy and Definitions for Terms in relation to propulsion automation systems for road motor vehicles” (e.g. Standard No. J 3016-201806 , published June 15, 2018, Standard No. J 3016-201609 , published September 30, 2016, as well as previous and future versions of this standard). In one or more embodiments, the vehicle 1800 may have functionality according to one or more of levels 1-5 of the autonomous driving levels. For example, in at least one embodiment, vehicle 1800 may have conditional automation (level 3), high automation (level 4), and/or full automation (level 5), depending on the embodiment.

In at least one embodiment, vehicle 1800 may include, without limitation, components such as a chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. In at least one embodiment, vehicle 1800 may include, without limitation, a propulsion system 1850, such as an internal combustion engine, hybrid electric power plant, all-electric motor, and/or other type of propulsion system. In at least one embodiment, the propulsion system 1850 may be coupled to a powertrain of the vehicle 1800, which may include, but is not limited to, a transmission to enable the vehicle 1800 to be propelled. In at least one embodiment, propulsion system 1850 may be controlled in response to receiving signals from one or more throttle/accelerators 1852 .

In at least one embodiment, a steering system 1854, which may include, among other things, a steering wheel, is used to steer a vehicle 1800 (e.g., along a desired path or route) when a propulsion system 1850 is operational (e.g., when the vehicle is in motion ). In at least one embodiment, a steering system 1854 may receive signals from steering actuator(s) 1856 . The steering wheel can be optional for full functionality of automation (level 5). In at least one embodiment, a brake sensor system 1846 may be used to actuate vehicle brakes in response to receiving signals from brake actuator(s) 1848 and/or brake sensors.

In at least one embodiment, the controller(s) 1836, which include, among other things, one or more system-on-chips ("SoCs") (not included in 18A shown) and/or graphics processing unit(s) ("GPU(s)"), may provide signals (eg, representative of commands) to one or more vehicle 1800 components and/or systems. For example, in at least one embodiment, controller(s) 1836 may send signals to actuate vehicle brakes via brake actuators 1848, to actuate steering system 1854 via steering actuator(s) 1856, to actuate powertrain system 1850 via throttle/accelerator 1852. Controller(s) 1836 may include one or more onboard (eg, integrated) computing devices (eg, supercomputers) that process sensor signals and issue operational commands (eg, signals representing commands) to enable autonomous driving and/or a human driver to assist in driving the vehicle 1800 . In at least one embodiment, controller(s) 1836 may include a first controller 1836 for autonomous driving functions, a second controller 1836 for functional safety functions, a third controller 1836 for artificial intelligence (e.g., computer vision) functionality, a fourth controller 1836 for infotainment functionality fifth controller 1836 for emergency redundancy and/or other controllers. In at least one embodiment, a single controller 1836 can handle two or more of the above functionalities, two or more controllers 1836 can handle a single functionality and/or any combination thereof.

In at least one embodiment, controller(s) 1836 provide signals to control one or more components and/or systems of vehicle 1800 in response to sensor data received from one or more sensors (eg, sensor inputs). In at least one embodiment, sensor data may be received from, for example and without limitation, Global Navigation Satellite System ("GNSS") sensor(s) 1858 (e.g., Global Positioning System sensor(s), RADAR sensor(s). ) 1860, ultrasonic sensor(s) 1862, LIDAR sensor(s) 1864, inertial measurement unit ("IMU") sensor(s) 1866 (eg, accelerometer, gyroscope(s), magnetic compass(es), magnetometer, etc.), microphone( e) 1896, stereo camera(s) 1868, wide angle camera(s) 1870 (e.g. fisheye cameras), infrared camera(s) 1872, surround camera(s) 1874 (e.g. 360 degree cameras), remote camera(s) (not in 18A shown), mid-range camera(s) (not in 18A shown), speed sensor(s) 1844 (e.g. for measuring the speed of driving 1800), vibration sensor(s) 1842, steering sensor(s) 1840, brake sensor(s) (eg, as part of brake sensor system 1846), and/or other types of sensors.

In at least one embodiment, one or more controller(s) 1836 may receive inputs (e.g., represented by input data) from an instrument cluster 1832 of vehicle 1800 and outputs (e.g., represented by output data, display data, etc.) via a human Provide a machine interface ("HMI") display 1834, an audible alarm, a speaker, and/or via other vehicle 1800 components. In at least one embodiment, the outputs may include information such as vehicle speed, speed, time, map data (e.g., a high-resolution map (not in 18A shown), location data (e.g., the position of vehicle 1800, as on a map), direction, position of other vehicles (e.g., an occupancy grid), information about objects and the status of objects as perceived by controller(s) 1836, etc . For example, in at least one embodiment, the HMI display 1834 may display information about the presence of one or more objects (e.g., a road sign, a warning sign, a traffic light change, etc.) and/or information about maneuvers that the vehicle has performed, is performing, or is performing will perform (e.g., change lanes now, exit 34B in two miles, etc.).

In at least one embodiment, the vehicle 1800 further includes a network interface 1824 that may use wireless antenna(s) 1826 and/or modem(s) to communicate over one or more networks. For example, in at least one embodiment, the network interface 1824 may support communication via Long Term Evolution ("LTE"), Wideband Code Division Multiple Access ("WCDMA"), Universal Mobile Telecommunications System ("UMTS"), Global System for Mobile Communication ( "GSM"), IMT-CDMA Multi-Carrier ("CDMA2000"), etc. In at least one embodiment, wireless antenna(s) 1826 can also enable communication between objects in the vicinity (e.g., vehicles, mobile devices, etc.) using local area networks such as Bluetooth, Bluetooth Low Energy ("LE"), Z-Wave, ZigBee etc. and/or low-power broadband networks ("LPWANs") such as LoRaWAN, SigFox, etc.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, the inference and/or training logic 1515 in the system 18A to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

18B 18 shows an example with camera positions and fields of view for the autonomous vehicle 1800 of FIG 18A according to at least one embodiment. In at least one embodiment, the camera locations and respective fields of view are exemplary and should not be construed in a limiting sense. For example, in at least one embodiment, additional and/or alternative cameras may be included and/or the cameras may be located at different locations on the vehicle 1800 .

In at least one embodiment, camera types for cameras may include, but are not limited to, digital cameras configured for use with vehicle 1800 components and/or systems. The camera(s) may operate at Safety Integrity Level (“ASIL”) B and/or any other ASIL level. In at least one embodiment, the camera types can achieve any frame rate, such as 60 frames per second (fps), 1220 fps, 240 fps, etc., depending on the embodiment. In at least one embodiment, cameras may be deployed using rolling shutters, global shutters, another shutter type, or a combination thereof. In at least one embodiment, a color filter array may be a red-clear-clear-clear color filter array ("RCCC"), a red-clear-clear-clear-blue color filter array ("RCCB"), a red Blue Green Clear Color Filter Array ("RBGC"), a Foveon X3 Color Filter Array, a Bayer Sensor Color Filter Array ("RGGB"), a Monochrome Sensor Color Filter Array, and/or other Kind of include color filter array. In at least one embodiment, cameras with clear pixels, such as cameras with an RCCC, an RCCB, and/or an RBGC color filter array, can be used to increase light sensitivity.

In at least one embodiment, one or more cameras may be used to provide advanced driver assistance system functionality "ADAS") (e.g. as part of a redundant or fail-safe design). For example, in at least one embodiment, a multifunction mono camera may be installed that includes features such as lane departure warning, traffic sign assist, and intelligent headlamp control. In at least one embodiment, one or more cameras (eg, all cameras) may record and provide image data (eg, video) concurrently.

In at least one embodiment, one or more cameras may be mounted in a mount, e.g. B. in a custom (three-dimensional ("3D") printed) assembly to eliminate stray light and reflections from the vehicle interior (e.g. reflections from the dashboard reflected in windshield mirrors) that could interfere with the image data collection capabilities of the camera. With respect to side mirror mounting assemblies, in at least one embodiment, side mirror assemblies may be custom 3D printed such that the camera mounting plate conforms to the shape of the side mirror. In at least one embodiment, the camera(s) may be integrated into the side view mirror. Side view cameras may also have the cameras integrated into four pillars at each corner of the cab, according to at least one embodiment.

In at least one embodiment, cameras with a field of view that includes portions of the environment in front of the vehicle 1800 (e.g., forward-facing cameras) may be used for all-around vision to identify forward-facing paths and obstacles, and to use one or more Provide controllers 1836 and/or control SoCs with information critical to creating an occupancy grid and/or determining preferred vehicle paths. In at least one embodiment, forward-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including, without limitation, emergency braking, pedestrian detection, and collision avoidance. In at least one embodiment, forward-facing cameras may also be used for ADAS functions and systems including, but not limited to, Lane Departure Warnings ("LDW"), Autonomous Cruise Control ("ACC") ") and/or other functions such as traffic sign recognition.

In at least one embodiment, a variety of cameras may be used in a forward-facing configuration, including, for example, a monocular camera platform that includes a color CMOS ("Complementary Metal Oxide Semiconductor") image sensor. In at least one embodiment, a wide-angle camera 1870 may be used to perceive objects coming into view from the periphery (eg, pedestrians, crossing traffic, or bicycles). Although only a wide-angle camera 1870 in 18B As shown, there may be any number (including zero) wide-angle cameras 1870 on the vehicle 1800 in other embodiments. In at least one embodiment, any number of wide-angle camera(s) 1898 (eg, a wide-angle stereo camera pair) may be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. In at least one embodiment, remote cameras 1898 can also be used for object detection and classification, and basic object tracking.

In at least one embodiment, any number of stereo camera(s) 1868 may also be included in a forward-facing configuration. In at least one embodiment, one or more of the stereo camera(s) 1868 may include an integrated controller that includes a scalable processing unit that includes programmable logic ("FPGA") and a multi-core microprocessor with an integrated controller area network ("CAN") or Ethernet interface can provide on a single chip. In at least one embodiment, such a unit can be used to generate a 3D environment map of the vehicle 1800, including a distance estimate for all pixels. In at least one embodiment, stereo camera(s) 1868 may include, among other things, one or more compact stereo vision sensor(s) that may include, among other things, two camera lenses (one each left and right) and an image processing chip that measure the distance from the vehicle 1800 to the target object and use the generated information (e.g. metadata) to activate autonomous emergency braking and lane departure warning functions. In at least one embodiment, other types of stereo camera(s) 1868 may be used in addition to or as an alternative to those described herein.

In at least one embodiment, cameras with a field of view that includes portions of the environment to the side of the vehicle 1800 (eg, side view cameras) may be used for the all-round view, providing information needed to create and update the occupancy grid and to generate side impact collision warnings. For example, in at least one embodiment, surround camera(s) 1874 (eg, four surround cameras 1874 as in 18B shown) positioned on vehicle 1800 . Surround camera(s) 1874 may include, without limitation, any number and combination of wide angle camera(s) 1870, fisheye camera(s), 360 degree camera(s), and/or the like. For example, four fisheye cameras may be located at the front, rear, and sides of vehicle 1800 in at least one embodiment. In at least one embodiment, the vehicle 1800 may use three surround camera(s) 1874 (eg, left, right, and rear) and use one or more other camera(s) (eg, a front-facing camera) as the fourth surround camera.

In at least one embodiment, cameras with a field of view that includes portions of the environment behind the vehicle 1800 (e.g., backup cameras) may be used for parking assistance, surround visibility, rear impact warning, and occupancy grid creation and updating. In at least one embodiment, a variety of cameras may be used, including but not limited to cameras that are also suitable as front-facing camera(s) (e.g., remote camera(s) 1898 and/or mid-range camera(s) 1876, stereo camera(s) 1868 ), infrared camera(s) 1872 etc.) as described here.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are here in connection with 15A and or 15B specified. In at least one embodiment, the inference and/or training logic 1515 in the system 18B to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

18C 18 is a block diagram depicting an example system architecture for the autonomous vehicle 1800 of FIG 18A shows, according to at least one embodiment. In at least one embodiment, all of the components, features, and systems of the vehicle 1800 in 18C shown connected via a bus 1802 . In at least one embodiment, bus 1802 may include, without limitation, a CAN data interface (alternatively referred to herein as "CAN bus"). In at least one embodiment, a CAN may be a network within vehicle 1800 used to support control of various features and functions of vehicle 1800, such as brake application, acceleration, braking, steering, windshield wipers, etc. In at least one embodiment, the bus 1802 can be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). In at least one embodiment, bus 1802 may be read to determine steering wheel angles, vehicle speed, engine revolutions per minute ("RPMs"), button positions, and/or other vehicle status indicators. In at least one embodiment, bus 1802 may be a CAN bus that is ASIL B compliant.

In at least one embodiment, CAN, FlexRay, and/or Ethernet may be used in addition to or as an alternative. In at least one embodiment, there may be any number of buses 1802, including, without limitation, zero or more CAN buses, zero or more FlexRay buses, zero or more Ethernet buses, and/or zero or more other types of buses using any other protocol can include. In at least one embodiment, two or more buses 1802 may be used to perform different functions and/or for redundancy. For example, a first bus 1802 may be used for collision avoidance functionality and a second bus 1802 may be used for actuation control. In at least one embodiment, each bus 1802 can communicate with any component of the vehicle 1800, and two or more buses 1802 can communicate with the same components. In at least one embodiment, any of any number of system on chip(s) ("SoC(s)") 1804, controller(s) 1836, and/or computer in the vehicle may have access to the same input data (e.g., inputs from sensors of the vehicle 1800) and be connected to a common bus, a CAN bus.

In at least one embodiment, the vehicle 1800 may include one or more controllers 1836, such as those referred to herein 18A described. Controller(s) 1836 can be used for a variety of functions. In at least one embodiment, controller(s) 1836 may be coupled to any other vehicle 1800 components and systems and for the Vehicle 1800 control, vehicle 1800 artificial intelligence, vehicle 1800 infotainment, and/or similar functions.

In at least one embodiment, the vehicle 1800 may include any number of SoCs 1804 . Each of the SoCs 1804 may include, without limitation, central processing unit (“CPU(s)”) 1806, graphics processing unit (“GPU(s)”) 1808, processor(s) 1810, cache(s) 1812, accelerator 1814, memory 1816, and/or or other components and features not shown. In at least one embodiment, SoC(s) 1804 may be used to control vehicle 1800 in a variety of platforms and systems. For example, in at least one embodiment, SoC(s) 1804 may be combined in a system (e.g., vehicle 1800 system) with a high definition (“HD”) map 1822 that accepts map updates and/or updates over the network interface 1824 can be obtained from one or more servers (in not shown).

In at least one embodiment, CPU(s) 1806 may comprise a CPU cluster or CPU complex (alternatively referred to herein as "CCPLEX"). In at least one embodiment, the CPU(s) 1806 may include multiple cores and/or level 2 ("L2") caches. For example, in at least one embodiment, CPU(s) 1806 may include eight cores in a coherent multiprocessor configuration. In at least one embodiment, the CPU(s) 1806 may include four dual-core clusters, with each cluster having a dedicated L2 cache (e.g., a 2MB L2 cache). In at least one embodiment, the CPU(s) 1806 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, such that any combination of CPU(s) 1806 clusters may be active at any one time.

In at least one embodiment, one or more of the CPU(s) 1806 may implement power management functions, including, without limitation, one or more of the following features: Individual hardware blocks may be automatically gated from the clock signal when idle. clock-gated") to dynamically save power; each core clock can be detached when the core is not actively executing commands due to the execution of Wait-For-Interrupt ("WFI")/Wait-For-Event ("WFE") instructions, each core can be independently power controlled .be "power gated"; each core cluster can be clocked independently if all cores are clocked or power controlled; and/or each core cluster may be independently power controlled if all cores are power controlled. In at least one embodiment, the CPU(s) 1806 may further implement an improved algorithm for managing power states, specifying the allowable power states and expected wake-up times, and the hardware/microcode determining the best power state to enter the core, cluster, and CCPLEX be able. In at least one embodiment, processing cores may support simplified sequences for entering power states in software, with the work being offloaded to microcode.

In at least one embodiment, GPU(s) 1808 may include an integrated GPU (alternatively referred to herein as "iGPU"). In at least one embodiment, GPU(s) 1808 may be programmable and efficient for parallel workloads. In at least one embodiment, GPU(s) 1808 may use an enhanced tensor instruction set. In at least one embodiment, GPU(s) 1808 may include one or more streaming microprocessors, each streaming microprocessor having an L1 cache (e.g., an L1 cache having at least 96 KB of storage capacity) and two or more streaming microprocessors having an L2 cache (e.g. an L2 cache with a storage capacity of 512 KB). In at least one embodiment, GPU(s) 1808 may include at least eight streaming microprocessors. In at least one embodiment, GPU(s) 1808 may use Compute Application Programming Interface(s) (API(s)). In at least one embodiment, GPU(s) 1808 may utilize one or more parallel computing platforms and/or programming models (e.g., NVIDIA's CUDA).

In at least one embodiment, one or more of the GPU(s) 1808 may be performance optimized to achieve the best performance in automotive and embedded use cases. For example, in one embodiment, GPU(s) 1808 could be fabricated on a Fin Field Effect Transistor ("FinFET"). In at least one embodiment, each streaming microprocessor may include a number of mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores could be divided into four processing blocks. In at least one embodiment, each processing block could have 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two NVIDIA TENSOR-CORES for mixed-precision deep learning matrix arithmetic, a level-zero ("L0") instruction cache , a warp planner, a shipping unit and/or a 64 KB register file. In at least one embodiment, streaming microprocessors may include independent parallel integer and floating point data paths to enable efficient execution of workloads with a mix of computational and addressing computations. In at least one embodiment, streaming microprocessors may include independent thread scheduling functions to enable finer grained synchronization and collaboration between parallel threads. In at least one embodiment, streaming microprocessors may include a combined LI data cache and shared memory unit to improve performance while simplifying programming.

In at least one embodiment, one or more of the GPU(s) 1808 may include high-bandwidth memory ("HBM") and/or a 16GB HBM2 memory subsystem to provide peak memory bandwidth of about 900GB/second in some examples. In at least one embodiment, synchronous graphics random access memory ("SGRAM"), e.g., double data rate synchronous graphics random access memory type 5 ("GDDR5"), may be used in addition to or as an alternative to HBM memory.

In at least one embodiment, GPU(s) 1808 may include a unified memory technology. In at least one embodiment, Address Translation Services ("ATS") support may be used to allow GPU(s) 1808 direct access to CPU(s) 1806 page tables. In at least one embodiment, an address translation request may be sent to the CPU(s) 1806 when the memory management unit ("MMU") GPU(s) 1808 experiences a miss. In response, the CPU(s) 1806 may look up their page tables for a virtual-to-physical mapping of the address and transmit the translation back to the GPU(s) 1808, according to at least one embodiment. In at least one embodiment, unified memory technology may enable a single unified virtual address space for memory of both CPU(s) 1806 and GPU(s) 1808, making programming of GPU(s) 1808 and porting of applications to the GPU(s) 1808 is simplified.

In at least one embodiment, the GPU(s) 1808 may include any number of access counters that may track the number of times GPU(s) 1808 access the memory of other processors. In at least one embodiment, access counters may help ensure that memory pages are moved to the physical memory of the processor accessing pages most frequently, thereby improving efficiency for processor-shared memory areas.

In at least one embodiment, one or more of the SoC(s) 1804 may include any number of cache(s) 1812, including those described herein. For example, in at least one embodiment, the cache(s) 1812 could comprise a level three (“L3”) cache dedicated to both the CPU(s) 1806 and the GPU(s). (s) 1808 is available (i.e., attached to both CPU(s) 1806 and GPU(s) 1808). In at least one embodiment, cache(s) 1812 may include a write-back cache that may track states of lines, e.g., using a cache coherency protocol (e.g., MEI, MESI, MSI, etc.). In at least one embodiment, the L3 cache may be 4MB or more, depending on the embodiment, although smaller cache sizes may be used.

In at least one embodiment, one or more of the SoC(s) 1804 may include one or more accelerators 1814 (eg, hardware accelerators, software accelerators, or a combination thereof). In at least one embodiment, SoC(s) 1804 may include a hardware acceleration cluster, which may include optimized hardware accelerators and/or large on-chip memory. In at least one embodiment, large on-chip memory (eg, 4MB SRAM) may enable a hardware acceleration cluster to speed up neural networks and other computations. In at least one embodiment, a hardware acceleration cluster using GPU(s) 1808 can be used to complement the graphics processor(s) 1808 and offload some tasks (e.g., by more cycles) from the graphics processor(s) 1808 free the graphics processor(s) 1808 to perform other tasks). In at least one embodiment, the accelerator(s) 1814 could be used for targeted workloads (e.g., perception, convolutional neural networks ("CNNs"), recurrent neural networks ("RNNs"), etc.) that are stable enough to support a acceleration to be accessible. In at least one embodiment, a CNN may be region-based or regional convolutional neural networks ("RCNNs") and fast RCNNs (e.g., as used for object detection) or other types of CNNs.

In at least one embodiment, the accelerator(s) 1814 (e.g., hardware acceleration cluster) may include a Deep Learning Accelerator(s) ("DLA"). DLA(s) may include, without limitation, one or more tensor processing units ("TPUs"), which may be configured to provide an additional ten trillion operations per second for deep learning applications and inferencing. In at least one embodiment, TPUs can be accelerators configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). DLA(s) can be further optimized for a specific set of neural network types and floating point operations, as well as for inferencing. In at least one embodiment, the design of DLA(s) can provide more performance per millimeter than a typical general purpose GPU and typically far exceeds the performance of a CPU. In at least one embodiment, TPU(s) may perform multiple functions, including a single-instance convolution function that supports, for example, INT8, INT16, and FP16 data types for both features and weights, and post-processor functions. In at least one embodiment, DLA(s) can quickly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any variety of functions including, for example and without limitation: a CNN for object identification and detection using data from camera sensors; a CNN for estimating distances using data from camera sensors; a CNN for emergency vehicle detection and identification and detection using data from microphones 1896; a CNN for facial recognition and vehicle owner identification using data from camera sensors; and/or a CNN for security-related events.

In at least one embodiment, DLA(s) may perform any function of GPU(s) 1808, and using an inference accelerator, for example, a designer may employ either DLA(s) or GPU(s) 1808 for any function. For example, in at least one embodiment, the designer may concentrate the processing of CNNs and floating point operations on DLA(s) and leave other functions to GPU(s) 1808 and/or other accelerator(s) 1814 .

In at least one embodiment, the accelerator(s) 1814 (e.g., hardware acceleration cluster) may include a programmable vision accelerator(s) ("PVA"), which may alternatively be referred to herein as a computer vision accelerator/ be able. In at least one embodiment, the PVA(s) may be designed and configured to use computer vision algorithms for Advanced Driver Assistance System ("ADAS") 1838, autonomous driving, augmented reality ("AR") ) applications and/or Virtual Reality (“VR”) applications. PVA(s) can provide a balance between performance and flexibility. In at least one embodiment, each PVA may include, for example and without limitation, any number of reduced instruction set ("RISC") cores, direct memory access ("DMA") cores, and/or any number of vector processors.

In at least one embodiment, RISC cores may interact with image sensors (e.g., image sensors of any of the cameras described herein), image signal processor(s), and/or the like. In at least one embodiment, each RISC core may include any memory location. In at least one embodiment, RISC cores may use one of several protocols, depending on the embodiment. In at least one embodiment, RISC cores can run a real-time operating system ("RTOS"). In at least one embodiment, RISC cores may be implemented using one or more integrated circuit devices, application specific integrated circuits ("ASICs"), and/or memory devices. For example, in at least one embodiment, RISC cores could include an instruction cache and/or tightly coupled RAM.

In at least one embodiment, DMA may allow components of PVA(s) to access system memory independently of CPU(s) 1806. In at least one embodiment, DMA may support any number of functions provided to optimize PVA, including but not limited to supporting multi-dimensional addressing and/or circular addressing. In at least one embodiment, DMA may support up to six or more addressing dimensions, which may include, but are not limited to, block width, block height, block depth, horizontal block gradation, vertical block gradation, and/or depth gradation.

In at least one embodiment, vector processors may be programmable processors that may be designed to efficiently and flexibly perform programming of image processing algorithms and to provide signal processing capabilities. In at least one embodiment, PVA may include a PVA core and two vector processing subsystem partitions. In at least one embodiment, the PVA core may include a processor subsystem, DMA engine(s) (eg, two DMA engines), and/or other peripherals. In at least one embodiment, the vector processing subsystem may operate as a primary PVA processing unit and may include a vector processing unit ("VPU"), an instruction cache, and/or vector memory (eg, "VMEM"). In at least one embodiment, the VPU core may include a digital signal processor, such as a Single-Instruction, Multiple Data ("SIMD"), Very Long Instruction Word ("VLIW") digital signal processor. In at least one embodiment, a combination of SIMD and VLIW can improve throughput and speed.

In at least one embodiment, each of the vector processors may include an instruction cache and be coupled to a dedicated memory. As a result, in at least one embodiment, each vector processor can be configured to operate independently of other vector processors. In at least one embodiment, vector processors included in a particular PVA can be configured to use data parallelism. For example, in at least one embodiment, multiple vector processors contained within a single PVA may execute the same computer image rendering algorithm, but in different regions of an image. In at least one embodiment, vector processors included in a given PVA may simultaneously execute different computer vision algorithms on the same image, or even execute different algorithms on sequential images or portions of an image. In at least one embodiment, any number of PVAs may be included in a hardware acceleration cluster and any number of vector processors may be included in each of the PVAs, among others. In at least one embodiment, PVA(s) may include additional ECC (Error Correction Code) memory to improve overall system security.

In at least one embodiment, the accelerator(s) 1814 (e.g., hardware acceleration cluster) may include an on-chip computer vision network and static random-access memory (“SRAM”) to provide high-bandwidth, low-latency SRAM for to provide the accelerator(s) 1814. In at least one embodiment, the on-chip memory may include at least 4 MB of SRAM consisting of, for example and without limitation, eight blocks of field-configurable memory that are accessible by both PVA and DLA. In at least one embodiment, each pair of memory blocks may include an Advanced Peripheral Bus ("APB") interface, configuration circuitry, a controller, and a multiplexer. In at least one embodiment, any type of memory can be used. In at least one embodiment, PVA and DLA can access memory through a framework that allows PVA and DLA high-speed memory access. In at least one embodiment, the backbone may include an on-chip computer vision network that connects PVA and DLA to storage (e.g., using APB).

In at least one embodiment, an on-chip computer vision network may include an interface that determines that both PVA and DLA provide ready and valid signals prior to transmission of control signals/address/data. In at least one embodiment, an interface may provide separate phases and separate channels for control signal/address/data transmission and burst communications for continuous data transmission. In at least one embodiment, an interface may conform to International Standardization Organization ("ISO") 26262 or International Electrotechnical Commission ("IEC") 61508 standards, although other standards and protocols may also be used.

In at least one embodiment, one or more of the SoC(s) 1804 may include a hardware accelerator for real-time ray tracing. In at least one embodiment, a hardware accelerator can be used to quickly and efficiently determine positions and extents of objects (e.g., within a real-world model), generate real-time visualization simulations, interpret RADAR signals, synthesize, and/or analyze of sound propagation, for simulating SONAR systems, for a general wave propagation simulation, for comparison with LIDAR data for the purpose of localization and/or other functions and/or for other purposes.

In at least one embodiment, the accelerator(s) 1814 (eg, hardware accelerator cluster) have a wide range of applications for autonomous driving. In at least one embodiment, PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. In at least one embodiment, the capabilities of PVA are well suited to algorithmic domains that require predictable, low-performance, low-latency processing. In other words, the PVA performs well in semi- or densely regular computations even on small data sets that require predictable, low-latency, low-performance runtimes. In at least one embodiment, PVAs for autonomous vehicles, such as vehicle 1800, are designed to execute classical computer vision algorithms because they are efficient at object recognition and use integer mathematics.

For example, according to at least one embodiment of the technology, PVA is used to perform computer stereo vision. In at least one embodiment, although not intended to be limiting, an algorithm based on semi-global matching may be used in some examples. In at least one embodiment, Level 3-5 autonomous driving applications use on-the-fly motion estimation/stereo matching (e.g., texture from motion, pedestrian detection, lane detection, etc.). In at least one embodiment, the PVA can perform a computer stereo vision function with inputs from two monocular cameras.

In at least one embodiment, PVA may be used to perform dense optical flow. For example, in at least one embodiment, PVA could process raw RADAR data (e.g., using a 4D Fast Fourier Transform) to provide processed RADAR data. In at least one embodiment, PVA is used for runtime depth processing, such as processing raw runtime data to provide processed runtime data.

In at least one embodiment, DLA may be used to operate any type of network to enhance control and driving safety, such as, without limitation, a neural network that outputs a confidence level for each object detection. In at least one embodiment, confidence may be represented or interpreted as a probability or as providing a relative "weight" of each detection compared to other detections. In at least one embodiment, the confidence allows a system to make further decisions about which discoveries should be considered true positive discoveries and not false positive discoveries. For example, in at least one embodiment, a system may set a confidence threshold and only consider discoveries that exceed the threshold to be true positive discoveries. In an embodiment using an automatic emergency braking ("AEB") system, false positive detections would cause the vehicle to automatically perform emergency braking, which is obviously undesirable. In at least one embodiment, very reliable detections can be considered as triggers for AEB. In at least one embodiment, DLA may operate a neural network to regress the confidence value. In at least one embodiment, the neural network may use as input at least a subset of parameters, such as the dimensions of the bounding box, the floor area estimate obtained (e.g., from another subsystem), the output of IMU sensor(s) 1866, the correlated to vehicle 1800 orientation, range, 3D pose estimates of the object obtained from the neural network and/or other sensors (e.g., LIDAR sensor(s) 1864 or RADAR sensor(s) 1860), and others .

In at least one embodiment, one or more of the SoC(s) 1804 may include the data storage(s) 1816 (e.g., memory). In at least one embodiment, data store(s) 1816 may be on-chip memory of SoC(s) 1804 that may store neural networks to be executed on GPU(s) 1808 and/or DLA. In at least one embodiment, data store(s) 1816 may be large enough to store multiple instances of neural networks for redundancy and security. In at least one embodiment, data store(s) 1812 may include L2 or L3 cache(s).

In at least one embodiment, one or more of the SoC(s) 1804 may include any number of processor(s) 1810 (eg, embedded processors). The processor(s) 1810 may include a boot and power management processor, which may be a dedicated one processor and a subsystem that handles boot performance and management functions and associated security enforcement. In at least one embodiment, the boot and power management processor may be part of the boot sequence of the SoC(s) 1804 and provide power management services at runtime. In at least one embodiment, the boot power and management processor may provide clock and voltage programming, support for state transitions in the system with lower power consumption, management of thermal and temperature sensors of the SoC(s) 1804, and/or management of the power states of the SoC(s) 1804. In at least one embodiment, each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and SoC(s) 1804 may use ring oscillators to measure CPU(s) 1806, GPU(s) 1808, and/or accelerator(s) temperatures. n) 1814 to record. In at least one embodiment, if temperatures are determined to exceed a threshold, then the boot and power management processor may enter a temperature fault routine and place the SoC(s) 1804 in a lower performance state and/or the vehicle 1800 in a lower power state place the vehicle 1800 in a safe stop mode (e.g., place the vehicle 1800 in a safe stop mode).

In at least one embodiment, processor(s) 1810 may further include a set of embedded processors that may serve as an audio processing engine. In at least one embodiment, an audio processing engine may be an audio subsystem that enables full hardware support for multi-channel audio across multiple interfaces and a wide and flexible range of audio I/O interfaces. In at least one embodiment, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

In at least one embodiment, processor(s) 1810 may further include an always-on processor engine that may provide the necessary hardware features to support management of low-power sensors and wake-up use cases. In at least one embodiment, the always-on processor engine may include, without limitation, a processor core, tightly coupled RAM memory, supporting peripherals (e.g., timers and interrupt controllers), various I/O control peripherals, and routing logic.

In at least one embodiment, processor(s) 1810 may further include a security cluster engine that includes, without limitation, a dedicated processor subsystem for handling security management for automotive applications. In at least one embodiment, the security cluster engine may include, without limitation, two or more processor cores, tightly coupled RAM memory, supporting peripherals (e.g., timers, an interrupt controller, etc.), and/or routing logic. In a security mode, in at least one embodiment, two or more cores may operate in a lockstep mode and act as a single core with comparison logic to detect differences between their operations. In at least one embodiment, the processor(s) 1810 may further include a real-time camera engine, which may include, among other things, a dedicated processor subsystem for handling real-time camera management. In at least one embodiment, the processor(s) 1810 may further include a high dynamic range signal processor, which may include, among other things, an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

In at least one embodiment, the processor(s) 1810 may include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to produce the final to generate an image for the playback window. In at least one embodiment, the video image compositor may perform lens distortion correction on the wide-angle camera(s) 1870, the surround camera(s) 1874, and/or the sensor(s) of the surveillance camera(s) in perform in the cabin. According to at least one embodiment, in-cabin security camera sensors are preferably monitored by a neural network running on another instance of SoC 1804 and configured to identify events in the cabin and respond accordingly. In at least one embodiment, an in-cabin system may, without limitation, perform lip reading to activate cellular service and make a phone call, dictate emails, change the vehicle's destination, activate or close the vehicle's infotainment system and settings change or provide voice-activated web surfing. In at least one embodiment, certain functions are for the driver available when the vehicle is operating in an autonomous mode and are disabled otherwise.

In at least one embodiment, the video image compositor may include improved temporal denoising for both spatial and temporal denoising. For example, in at least one embodiment where motion occurs in a video, noise reduction may weight spatial information accordingly, thereby reducing the weight of information provided by neighboring frames. In at least one embodiment where an image or portion of an image does not include motion, the temporal noise reduction performed by the video image compositor may use information from the previous image to reduce noise in the current image.

In at least one embodiment, the video image compositor may also be configured to perform stereo equalization on the input stereo lens frames. In at least one embodiment, the video image compositor may be further used for user interface composition when the operating system desktop is used and the GPU(s) 1808 is/are not required to continuously render new interfaces. In at least one embodiment, the video image compositor may be used using the video image compositor to offload the graphics processor(s) 1808 when the graphics processor(s) 1808 is/are powered on and actively performing 3D rendering to improve performance and improve responsiveness.

In at least one embodiment, one or more SoC(s) 1804 may further comprise a camera Mobile Industry Processor Interface (MIPI) serial interface for receiving video and input from cameras, a high-speed interface, and/or a video input block configured for camera - and associated pixel input functions can be used. In at least one embodiment, one or more SoC(s) 1804 may further include an input/output controller(s) that may be controlled by software and used to receive I/O signals other than for a specific roles are provided.

In at least one embodiment, one or more of the SoC(s) 1804 may further include a wide range of peripheral interfaces to enable communication with peripherals, audio encoders/decoders ("codecs"), power management, and/or other devices. SoC(s) 1804 can be used to collect data from cameras (e.g. connected via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g. LIDAR sensor(s) 1864, RADAR sensor(s) 1860 etc. connected via Ethernet may be connected), data from bus 1802 (e.g., vehicle speed 1800, steering wheel position, etc.), data from GNSS sensor(s) 1858 (e.g., connected via Ethernet or CAN bus), etc. In at least one embodiment, one or more SoC(s) 1804 may further include dedicated high-performance mass storage controllers, which may contain their own DMA engines, and which may be used to offload the CPU(s) 1806 from routine data management tasks.

In at least one embodiment, SoC(s) 1804 may be an end-to-end platform with a flexible architecture that includes automation levels 3-5, thereby providing a comprehensive functional safety architecture that includes computer vision and ADAS techniques for diversity and redundancy, provides a platform for a flexible, reliable drive software stack together with deep learning tools. In at least one embodiment, SoC(s) 1804 may be faster, more reliable, and even more power and space efficient than conventional systems. For example, in at least one embodiment, the accelerator(s) 1814 in combination with CPU(s) 1806, GPU(s) 1808, and memory(s) 1816 can provide a fast, efficient platform for level 3-5 autonomous vehicles.

In at least one embodiment, computational vision algorithms may execute on CPUs that may be configured using a high level programming language, such as the C programming language, to perform a variety of processing algorithms on a variety of visual data. However, in at least one embodiment, CPUs are often unable to meet the performance requirements of many computer vision applications, such as those related to runtime and power consumption. In at least one embodiment, many CPUs are unable to execute complex real-time object detection algorithms used in in-vehicle ADAS applications and in practical level 3-5 autonomous vehicles.

The exemplary embodiments described here make it possible to execute multiple neural networks simultaneously and/or sequentially and to combine the results with one another to enable level 3-5 autonomous functionality of autonomous driving. For example, in at least one embodiment, a CNN running on DLA or discrete GPU (e.g., GPU(s) 1820) may include text and word recognition so that a supercomputer can read and understand traffic signs, including signs for which the neural network has not been specially trained. In at least one embodiment, DLA may further include a neural network capable of identifying, interpreting, and providing a semantic understanding of characters, and propagating this semantic understanding to path planning engines running on a CPU complex.

In at least one embodiment, multiple neural networks may be operated simultaneously, such as when driving at level 3, 4, or 5. For example, in at least one embodiment, a warning sign reading "Caution: Flashing Light Indicates Icing" along with an electric light from multiple neural networks Networks can be interpreted independently or together. In at least one embodiment, the sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), the text "Flashing lights indicate icing conditions" may be interpreted by a second deployed neural network that understands the vehicle's path planning software ( preferably running on CPU Complex) that icing conditions are present when flashing lights are detected. In at least one embodiment, "turn signals" may be identified through the operation of a third deployed neural network over multiple frames that informs the vehicle's path planning software of the presence (or absence) of turn signals. In at least one embodiment, all three neural networks may operate concurrently, e.g., within DLA and/or on GPU(s) 1808.

In at least one embodiment, a face recognition and vehicle owner identification CNN may use data from camera sensors to identify the presence of an authorized driver and/or vehicle owner of vehicle 1800 . In at least one embodiment, an always-on sensor processing engine may be used to unlock the vehicle when the owner approaches the driver's door and turns on the lights, and to disarm the vehicle in security mode when the owner exits the vehicle. In this manner, the SoC(s) 1804 provide security against theft and/or violent car hijacking.

In at least one embodiment, the CNN for emergency vehicle detection and identification may use data from emergency vehicle siren detection and identification microphones 1896 . In at least one embodiment, the SoC(s) 1804 use a CNN to classify environmental and city sounds, as well as to classify visual data. In at least one embodiment, the CNN running on DLA is trained to identify the relative closing speed of the emergency vehicle (e.g., using the Doppler effect). In at least one embodiment, the CNN can also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by GNSS sensor(s) 1858 . In at least one embodiment, when operating in Europe, CNN will attempt to identify European sirens and in the United States, CNN will attempt to identify only North American sirens. In at least one embodiment, once an emergency vehicle is detected, a control program using ultrasonic sensor(s) 1862 can be used to perform an emergency vehicle safety routine, brake the vehicle, pull to the side, park the vehicle, and/or to keep the vehicle idling until the emergency vehicle(s) drive(s) past.

In at least one embodiment, vehicle 1800 may include CPU(s) 1818 (e.g., discrete CPU(s) or dCPU(s)) that may be coupled to SoC(s) 1804 via a high-speed interconnect (e.g., PCIe). . For example, in at least one embodiment, the CPU(s) 1818 may comprise an X86 processor. CPU(s) 1818 may be deployed using any functionality, including arbitrating potentially inconsistent results between ADAS sensors and SoC(s) 1804 and/or monitoring status and condition of controller(s) 1836 and/or a Infotainment system on a chip (“Infotainment SoC”) 1830, for example.

In at least one embodiment, vehicle 1800 may include GPU(s) 1820 (e.g., discrete GPU(s) or dGPU(s)) coupled to SoC(s) 1804 via a high-speed interconnect (e.g., NVIDIA's NVLINK). be able. In at least one embodiment, GPU(s) 1820 may provide additional artificial intelligence functionality, eg, by performing redundant and/or different neural networks, and may be used to train and/or update neural networks based at least in part on inputs (eg, sensor data) from vehicle 1800 sensors.

In at least one embodiment, the vehicle 1800 may further include a network interface 1824, which may include, among other things, the wireless antenna(s) 1826 (e.g., one or more wireless antennas 1826 for various communication protocols, such as a cellular antenna, a Bluetooth antenna etc.). In at least one embodiment, network interface 1824 may be used to provide wireless connectivity over the Internet to a cloud (e.g., to server(s) and/or other network devices), to other vehicles, and/or to computing devices (e.g., passenger client devices). ) to allow. In at least one embodiment, vehicle 180 may connect directly to other vehicles and/or connect indirectly (e.g., via networks and via the Internet) to communicate with other vehicles. In at least one embodiment, direct connections may be provided using a vehicle-to-vehicle communication link. A vehicle-to-vehicle communication link may provide vehicle 1800 with information about vehicles in the vicinity of vehicle 1800 (e.g., vehicles in front, to the side, and/or behind of vehicle 1800). In at least one embodiment, the foregoing functionality may be part of a cooperative adaptive cruise control functionality of vehicle 1800 .

In at least one embodiment, network interface 1824 may include an SoC that provides modulation and demodulation functionality and enables controller(s) 1836 to communicate over wireless networks. In at least one embodiment, the network interface 1824 may include a radio frequency front end for baseband to radio frequency upconversion and radio frequency to baseband downconversion. In at least one embodiment, frequency translations can be performed in any technically suitable manner. For example, frequency conversions can be performed by known methods and/or using super-heterodyne methods. In at least one embodiment, the radio frequency front end functionality may be provided by a separate chip. In at least one embodiment, the network interface may include wireless functionality for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and/or other wireless data transmission protocols.

In at least one embodiment, the vehicle 1800 may further include data storage(s) 1828, which may include, without limitation, off-chip memory (e.g., off-SoC(s) 1804). In at least one embodiment, the data memory(s) 1828 may include, without limitation, one or more memory elements including RAM, SRAM, dynamic random-access memory ("DRAM"), video RAM (video random- access memory ("VRAM"), flash, hard drives, and/or other components and/or devices capable of storing at least one bit of data.

In at least one embodiment, vehicle 1800 may further include GNSS sensor(s) 1858 (e.g., GPS and/or assisted GPS sensors) to support mapping, perception, mapping, and/or path planning functions. In at least one embodiment, any number of GNSS sensor(s) 1858 may be used, including, for example and without limitation, a GPS using a USB port with an Ethernet-to-serial (e.g., RS-232) bridge.

In at least one embodiment, the vehicle 1800 may further include RADAR sensor(s) 1860 . The RADAR sensor(s) 1860 may be used by the vehicle 1800 for long range vehicle detection, including in darkness and/or poor weather conditions. In at least one embodiment, the RADAR functionality assurance levels may be ASIL B. RADAR sensor(s) 1860 may use CAN and/or bus 1802 (e.g., to transmit data generated by RADAR sensor(s) 1860) for control and access object tracking data, with access to Ethernet to in some examples access raw data. In at least one embodiment, a variety of RADAR sensor types may be used. For example and without limitation, RADAR sensor(s) 1860 may be suitable for use of front, rear, and side RADAR. In at least one embodiment, one or more RADAR sensors 1860 are pulse Doppler RADAR sensors.

In at least one embodiment, the RADAR sensor(s) 1860 may include various configurations, such as long range with a narrow field of view, short range with a wide field of view, short range side coverage, etc. In at least one embodiment, the RADAR may over large distances are used for adaptive cruise control functionality. In at least one embodiment, long-range RADAR systems may provide a wide field of view realized by two or more independent scans, eg, within a 250 m range. In at least one embodiment, the RADAR sensor(s) 1860 can help distinguish between static and moving objects and can be used by the ADAS system 1838 for emergency brake assist and forward collision warning. Sensors 1860(s) included in a long-range RADAR system may include, without limitation, monostatic multi-mode RADAR with multiple (eg, six or more) fixed RADAR antennas and a high-speed CAN and FlexRay interface. In at least one six antenna embodiment, four central antennas can produce a focused radiation pattern designed to capture the environment of 1800 vehicles at higher speeds with minimal interference from traffic in adjacent lanes. In at least one embodiment, the other two antennas can expand the field of view so that vehicles entering or leaving the lane of vehicle 1800 can be quickly detected.

In at least one embodiment, mid-range RADAR systems may include, as an example, a range of up to 160 m (front) or 80 m (back) and a field of view of up to 42 degrees (front) or 150 degrees (back). In at least one embodiment, short-range RADAR systems may include, without limitation, any number of RADAR sensor(s) 1860 intended for installation at either end of the rear bumper. In at least one embodiment, when a RADAR sensor system is installed at both ends of the rear bumper, it can generate two beams that continuously monitor the blind spot in the rear and adjacent areas of the vehicle. In at least one embodiment, short-range RADAR systems may be used in ADAS system 1838 for blind spot detection and/or lane change assistance.

In at least one embodiment, the vehicle 1800 may further include ultrasonic sensor(s) 1862 . Ultrasonic sensor(s) 1862, which may be attached to the front, rear, and/or side of vehicle 1800, may be used for parking assistance and/or to create and update a population grid. In at least one embodiment, a wide variety of ultrasonic sensor(s) 1862 may be used, and different ultrasonic sensor(s) 1862 may be used for different detection ranges (e.g., 2.5m, 4m). In at least one embodiment, the ultrasonic sensor(s) 1862 may operate at ASIL B functional safety levels.

In at least one embodiment, the vehicle 1800 may include a LIDAR sensor(s) 1864 . The LIDAR sensor(s) 1864 may be used for object and pedestrian detection, emergency braking, collision avoidance, and/or other functions. In at least one embodiment, LIDAR sensor(s) 1864 may be ASIL B functional safety level. In at least one embodiment, vehicle 1800 may include multiple LIDAR sensors 1864 (e.g., two, four, six, etc.) that may be deployed using Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

In at least one embodiment, LIDAR sensor(s) 1864 may provide a list of objects and their distances for a 360 degree field of view. In at least one embodiment, commercially available LIDAR sensor(s) 1864 may have a specified range of approximately 100 m with an accuracy of 2 cm - 3 cm and with support for a 100 Mbps Ethernet connection, for example. In at least one embodiment, one or more non-protruding LIDAR sensors 1864 may be used. In such an embodiment, the LIDAR sensor(s) 1864 may be implemented as a small device that may be embedded in the front, rear, sides, and/or corners of the vehicle 1800 . In at least one embodiment, LIDAR sensor(s) 1864 in such an implementation can provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees, with a range of 200 m even for objects with low reflectivity. In at least one embodiment, a front-mount LIDAR sensor(s) 1864 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

In at least one embodiment, lidar technologies such as 3D flash lidar may also be used. 3D flash LIDAR uses a laser flash as a transmission source to illuminate the area around the vehicle 1800 up to a distance of approximately 200 m. In at least one embodiment, without limitation, a flash LIDAR unit includes a receptor that records the laser pulse travel time and reflected light at each pixel, which in turn corresponds to the range from the vehicle 1800 to objects. In at least one embodiment, flash LIDAR can produce highly accurate and distortion-free images of the environment with each laser flash. In at least one embodiment, four flash LIDAR sensors may be deployed, one on each side of the vehicle 1800. In at least one embodiment, 3D flash LIDAR systems include, without limitation, a solid-state 3D flash LIDAR camera with a fixed beam Array with no moving parts other than a fan (e.g. a non-scanning LIDAR device). In at least one embodiment, a 3D flash LIDAR device may use a 5 nanosecond Class I (eye safe) laser pulse per frame and collect reflected laser light in the form of 3D range point clouds and co-registered intensity data.

In at least one embodiment, the vehicle may further include IMU sensor(s) 1866 . In at least one embodiment, IMU sensor(s) 1866 may be located at the center of the rear axle of vehicle 1800 in at least one embodiment. In at least one embodiment, the IMU sensor(s) 1866 may include, for example and without limitation, accelerometers, magnetometers, gyroscope(s), magnetic compass(es), and/or other types of sensors. In at least one embodiment, such as in six-axis applications, the IMU sensor(s) 1866 may include, without limitation, accelerometers and gyroscopes. In at least one embodiment, such as For example, in nine-axis applications, the IMU sensor(s) 1866 may include, without limitation, accelerometers, gyroscopes, and magnetometers.

In at least one embodiment, the IMU sensor(s) 1866 may be implemented as a miniature High Performance GPS-Aided Inertial Navigation System (GPS/INS) incorporating inertial sensors of micro-electromechanical systems ( MEMS = Micro-Electro-Mechanical Systems), a highly sensitive GPS receiver and advanced Kalman filter algorithms to estimate position, velocity and attitude. In at least one embodiment, the IMU sensor(s) 1866 may enable the vehicle 1800 to estimate heading without requiring input from a magnetic sensor by using GPS to IMU sensor speed changes ( en) 1866 can be directly observed and correlated. In at least one embodiment, the IMU sensor(s) 1866 and the GNSS sensor(s) 1858 may be combined into a single integrated unit.

In at least one embodiment, the vehicle 1800 may include one or more microphones 1896 mounted in and/or around the vehicle 1800 . In at least one embodiment, the microphone(s) 1896 may be used for, among other things, emergency vehicle detection and identification.

In at least one embodiment, vehicle 1800 may further include any number of camera types, including stereo camera(s) 1868, wide-angle camera(s) 1870, infrared camera(s) 1872, surround camera(s) 1874, far camera(s) 1898, mid-range camera (s) 1876 and/or other camera types. In at least one embodiment, cameras may be used to capture image data around the entire periphery of vehicle 1800 . In at least one embodiment, the types of cameras used depend on the vehicle 1800 . In at least one embodiment, any combination of camera types may be used to provide the required coverage around vehicle 1800 . In at least one embodiment, the number of cameras used can vary depending on the embodiment. For example, in at least one embodiment, vehicle 1800 could include six cameras, seven cameras, ten cameras, twelve cameras, or another number of cameras. For example and without limitation, the cameras may support Gigabit Multimedia Serial Link ("GMSL") and/or Gigabit Ethernet. In at least one embodiment, each of the cameras is provided with more details regarding 18A and 18B described.

In at least one embodiment, the vehicle 1800 may further include vibration sensor(s) 1842 . The vibration sensor(s) 1842 can measure vibrations of components of the vehicle 1800, such as axle(s). For example, in at least one embodiment For example, changes in vibration indicate a change in the road surface. In at least one embodiment, when two or more vibration sensors 1842 are used, differences in vibration can be used to determine friction or slippage of the road surface (eg, when there is a difference in vibration between a driven axle and a freely rotating axle).

In at least one embodiment, the vehicle 1800 may include an ADAS system 1838 . ADAS system 1838 may include a SoC in some examples without limitation. In at least one embodiment, ADAS system 1838 may include, without limitation, any number and combination of an autonomous/adaptive/automatic cruise control ("ACC") system, a cooperative adaptive cruise control ("CACC") system, a front collision warning ("FCW") system, and an automatic Emergency Braking (“AEB”) systems include, a Lane Departure Warning (“LDW”) system, a Lane Keeping Assist (“LKA”) system, a Blind Spot Warning (“BSW”) system, a Rear Cross Traffic Warning ("RCTW") system, a Collision Warning ("CW") system, a Lane Centering ("LC") system and/or other systems, features and/or functions.

In at least one embodiment, the ACC system may use RADAR sensor(s) 1860, LIDAR sensor(s) 1864, and/or any number of camera(s). In at least one embodiment, the ACC system may include a longitudinal ACC system and/or a lateral ACC system. In at least one embodiment, the longitudinal ACC system monitors and controls the distance to the vehicle immediately ahead of the vehicle 1800 and automatically adjusts the speed of the vehicle 1800 to maintain a safe distance from vehicles ahead. In at least one embodiment, the side ACC system maintains the distance and advises the vehicle 1800 to change lanes when necessary. In at least one embodiment, side ACC is related to other ADAS applications such as LC and CW.

In at least one embodiment, the CACC system uses information from other vehicles received via network interface 1824 and/or wireless antenna(s) 1826 from other vehicles over a wireless connection or indirectly over a network connection (e.g., over the Internet). can be received. In at least one embodiment, direct connections may be provided through a vehicle-to-vehicle ("V2V") communication link, while indirect connections may be provided through an infrastructure-to-vehicle ("I2V") communication link. In general, the V2V communication concept provides information about immediately preceding vehicles (e.g., vehicles immediately ahead and in the same lane as vehicle 1800), while the I2V communication concept provides information about traffic at a greater distance. In at least one embodiment, the CACC system may include either or both of information sources 12V and V2V. In at least one embodiment, given the information about vehicles ahead of vehicle 1800, the CACC system may be more reliable and has the potential to improve traffic flow and reduce congestion on the road.

In at least one embodiment, the FCW system is designed to alert the driver to a hazard so that corrective action can be taken. In at least one embodiment, the FCW system uses a forward-looking camera and/or RADAR sensor(s) 1860 coupled to a dedicated processor, DSP, FPGA, and/or ASIC that is electrically coupled to the feedback of the driver, such as a display, speaker, and/or vibrating component. In at least one embodiment, the FCW system may provide a warning such as a tone, a visual warning, a vibration, and/or a rapid brake pulse.

In at least one embodiment, the AEB system detects an imminent forward collision with another vehicle or object and may automatically apply the brakes if the driver fails to take corrective action within a specified time or distance parameter. In at least one embodiment, the AEB system may utilize one or more forward-looking cameras and/or one or more RADAR sensors 1860 coupled to a dedicated processor, DSP, FPGA, and/or ASIC. In at least one embodiment, when the AEB system detects a hazard, the AEB system typically first alerts the driver to take corrective action to avoid a collision, and if the driver fails to take corrective action, the AEB system may automatically brake operate to prevent or at least mitigate the effects of the predicted collision. In at least one embodiment the AEB system may include techniques such as dynamic braking assistance and/or braking when a collision is imminent.

In at least one embodiment, the LDW system is equipped with visual, audible, and/or tactile alerts such as steering wheel or seat vibrations to alert the driver when the vehicle 1800 is crossing lane markings. In at least one embodiment, the LDW system is not activated when the driver indicates an intention to depart from the lane by actuating a turn signal. In at least one embodiment, the LDW system may utilize forward-looking cameras coupled to a dedicated processor, DSP, FPGA, and/or ASIC that is electrically coupled to driver feedback, such as a display, speaker, and/or or a vibration component. In at least one embodiment, the LKA system is a variant of the LDW system. The LKA system is designed to provide steering or braking inputs to correct the vehicle 1800 when the vehicle 1800 begins to drift out of lane.

In at least one embodiment, the BSW system detects and alerts the driver to vehicles in an automobile's blind spot. In at least one embodiment, the BSW system may provide a visual, audible, and/or tactile alert to indicate that collapsing or lane changing is unsafe. In at least one embodiment, the BSW system may provide an additional warning when the driver uses a turn signal. In at least one embodiment, the BSW system may utilize rear-facing camera(s) and/or RADAR sensor(s) 1860 coupled to a dedicated processor, DSP, FPGA, and/or ASIC that is electrically coupled to the driver's coupled to feedback, such as a display, speaker, and/or vibration component.

In at least one embodiment, the RCTW system may provide a visual, audible, and/or tactile notification when an object is detected out of range of the backup camera while vehicle 1800 is backing up. In at least one embodiment, the RCTW system includes an AEB system to ensure vehicle brakes are applied to avoid a collision. In at least one embodiment, an RCTW system may utilize one or more rear-facing RADAR sensors 1860 coupled to a dedicated processor, DSP, FPGA, and/or ASIC that is/are electrically coupled to the driver feedback, e.g. B. a display, a speaker and / or a vibrating component.

In at least one embodiment, conventional ADAS systems can be prone to false positives, which can be annoying and distracting to the driver, but are typically not catastrophic because conventional ADAS systems warn the driver and provide the opportunity to decide whether a security condition actually exists and act accordingly. In at least one embodiment, in the event of conflicting results, the vehicle 1800 will self-determine whether to consider the result from a primary or a secondary computer (e.g., first controller 1836 or second controller 1836). For example, in at least one embodiment, ADAS system 1838 may be a backup and/or secondary computer that provides perceptual information to a rationality module of a backup computer. In at least one exemplary embodiment, redundant diverse software can run on hardware components on the rationality monitor of the reserve computer in order to detect perception errors and dynamic driving tasks. In at least one embodiment, ADAS system 1838 outputs may be provided to a higher-level MCU. In at least one embodiment, when outputs from the primary computer and secondary computer conflict, the higher-level MCU determines how to resolve the conflict to ensure safe operation.

In at least one embodiment, the primary computer may be configured to provide the high-level MCU with a confidence score indicative of the primary computer's confidence in the selected outcome. In at least one embodiment, if the confidence exceeds a threshold, the master MCU may follow the direction of the primary computer, regardless of whether the secondary computer provides a conflicting or inconsistent result. In at least one embodiment where the confidence does not meet a threshold and where the primary and secondary computers display different results (e.g., a conflict), the master MCU may mediate between the computers to determine an appropriate result.

In at least one embodiment, the high-level MCU may be configured to operate a trained and configured neural network(s). is to determine, based at least in part on outputs from the primary computer and the secondary computer, conditions under which the secondary computer will provide false alarms. In at least one embodiment, the neural network(s) in the high-level MCU can learn when the output of the secondary computer can and cannot be trusted. For example, in at least one embodiment, if the secondary computer is a RADAR-based FCW system, the neural network(s) in the high-level MCU can learn when the FCW system identifies metallic objects where it are not actually hazards, such as a drainage grate or a manhole cover activating an alarm. In at least one embodiment, when the secondary computer is a camera-based LDW system, a neural network in the higher-level MCU can learn to override the LDW when cyclists or pedestrians are present and lane departure is actually the safest maneuver. In at least one embodiment, the high-level MCU may include at least one DLA or GPU unit capable of operating one or more neural networks with associated memory. In at least one embodiment, the control MCU may comprise and/or be included as a component of SoC(s) 1804 .

In at least one embodiment, ADAS system 1838 may include a secondary computer that executes ADAS functionality using traditional computer-aided vision rules. In at least one embodiment, the secondary computer may operate using classic computer-aided vision (if-then) rules, and having one or more neural networks in the parent MCU may improve reliability, security, and performance . For example, in at least one embodiment, differential implementation and intentional non-identity make the overall system more fault-tolerant, particularly to faults caused by software (or software-hardware interface) functionality. For example, if in at least one embodiment there is a software error or bug in the software running on the primary computer and non-identical software code on the secondary computer produces the same overall result, then the higher-level MCU may have greater confidence that the overall result is correct and the error in does not cause a material error in the software or hardware of the primary computer.

In at least one embodiment, the output of the ADAS system 1838 may be fed into the perceptual block of the host computer and/or the dynamic driving task block of the host computer. For example, in at least one embodiment, if the ADAS system 1838 is displaying a forward crash warning due to an object immediately ahead, the perception block may use that information in identifying objects using that information. In at least one embodiment, the secondary computer may have its own neural network that is trained to reduce the risk of false positives as described herein.

In at least one embodiment, the vehicle 1800 may further include an infotainment SoC 1830 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, in at least one embodiment, infotainment system 1830 may not be an SoC and may include, without limitation, two or more discrete components. In at least one embodiment, the infotainment SoC 1830 may include, without limitation, a combination of hardware and software configured to provide audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., television, movies, streaming etc.), telephone (e.g. hands-free calls), network connection (e.g. LTE, WiFi etc.) and/or information services (e.g. navigation systems, parking assistance, a radio data system, vehicle-related information such as fuel level, total distance traveled, brake fluid level, oil level, door open/closed, air filter information, etc.) to the vehicle 1800. Infotainment SoC 1830 could e.g. radios, turntables, navigation systems, video players, USB and Bluetooth connectivity, carputer, in-car entertainment, WiFi, steering wheel audio controls, hands-free voice control, a heads-up -Display (“HUD”), HMI display 1834, a telematics device, a control panel (e.g. for controlling ng and/or interaction with various components, functions and/or systems) and/or other components. In at least one embodiment, infotainment SoC 1830 may be further used to provide information (e.g., visual and/or audible) to the vehicle user(s), such as information from ADAS system 1838, autonomous driving information such as planned vehicle maneuvers, trajectories, information about the environment (e.g., intersection information, vehicle information, road information, etc.) and/or other information.

In at least one embodiment, the infotainment SoC 1830 may include any amount and type of GPU functionality. In at least one embodiment, the infotainment SoC 1830 can communicate with other devices, systems, and/or components of the vehicle 1800 via the bus 1802 (e.g., CAN bus, Ethernet, etc.). In at least one embodiment, the infotainment SoC 1830 may be coupled to a higher-level MCU such that the infotainment system's GPU may perform some self-driving functions if the primary controller(s) 1836 (e.g., primary and/or backup -Computer of vehicle 1800) fail. In at least one embodiment, the infotainment system SoC 1830 may place the vehicle 1800 in a chaffeur to safe stop mode, as described herein.

In at least one embodiment, the vehicle 1800 may further include a center display device 1832 (e.g., a digital dashboard, an electronic instrument cluster, a digital instrument panel, etc.). Central display device 1832 may include, without limitation, a controller and/or supercomputer (e.g., a discrete controller or supercomputer). In at least one embodiment, central display device 1832 may include, without limitation, any number and combination of gauges such as speedometer, fuel level, oil pressure, tachometer, odometer, turn signals, gear shift indicator, seat belt warning light(s), parking brake warning light(s), engine trouble light(s), information about supplemental restraint systems (e.g. airbag), lighting controls, security system controls, navigation information, etc. In some examples, information may be displayed and/or shared between infotainment SoC 1830 and central display device 1832 . In at least one embodiment, center display device 1832 may be included as part of infotainment SoC 1830 or vice versa.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are here in connection with 15A and or 15B provided. In at least one embodiment, the inference and/or training logic 1515 in the system 18C to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

18D 1876 is a diagram of a system 1876 for communication between the cloud-based server(s) and the autonomous vehicle 1800 of FIG 18A , according to at least one embodiment. In at least one embodiment, system 1876 may include, without limitation, server 1878, network(s) 1890, and any number and type of vehicles, including vehicle 1800. The server(s) 1878 may include, without limitation, a variety of GPUs 1884(A)-1884(H) (collectively referred to herein as GPUs 1884), PCIe switches 1882(A)-1882(H) (collectively referred to herein as PCIe switches 1882) and/or CPUs 1880(A)-1880(B) (collectively referred to herein as CPUs 1880). GPUs 1884, CPUs 1880, and PCIe switches 1882 may be interconnected using high-speed interconnects such as, without limitation, NVLink interfaces 1888 and/or PCIe interconnects 1886 developed by NVIDIA. In at least one embodiment, the GPUs 1884 are interconnected via an NVLink and/or NVSwitch SoC, and the GPUs 1884 and PCIe switches 1882 are interconnected via PCIe connections. While eight GPUs 1884, two CPUs 1880, and four PCIe switches 1882 are shown in at least one embodiment, this is not intended to be limiting. In at least one embodiment, each of the servers 1878 may include any number of GPUs 1884, CPUs 1880, and/or PCIe switches 1882 in any combination, without limitation. For example, in at least one embodiment, each server 1878 could include eight, sixteen, thirty-two, and/or more GPUs 1884 .

In at least one embodiment, the server(s) 1878 may receive, via the network(s) 1890 and from vehicles, image data representing images showing unexpected or changed road conditions, such as recently started road works. In at least one embodiment, the server(s) 1878 may, via the network(s) 1890 and to the vehicles, provide neural networks 1892, updated neural networks 1892, and/or map information 1894, including, without limitation, information about traffic and road conditions , transfer. In at least one embodiment, updates to the map information 1894 may include, but are not limited to, updates to an HD map 1822, eg, information about construction sites, potholes, detours, flooding, and/or other obstructions. In at least one embodiment, neural networks 1892, updated neural networks 1892, and/or map information 1894 may be selected new training and/or new experiences represented in data received from any number of vehicles in the area and/or based at least in part on training performed in a data center (e.g. using servers (n) 1878 and/or other servers).

In at least one embodiment, server(s) 1878 may be used to train machine learning models (e.g., neural networks) based at least in part on training data. Training data can be generated by vehicles and/or in a simulation (e.g. using the engine). In at least one embodiment, any amount of training data is tagged (e.g., if the associated neural network benefits from supervised learning) and/or undergoes other pre-processing. In at least one embodiment, any set of training data is not tagged and/or pre-processed (e.g., when the associated neural network does not require supervised learning). In at least one embodiment, once machine learning models are trained, they may be used by vehicles (e.g., transmitted to vehicles over network(s) 1890 and/or machine learning models may be used by server(s) 1878 to remotely monitor vehicles.

In at least one embodiment, server(s) 1878 may receive data from vehicles and apply data to live neural networks to obtain intelligent inference in real-time. In at least one embodiment, the server(s) 1878 may include deep learning supercomputers and/or dedicated AI computers with GPU(s) 1884, such as the DGX and DGX Station machines developed by NVIDIA. However, in at least one embodiment, server(s) 1878 may include a deep learning infrastructure using CPU-based data centers.

In at least one embodiment, the server(s) 1878 deep learning infrastructure may provide fast, real-time inference services and utilize this capability to assess and verify the health of processors, software, and/or associated hardware in the vehicle 1800 . For example, in at least one embodiment, the deep learning infrastructure may receive periodic updates from vehicle 1800, such as a sequence of images and/or objects that vehicle 1800 located in that sequence of images (e.g., through computer-aided vision and/or other classification techniques for machine learning objects). In at least one embodiment, the deep learning infrastructure can run its own neural network to identify objects and compare them to the objects identified by vehicle 1800 . If the results do not match and the deep learning infrastructure concludes that the AI in vehicle 1800 is faulty, then server(s) 1878 may send a signal to vehicle 1800 and instruct a vehicle 1800 failsafe computer to take control, notify passengers and perform safe parking manoeuvres.

In at least one embodiment, server(s) 1878 may include graphics processor(s) 1884 and one or more programmable inference accelerators (eg, NVIDIA's TensorRT 3). In at least one embodiment, the combination of GPU-powered servers and inference acceleration can enable real-time responsiveness. In at least one embodiment, eg where performance is less critical, servers powered by CPUs, FPGAs and other processors may be used for inference purposes. In at least one embodiment, the hardware structure(s) 1515 are used to perform one or more embodiments. Details of hardware structure(x) 1515 are here in connection with 15A and or 15B provided.

19 19 is a block diagram showing an example computer system, which may be a system with interconnected devices and components, a system-on-a-chip (SOC), or a combination thereof 1900 formed with a processor. which may comprise execution units for executing an instruction, according to at least one embodiment. In at least one embodiment, the computer system 1900 may include, without limitation, a component such as a processor 1902 for using execution units including logic to perform algorithms on process data consistent with the present disclosure, such as in an embodiment described herein. In at least one embodiment, computer system 1900 may include processors such as the PENTIUM® processor family, Xeon™, Itanium®, XScale™, and/or StrongARM™, Intel® Core™, or Intel® Nervana™ microprocessors manufactured by Intel Corporation of Santa Clara, California, although other systems (including personal computers with other microprocessors, engineering workstations, set-top boxes, and the like) may be used. in minutes In at least one embodiment, computer system 1900 may run a version of the WINDOWS operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (eg, UNIX and Linux), embedded software, and/or graphical user interfaces may also be used.

Example embodiments can be used in other devices such as handheld devices and embedded applications. Some examples of handheld devices include cell phones, Internet protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld personal computers. In at least one embodiment, embedded applications may include a microcontroller, digital signal processor ("DSP"), system on a chip, network computers ("NetPCs"), set top boxes, network hubs, wide area network ("WAN") include switches or any other system capable of executing one or more instructions, according to at least one embodiment.

In at least one embodiment, the computer system 1900 may include, among other things, a processor 1902, which may include, among other things, one or more units 1908 for training and/or inferring machine learning models according to the techniques described herein. In at least one embodiment, system 19 is a single processor desktop or server system, but in another embodiment system 19 may be a multiprocessor system. In at least one embodiment, processor 1902 may include, without limitation, a complex instruction set computer (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (CISC) microprocessor . very long instruction word, VLIW), a processor implementing a combination of instruction sets, or any other processing device such as a digital signal processor. In at least one embodiment, processor 1902 may be coupled to a processor bus 1910 that may transfer data signals between processor 1902 and other components in computer system 1900.

In at least one embodiment, processor 1902 may include an internal level 1 ("L1") cache memory ("cache") 1904 without limitation. In at least one embodiment, processor 1902 may have a single internal cache or multiple levels of internal cache memory. In at least one embodiment, the cache memory may be external to processor 1902. Other embodiments may also include a combination of internal and external caches, depending on the particular implementation and need. In at least one embodiment, register file 1906 may store various types of data in various registers including, without limitation, integer registers, floating point registers, status registers, and instruction pointer registers.

In at least one embodiment, processor 1902 also resides in execution unit 1908, which includes, among other things, logic for performing integer and floating point operations. Processor 1902 may also include microcode ("ucode") read only memory ("ROM") , which stores microcode for specific macroinstructions. In at least one embodiment, the execution unit 1908 may include logic for handling a packed instruction set 1909 . In at least one embodiment, by including the packed instruction set 1909 in the instruction set of a general purpose processor 1902, along with associated circuitry for executing instructions, operations used by many multimedia applications can be performed using packed data in a general purpose processor 1902. In one or more embodiments, many multimedia applications may run faster and more efficiently by utilizing the full width of a processor's data bus to perform operations on packed data, thereby eliminating the need to transfer smaller chunks of data across the processor's data bus to transfer to perform one or more operations one data item at a time.

In at least one embodiment, execution unit 1908 may also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits. In at least one embodiment, computer system 1900 may include memory 1920 without limitation. In at least one embodiment, memory 1920 may be implemented as a dynamic random access memory (“DRAM”) device, static random access memory (“SRAM”) device, flash memory device, or other storage device. Memory 1920 may store instruction(s) 1919 and/or data 1921 represented by data signals executable by processor 1902.

In at least one embodiment, the system logic chip may be coupled to processor bus 1910 and memory 1920. In at least one embodiment, the system logic chip may include, without limitation, a memory controller hub ("MCH") 1916 and the processor 1902 may communicate with the MCH 1916 via the processor bus 1910 . In at least one embodiment, MCH 1916 may provide a high-bandwidth memory path from 1918 to 1920 for command and data storage and for graphics command, data, and texture storage. In at least one embodiment, MCH 1916 may route data signals between processor 1902, memory 1920, and other components in computer system 1900, and bridge data signals between processor bus 1910, memory 1920, and a system I/O 1922. In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, MCH 1916 may be coupled to MCH 1916 via a high-speed memory path 1918 to memory 1920 and graphics/video card 1912 via an Accelerated Graphics Port ("AGP") interconnect 1914 .

In at least one embodiment, computer system 1900 may use system I/O 1922, which is a proprietary hub interface bus, to couple MCH 1916 to I/O control hub ("ICH") 1930. In at least one embodiment, ICH 1930 may provide direct connections to some I/O devices via a local I/O bus. In at least one embodiment, the local I/O bus may include, without limitation, a high speed I/O bus for connecting peripheral devices to the memory 1920, chipset, and processor 1902. Examples may include, without limitation, an audio controller 1929, a firmware hub ("flash BIOS") 1928, a wireless transceiver 1926, a data store 1924, a legacy I/O controller 1923 containing user input and keyboard interfaces serial expansion port 1927 , such as Universal Serial Bus ("USB"), and network controller 1934 . Data storage 1924 may include a hard drive, floppy disk drive, CD-ROM device, flash memory device, or other mass storage device.

According to at least one embodiment 19 a system that includes interconnected hardware devices or "chips" while 19 in other embodiments, an example system-on-a-chip ("SoC") may show. In at least one embodiment, the 19 Devices shown may be connected using proprietary connections, standardized connections (e.g. PCIe) or a combination thereof. In at least one embodiment, one or more components of system 1900 are interconnected by Compute Express Link (CXL) connections.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, the inference and/or training logic 1515 in the system 19 to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

20 12 is a block diagram showing an electronic device 2000 for use with a processor 2010, according to at least one embodiment. In at least one embodiment, electronic device 2000 may be, for example and without limitation, a notebook, tower server, rack server, blade server, laptop, desktop, tablet, mobile device, phone, embedded computer or any other suitable electronic device.

In at least one embodiment, the system 2000 may include, without limitation, a processor 2010 communicatively coupled to any suitable number or type of components, peripherals, modules, or devices. In at least one embodiment, processor 2010 is coupled via a bus or interface, such as a 1°C bus, a System Management Bus ("SMBus"), a Low Pin Count (LPC) bus, a Serial Peripheral Interface ( "SPI"), a High Definition Audio ("HDA") bus, a Serial Advance Technology Attachment ("SATA") bus, a Universal Serial Bus ("USB") (versions 1, 2, 3), or a Universal Asynchronous receiver/transmitter ("UART") bus. 20 FIG. 10 shows a system comprising interconnected hardware devices or “chips” while FIG 20 in other embodiments, an example system on a chip ("SoC") may show. In at least one embodiment, the 20 devices represented by proprietary connections, standardized connections (e.g. PCIe) or a combination of these connected together. In at least one embodiment, one or more components of 20 interconnected by Compute Express Link (CXL) connections.

In at least one embodiment, 20 a display 2024, a touch screen 2025, a touchpad 2030, a near field communication ("NFC") unit 2045, a sensor hub 2040, a thermal sensor 2046, an express chipset ("EC") 2035, a trusted platform module ("TPM") ) 2038, BIOS/firmware/flash memory ("BIOS", "FW Flash") 2022, a DSP 2060, a drive "SSD or HDD") 2020 such as a solid state disk ("SSD") or hard disk drive (" HDD”), a Wireless Local Area Network (“WLAN”) unit 2050, a Bluetooth unit 2052, a Wireless Wide Area Network (“WWAN”) unit 2056, a Global Positioning System (GPS) 2055, a camera ( "USB 3. 0 camera") 2054, such as a USB 3.0 camera, or a Low Power Double Data Rate ("LPDDR") storage device ("LPDDR3") 2015, such as those contained in the LPDDR3 standard is implemented.These components may each be implemented in any suitable manner.

In at least one embodiment, other components may be communicatively coupled to processor 2010 via the components discussed above. In at least one embodiment, an accelerometer 2041 , an ambient light sensor ("ALS") 2042 , a compass 2043 , and a gyroscope 2044 may be communicatively coupled to the sensor hub 2040 . In at least one embodiment, a temperature sensor 2039, a fan 2037, a keyboard 2046, and a touchpad 2030 may be communicatively coupled to the EC 2035. In at least one embodiment, the speaker 2063, a headphone 2064, and a microphone ("mic") 2065 may be communicatively coupled to an audio unit ("audio codec and class D amp") 2064, which in turn is communicatively coupled to the DSP 2060 can be. In at least one embodiment, audio unit 2064 may include, for example and without limitation, an audio encoder/decoder ("codec") and a class-D amplifier. In at least one embodiment, SIM card ("SIM") 2057 may be communicatively coupled to WWAN unit 2056 . In at least one embodiment, components such as WLAN unit 2050 and Bluetooth unit 2052, as well as WWAN unit 2056 may be implemented in a Next Generation Form Factor ("NGFF").

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, the inference and/or training logic 1515 in the system 20 to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

21 12 shows a computer system 2100, according to at least one embodiment. In at least one embodiment, computer system 2100 is configured to implement various methods and methods described in this disclosure.

In at least one embodiment, the computer system 2100 includes, without limitation, at least one central processing unit (“CPU”) 2102 connected to a communication bus 2110, which may be communicated using any suitable protocol such as Peripheral Component Interconnect (PCI), Peripheral Component Interconnect Express (“PCI-Express”), AGP (“Accelerated Graphics Port”), HyperTransport or any other bus or point-to-point communication protocol or any other bus or point-to-point communication protocol. In at least one embodiment, computer system 2100 includes, without limitation, main memory 2104 and control logic (e.g., implemented as hardware, software, or a combination thereof), and data is stored in main memory 2104, which may take the form of random access memory ("RAM"). In at least one embodiment, a network interface subsystem ("network interface") 2122 is provided by computer system 2100 to interface to other computing devices and networks for receiving data from and transmitting data to other systems.

In at least one embodiment, the computing system 2100 includes, in at least one embodiment, the input devices 2108, the parallel processing system 2112, and the display devices 2106, which may be a conventional cathode ray tube ("CRT"), a liquid crystal display ("LCD"), a light emitting diode ( "LED"), a plasma display or other suitable display technologies rea can be lised. In at least one embodiment, user input is received from input devices 2108 such as a keyboard, mouse, touchpad, microphone, and more. In at least one embodiment, each of the foregoing modules may be arranged on a single semiconductor platform to form a processing system.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, the inference and/or training logic 1515 in the system 21 to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

22 12 shows a computer system 2200, according to at least one embodiment. In at least one embodiment, computer system 2200 includes, without limitation, computer 2210 and thumb drive 2220. In at least one embodiment, computer 2210 may include any number and type of processor(s) (not shown) and memory (not shown). In at least one embodiment, computer 2210 includes, without limitation, a server, a cloud instance, a laptop, and a desktop computer.

In at least one embodiment, USB key 2220 includes, without limitation, processing unit 2230, USB interface 2240, and USB interface logic 2250. In at least one embodiment, processing unit 2230 may be any instruction execution system, device, or device , capable of executing commands. In at least one embodiment, processing unit 2230 may include any number and type of processing cores (not shown), without limitation. In at least one embodiment, the processing core 2230 comprises an application specific integrated circuit ("ASIC") optimized to perform any number and type of operations associated with machine learning. For example, in at least one embodiment, the processing core 2230 is a tensor processing unit ("TPC") optimized for performing machine learning inference operations. In at least one embodiment, the processing core 2230 is an image processing unit ("VPU") optimized for performing inference operations related to machine vision and machine learning.

In at least one embodiment, the USB interface 2240 can be any type of USB plug or USB socket. For example, in at least one embodiment, the USB interface 2240 is a USB 3.0 Type-C socket for data and power. In at least one embodiment, USB interface 2240 is a USB 3.0 Type A connector. In at least one embodiment, USB interface logic 2250 may include any amount and type of logic that enables processing unit 2230 to communicate over USB Port 2240 to interface with devices (e.g. computer 2210).

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, the inference and/or training logic 1515 in the system 22 to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

23A Figure 12 shows an example architecture in which a plurality of GPUs 2310-2313 are communicatively coupled to a plurality of multi-core processors 2305-2306 via high-speed links 2340-2343 (eg, buses, point-to-point links, etc.). In one embodiment, high-speed links 2340-2343 support a communication throughput of 4 GB/s, 30 GB/s, 80 GB/s, or higher. Various connection protocols can be used including, but not limited to, PCIe 4.0 or 5.0 and NVLink 2.0.

Additionally, and according to one embodiment, two or more of the GPUs 2310-2313 are interconnected via high-speed links 2329-2330, which may be implemented with the same or different protocols/links than those used for the high-speed links 2340-2343. Similarly, two or more multi-core processors 2305-2306 can be connected via high-speed interconnect 2328, which can be symmetric multiprocessor (SMP) buses operating at 20 GB/s, 30 GB/s, 120 GB/s, or work higher. Alternatively, all communication between different in 23A system components shown take place via the same protocols/links (e.g. via a common connection structure).

In one embodiment, each multi-core processor 2305-2306 is communicatively coupled to processor memory 2301-2302 via memory interconnects 2326-2327, and each graphics processor 2310-2313 is communicatively coupled to GPU memory 2320-2323 via GPU memory interconnects 2350-2353. Memory interconnects 2326-2327 and 2350-2353 may use the same or different memory access technologies. As an example, without limitation, processor memory 2301-2302 and GPU memory 2320-2323 may be volatile memory such as dynamic random access memory (DRAM) (including stacked DRAMs), graphics DDR-SDRAM (GDDR) (e.g., GDDR5, GDDR6), or High bandwidth memory (HBM) and/or non-volatile memory such as 3D XPoint or Nano-Ram. In one embodiment, some of the processor memories 2301-2302 may be volatile memory and some may be non-volatile memory (e.g., using a two-level memory hierarchy (2LM)).

As described herein, while different processors 2305-2306 and GPUs 2310-2313 can be physically coupled to a particular memory 2301-2302 and 2320-2323, respectively, a unified memory architecture can be implemented using the same virtual system address space (aka referred to as the “effective address space”) is distributed across different physical memories. For example, processor memories 2301-2302 may each include 64 GB of system memory address space and GPU memories 2320-2323 may each include 32 GB of system memory address space (resulting in a total of 256 GB of addressable memory in this example).

23B 13 shows additional details for a connection between a multi-core processor 2307 and a graphics accelerator module 2346, according to one embodiment. Graphics accelerator module 2346 may include one or more GPU chips integrated on a line card coupled to processor 2307 via high speed interconnect 2340 . Alternatively, the 2346 graphics accelerator module may be integrated into the same package or chip as the 2307 processor.

In at least one embodiment, the processor 2307 depicted includes a plurality of cores 2360A-2360D, each with a translation lookaside buffer 2361A-2361D and one or more caches 2362A-2362D. In at least one embodiment, cores 2360A-2360D may include various other components for executing instructions and processing data that are not shown. Caches 2362A-2362D may include level 1 (L1) and level 2 (L2) caches. Additionally, one or more shared caches 2356 may be included within caches 2362A-2362D and shared between sets of cores 2360A-2360D. For example, one embodiment of processor 2307 includes 24 cores, each with its own LI cache, twelve shared L2 caches, and twelve shared L3 caches. In this embodiment, one or more L2 and L3 caches are shared between two adjacent cores. The processor 2307 and the graphics accelerator module 2346 are connected to the system memory 2314, which contains the processor memories 2301-2302 of 23A may include.

Coherency is maintained for data and instructions stored in various caches 2362A-2362D, 2356 and in system memory 2314 via communication between the cores over a coherency bus 2364. For example, cache coherency logic/circuitry may be associated with each cache to communicate over the coherency bus 2364 in response to detected reads or writes to particular cache lines. In one implementation, a cache snooping protocol is implemented over the coherency bus 2364 to snoop on cache accesses.

In one embodiment, a proxy circuit 2325 communicatively couples the graphics accelerator module 2346 to the coherency bus 2364 so that the graphics accelerator module 2346 can participate in a cache coherency protocol as a peer of the cores 2360A-2360D. In particular, an interface 2335 is for connection to the proxy circuitry 2325 over the high speed link 2340 (e.g. PCIe bus, NVLink, etc.) and an interface 2337 connects the graphics accelerator module 2346 to the connection 2340.

In one implementation, an accelerator integration circuit 2336 is provided for a plurality of graphics processing engines 2331, 2332, N of the graphics accelerator module 2346 for cache management, memory access, context management, and interrupt management services. The engines 2331, 2332, N can each comprise a separate graphics processing unit (GPU). Alternatively, the graphics processing engines 2331, 2332, N can include different types of graphics processing engines within a GPU, such as e.g. B. graphics execution units, media processing engines (e.g. video encoders/decoders), samplers and blit engines. In at least one embodiment, graphics accelerator module 2346 may be a GPU with a plurality of graphics processing engines 2331-2332,N or graphics processing engines 2331-2332,N may be individual GPUs integrated into a common chassis, line card, or chip.

In one embodiment, the accelerator integration circuit 2336 includes a memory management unit (MMU) 2339 for performing various memory management functions such as virtual to physical memory translations (also referred to as effective to real memory translations) and memory access protocols for accessing the system memory 2314. MMU 2339 can may also include a translation lookaside buffer (TLB) (not shown) for caching translations from virtual/effective to physical/real addresses. In one embodiment, a cache 2338 stores instructions and data for efficient access by engines 2331-2332,N. In one embodiment, the data stored in cache 2338 and graphics memories 2333-2334,M is shared with core caches 2362A-2362D , 2356 and the system memory 2314 are kept coherent. As previously mentioned, this can be done via proxy circuitry 2325 for cache 2338 and memories 2333-2334, M (e.g. sending updates to cache 2338 related to changes/accesses to cache lines in processor caches 2362A-2362D , 2356 and receiving updates from cache 2338).

A set of registers 2345 stores context data for threads executed by the graphics processing engines 2331-2332,N and a context management circuit 2348 manages thread contexts. For example, the context management circuitry 2348 may perform save and restore operations to save and restore the contexts of different threads during context switching (e.g., when a first thread is saved and a second thread saved so that a second thread is executed by the graphics processing engine can be). For example, upon a context switch, the context management circuit 2348 may store current register values in a particular area in memory (e.g., identified by an identifier). It can then restore register values when returning to a context. In one embodiment, an interrupt management circuit 2347 receives and processes interrupts received from system devices.

In one implementation, virtual/effective addresses from graphics processing engine 2331 are translated by MMU 2339 to real/physical addresses in system memory 2314 . One embodiment of the accelerator integration circuit 2336 supports multiple (e.g., 4, 8, 16) graphics accelerator modules 2346 and/or other accelerator devices. The graphics accelerator module 2346 can be dedicated to a single application running on the processor 2307 or shared between multiple applications. In one embodiment, a virtualized graphics execution environment is presented in which resources of graphics processing engines 2331-2332, N are shared between multiple applications or virtual machines (VMs). In at least one embodiment, resources may be divided into "slices" that are allocated to different VMs and/or applications based on the processing requirements and priorities associated with VMs and/or applications.

In at least one embodiment, accelerator integration circuit 2336 functions as a bridge to a system graphics accelerator module 2346 and provides address translation and system memory caching services. In addition, the accelerator integration circuit 2336 may provide virtualization facilities for a host processor to manage virtualization of the graphics processing modules 2331-2332, interrupts, and memory management.

Since the hardware resources of the graphics processing engines 2331-2332,N are explicitly mapped to a real address space seen by the host processor 2307, any host processor can directly address these resources via an effective address value. A functionality of Accelerator integration circuit 2336, in one embodiment, is the physical separation of engines 2331-2332, N so that they appear to a system as independent entities.

In at least one embodiment, one or more graphics memories 2333-2334,M are coupled to engines 2331-2332,N, respectively. Graphics memories 2333-2334,M store commands and data to be processed by the respective graphics processing engine 2331-2332,N. Graphics memory 2333-2334,M can be volatile memory such as DRAM (including stacked DRAM), GDDR memory (e.g. GDDR5, GDDR6) or HBM and/or non-volatile memory such as 3D XPoint or Nano-Ram .

In one embodiment, to reduce data traffic over link 2340, biasing techniques are used to ensure that the data stored in graphics memories 2333-2334,M is the data most frequently used by the graphics processing engines. Engines 2331-2332, N and preferably not used by cores 2360A-2360D (at least not often). Similarly, a default mechanism attempts to hold the data needed by the cores (and preferably not by the graphics processing engines 2331-2332,N) in the cores' caches 2362A-2362D, 2356 and system memory 2314.

23C FIG. 12 shows another exemplary embodiment in which accelerator integrated circuit 2336 is integrated in processor 2307. FIG. In this embodiment, engines 2331-2332,N communicate directly over high speed link 2340 with accelerator integrated circuit 2336 via interface 2337 and interface 2335 (which in turn may use any type of bus or interface protocol). The accelerator integrated circuit 2336 can perform the same operations as those in FIG 23B as described, but possibly with higher throughput since it is in close proximity to the coherency bus 2364 and caches 2362A-2362D, 2356. In one embodiment, various programming models are supported, including a dedicated process programming model (no graphics accelerator virtualization) and shared programming models (with virtualization), which may include programming models controlled by the accelerator integration circuit 2336 and programming models controlled by the Graphics accelerator module 2346 can be controlled.

In at least one embodiment, the graphics processing engines 2331-2332,N are dedicated to a single application or process under a single operating system. In at least one embodiment, a single application can route other application requests to the graphics processing engines 2331-2332,N, thus providing virtualization within a VM/partition.

In at least one embodiment, the graphics processing engines 2331-2332,N may be shared between multiple VM/application partitions. In at least one embodiment, shared models may use a system hypervisor to virtualize the graphics processing engines 2331-2332,N to allow access by any operating system. On systems with a partition without a hypervisor, the engines 2331-2332, N for graphics processing applications are part of an operating system. In at least one embodiment, an operating system may virtualize the graphics processing engines 2331-2332,N to allow access to any process or application.

In at least one embodiment, the graphics accelerator module 2346 or an individual graphics processing engine 2331-2332, N selects a process item using a process handle. In one embodiment, the process elements are stored in system memory 2314 and are addressable using an effective address-to-real address translation technique described herein. In at least one embodiment, a process handle may be an implementation-specific value provided to a host process upon registering its context with the graphics processing engine 2331-2332, N (i.e., calling system software to add a process item to a process item linked list to add). In at least one embodiment, the lower 16 bits of a process handle may be an offset of the process item within a process item linked list.

23D Figure 12 shows an exemplary accelerator integration slice 2390. As used herein, a "slice" includes a specific portion of the accelerator integration circuit's 2336 processing resources. The effective application address space 2382 within system memory 2314 stores the process items 2383. In one embodiment, process items 2383 are stored in response to GPU calls 2381 from applications 2380 executing on processor 2307. A process element 2383 contains the process state for the corresponding application 2380. A work descriptor (WD) 2384 contained in process element 2383 may be a single job requested by an application or may contain a pointer to a queue of jobs. In at least one embodiment, WD 2384 is a pointer to a job request queue in an application's address space 2382 .

Graphics accelerator module 2346 and/or individual graphics processing engines 2331-2332,N may be shared by all or a subset of processes in a system. In at least one embodiment, infrastructure for establishing process state and sending a WD 2384 to a graphics accelerator 2346 to start a job in a virtualized environment may be included.

In at least one embodiment, a dedicated process programming model is implementation specific. In this model, a single process owns the graphics accelerator module 2346 or a single graphics processing engine 2331. Since the graphics accelerator module 2346 is owned by a single process, a hypervisor initializes the accelerator IC 2336 for an owning partition and an operating system initializes the accelerator IC 2336 for an owning process when the Graphics Accelerator Module 2346 is allocated.

In operation, a WD fetch unit 2391 in the accelerator integration section 2390 fetches the next WD 2384, which includes an indication of the work to be done by one or more graphics processing engines of the graphics accelerator module 2346. Data from WD 2384 may be stored in registers 2345 and used by MMU 2339, interrupt management circuitry 2347, and/or context management circuitry 2348 as shown. For example, one embodiment of MMU 2339 includes segment/page run circuitry for accessing segment/page tables 2386 within operating system virtual address space 2385 . Interrupt management circuitry 2347 may process interrupt events 2392 received from graphics accelerator module 2346 . When performing graphics operations, an effective address 2393 generated by a graphics processing engine 2331-2332,N is translated by the MMU 2339 into a real address.

In one embodiment, the same set of registers 2345 is duplicated for each graphics processing engine 2331-2332, N and/or graphics accelerator module 2346 and may be initialized by a hypervisor or operating system. Each of these duplicated registers may be included in an accelerator integration section 2390. Exemplary registers that can be initialized by a hypervisor are shown in Table 1. Table 1 - Registers initialized by the hypervisor 1 Tabs for controlling sections 2 Area pointer for processes scheduled with real addresses (RA). 3 Authority mask override register 4 Offset of an interrupt vector table entry 5 Delimitation of an interrupt vector table entry 6 state register 7 ID of a logical partition 8th Pointer to a real address (RA) hypervisor accelerator usage record 9 Register for memory description

Example registers that can be initialized by an operating system are shown in Table 2. Table 2 - Registers initialized by the operating system 1 Process and thread identification 2 Pointer for save/restore context with effective address (EA) 3 Pointer to usage record of a virtual address accelerator (VA) 4 Virtual address (VA) memory segment table pointer 5 mask of authority 6 work descriptor

In one embodiment, each WD 2384 is specific to a particular graphics accelerator module 2346 and/or graphics processing engines 2331-2332,N. It contains all of the information that a graphics processing engine 2331-2332,N needs, or can, to perform its work be a pointer to a memory location where an application has established a command queue for work to be performed.

23E Figure 12 shows additional details for an example embodiment of a common model. This embodiment includes a hypervisor real address space 2398 in which a process item list 2399 is stored. The hypervisor's physical address space 2398 is accessible through a hypervisor 2396 that virtualizes the graphics accelerator engines for the operating system 2395 .

In at least one embodiment, shared programming models allow all or a subset of processes from all or a subset of partitions in a system to use a graphics accelerator engine 2346. There are two programming models in which the graphics accelerator engine 2346 is shared between multiple processes and partitions: epoch-wise shared and graphics-directed shared.

In this model, the system hypervisor 2396 contains the graphics accelerator module 2346 and makes its functionality available to all operating systems 2395 . For a graphics accelerator 2346 to support virtualization through the system hypervisor 2396, the graphics accelerator 2346 may meet the following conditions: 1) An application's job request must be autonomous (i.e., state need not be maintained between jobs), or the graphics accelerator 2346 must provide a mechanism provide for backing up and restoring the context. 2) For an application's job request, the Graphics Accelerator 2346 engine guarantees that it will be completed in a specified amount of time, including any translation errors, or the Graphics Accelerator 2346 engine provides the ability to anticipate the processing of a job. 3) The Graphics Accelerator Module 2346 must guarantee inter-process balance when operating in a directed shared programming model.

In at least one embodiment, the 2380 application is required to invoke an operating system 2395 system call with a 2346 graphics accelerator engine, a work descriptor (WD), an authority mask register (AMR) value, and a context save (CSRP) value /Restore Area Pointer). In at least one embodiment, the graphics accelerator module 2346 type describes a targeted acceleration function for a system call. In at least one embodiment, the type of graphics accelerator module 2346 may be a native value. In at least one embodiment, the WD is formatted specifically for the graphics accelerator module 2346 and may be in the form of a graphics accelerator module 2346 command, an effective address pointer to a user-defined structure, an effective address pointer to a command queue, or any other data structure describing the graphics accelerator module 2346 to be executed works are available. In one embodiment, an AMR value is an AMR state to use for a current process. In at least one embodiment, a value passed to an operating system is similar to an application setting an AMR. If the accelerator integration circuit 2336 and graphics accelerator module 2346 implementations do not support a User Authority Mask Override Register (UAMOR), an operating system can apply a current UAMOR value to an AMR value before passing an AMR in a hypervisor call. Hypervisor 2396 can optionally apply a current AMOR (Authority Mask Override Register) value before committing an AMR in process element 23 83 . In In at least one embodiment, CSRP is one of registers 2345 that contains an effective address of a region in an application's address space 2382 for graphics accelerator module 2346 to save and restore context state. This pointer is optional if no state needs to be saved between jobs or if a job is to be brought forward. In at least one embodiment, the context save/restore area may be fixed system memory.

Upon receiving a system call, the operating system 2395 can verify that the application 2380 has been registered and authorized to use the graphics accelerator module 2346 . Operating system 2395 then calls hypervisor 2396 with the information listed in Table 3. Table 3 - Invocation Parameters from OS to Hypervisor 1 Work Descriptor (WD) 2 Authority Mask Register Value (AMR) (potentially masked) 3 Context Save/Restore Area Pointer (CSRP) for context save/restore with effective address (EA) 4 Process ID (PID) and optional Thread ID (TID) 5 Accelerator utilization record pointer (AURP) with virtual address (VA) 6 Stored segment table pointer (SSTP) pointer with virtual address 7 Logical interrupt service number (LISN)

Upon receiving a hypervisor call, the hypervisor 2396 verifies that the operating system 2395 is registered and has been granted permission to use the graphics accelerator module 2346 . Hypervisor 2396 then inserts process item 2383 into a process item linked list for a corresponding graphics accelerator engine 2346 type. A process item may include the information listed in Table 4. Table 4 - Process Item Information 1 Work Descriptor (WD) 2 Authority Mask Register Value (AMR) (potentially masked) 3 Context Save/Restore Area Pointer (CSRP) for context save/restore with effective address (EA) 4 Process ID (PID) and optional Thread ID (TID) 5 Accelerator utilization record pointer (AURP) with virtual address (VA) 6 Stored segment table pointer (SSTP) pointer with virtual address 7 Logical interrupt service number (LISN) 8th Interrupt vector table derived from hypervisor invocation parameters 9 Value of a status register (SR) 10 Logical Partition ID (LPID) 11 Real address (RA) hypervisor accelerator usage record pointer 12 Storage Descriptor Register (SDR)

In at least one embodiment, the hypervisor initializes a plurality of registers 2390 of accelerator integration sections 2390-2345.

As in 23F As shown, in at least one embodiment, a unified memory is used that is addressable via a shared virtual memory address space used to access physical processor memory 2301-2302 and GPU memory 2320-2323. In this implementation, operations performed on GPUs 2310-2313 use a shared virtual/effective memory address space to access processor memories 2301-2302 and vice versa, simplifying programmability. In one embodiment, a first portion of a virtual/effective address space is allocated to processor memory 2301, a second portion to second processor memory 2302, a third portion to GPU memory 2320, and so on. In at least one embodiment, this distributes an entire virtual/effective memory space (sometimes referred to as effective address space) across each of processor memory 2301-2302 and GPU memory 2320-2323, giving each processor or GPU access to each physical memory with a virtual memory associated with that memory address can access.

In one embodiment, policy/coherency management circuitry 2394A-2394E within one or more MMUs 2339A-2339E provides cache coherency between the caches of one or more host processors (e.g., 2305) and GPUs 2310- 2313 secure and implements default techniques specifying physical memory in which to store certain types of data. While multiple constraints of constraint/coherency management circuitry 2394A-2394E in 23F 1, the circuitry for constraint/coherency management may be implemented within an MMU by one or more host processors 2305 and/or within the accelerator integrated circuit 2336.

In one embodiment, the GPU-bound memory 2320-2323 may be allocated as part of the system memory and accessed using shared virtual memory (SVM) technology, but without incurring the performance penalties associated with full system cache coherency. In at least one embodiment, the ability to access GPU-bound memory 2320-2323 as part of system memory without the cumbersome cache coherency overhead is an advantageous operating environment for offloading to GPUs. This arrangement allows host processor 2305 software to construct operands and access calculation results without the overhead of conventional I/O DMA data copies. Such traditional copies include driver calls, interrupts, and memory mapped I/O (MMIO) accesses, all of which are inefficient compared to simple memory accesses. In at least one embodiment, the ability to access GPU-bound memory 2320-2323 without cache coherency overhead may be critical to offloaded computation execution time. For example, in cases with significant streaming write memory traffic, the cache coherence overhead can significantly reduce an effective write bandwidth seen by a GPU 2310-2313. In at least one embodiment, operand construction efficiency, result access efficiency, and GPU computation efficiency may play a role in determining GPU offloading effectiveness.

In at least one embodiment, the selection of the GPU policy and the host processor policy is controlled by a policy tracker data structure. For example, a bias table can be a page-granular structure (i.e., controlled with the granularity of a memory page) containing 1 or 2 bits per GPU-attached memory page. In at least one embodiment, a default table may be implemented in a stolen memory area of one or more GPU-bound memories 2320-2323 with or without a default cache in GPU 2310-2313 (e.g., to cache frequently/recently used entries of a default table). Alternatively, an entire default table can be maintained within a graphics processor.

In at least one embodiment, a default table entry associated with each access to GPU-bound memory 2320-2323 is accessed prior to the actual access to GPU memory, causing the following operations. First, local requests from the GPU 2310-2313 that find their page in the GPU policy are routed directly to a corresponding GPU memory 2320-2323. Local requests from a graphics processor that find their page in the host memory default are forwarded to processor 2305 (eg, over a high-speed link, as described above). In one embodiment, requests from processor 2305 that find a requested page in the host processor bias complete a request like a normal memory read. Alternatively, requests directed to a GPU designated page may be forwarded to GPU 2310-2313. In at least one embodiment, a GPU may then commit a page to host processor default when it is not using a page. In at least one embodiment, a page's default state may be set either by a software-based mecha mechanism, a hardware-assisted software-based mechanism, or, for a limited set of cases, a purely hardware-based mechanism.

One mechanism for changing the default state uses an API call (e.g. OpenCL), which in turn calls a GPU's device driver, which in turn sends a message (or queues a command description) to a GPU to instruct it to change a default state and to perform a cache flush operation in a host for some transitions. In at least one embodiment, the cache flush operation is used for a transition from host processor 2305 default to GPU default, but not for a reverse transition.

In one embodiment, cache coherency is maintained by temporarily not caching GPU-given pages from host processor 2305 . To access these pages, processor 2305 may request access from GPU 2310, which may not grant access immediately. Therefore, in order to reduce communication between processor 2305 and GPU 2310, it is beneficial to ensure that GPU-given pages are those required by a GPU but not required by the host processor 2305, and vice versa.

The hardware structure(s) 1515 is/are used to implement one or more embodiments. Details regarding the hardware structure(s) 1515 are in connection with here 15A and or 15B provided.

24 FIG. 1 shows exemplary integrated circuits and associated graphics processors that may be fabricated using one or more IP cores, according to various embodiments described herein. In addition to what is shown herein, other logic and circuitry may be included in at least one embodiment, including additional graphics processors/cores, peripheral interface controllers, or general purpose processor cores.

24 14 is a block diagram depicting an exemplary system on an on-chip integrated circuit 2400 that may be fabricated with one or more IP cores, according to at least one embodiment. In at least one embodiment, the integrated circuit 2400 includes one or more application processor(s) 2405 (e.g., CPUs), at least one graphics processor 2410, and may additionally include an image processor 2415 and/or a video processor 2420, each of these processors being a modular IP core can. In at least one embodiment, integrated circuit 2400 includes peripheral or bus logic including USB controller 2425, UART controller 2430, SPI/SDIO controller 2435, and I.sup.2S/I.sup.2C controller 2440. In In at least one embodiment, the integrated circuit 2400 may include a display device 2445 coupled to one or more of the following controllers: a High-Definition Multimedia Interface (HDMI) controller 2450 and a Mobile Industry Processor Interface (MIPI) display interface 2455 In at least one embodiment, the memory may be provided by a flash memory subsystem 2460, which includes flash memory and a flash memory controller. In at least one embodiment, the memory interface may be provided via a controller 2465 to access SDRAM or SRAM memory devices. In at least one embodiment, some integrated circuits additionally include an embedded security engine 2470.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, the inference and/or training logic 1515 in the integrated circuit 2400 may be used to infer or predict operations based, at least in part, on weight parameters obtained using training operations with neural networks, functions and/or neural network architectures or use cases of neural networks described herein.

25A-25B 12 show exemplary integrated circuits and associated graphics processors that may be fabricated with one or more IP cores according to various embodiments. In addition to what is shown herein, other logic and circuitry may be included in at least one embodiment, including additional graphics processors/cores, peripheral interface controllers, or general purpose processor cores.

25A-25B 12 show, as block diagrams, exemplary graphics processors for use in a SoC in accordance with the exemplary embodiments described here. 25A FIG. 25 shows an example integrated circuit graphics processor 2510 on a system-on-a-chip that can be fabricated with one or more IP cores, according to at least one embodiment. 25B FIG. 2540 shows an additional example integrated circuit graphics processor 2540 on a system-on-a-chip that may be fabricated using one or more IP cores, in accordance with at least one embodiment. In at least one embodiment, the graphics processor 2510 is from 25A a graphics processor core with low power consumption. In at least one embodiment, the graphics processor is 2540 from 25B a GPU core with higher performance. In at least one embodiment, each of the graphics processors 2510, 2540 may be a variant of the graphics processor 2410 24 be.

In at least one embodiment, graphics processor 2510 includes a vertex processor 2505 and one or more fragment processors 2515A-2515N (e.g., 2515A, 2515B, 2515C, 2515D, through 2515N-1 and 2515N). In at least one embodiment, graphics processor 2510 may execute different shader programs via separate logic such that vertex processor 2505 is optimized for executing vertex shader operations, while one or more fragment processor(s) 2515A-2515N execute fragment shader operations (e.g., Pixel) for fragment or pixel shader programs. In at least one embodiment, vertex processor 2505 performs a vertex processing stage of a 3D graphics pipeline and generates primitives and vertex data. In at least one embodiment, fragment processor(s) 2515A-2515N uses the primitive and vertex data generated by vertex processor 2505 to generate a frame buffer that is displayed on a display device. In at least one embodiment, the fragment processor(s) 2515A-2515N is/are optimized for executing fragment shader programs as provided in an OpenGL API that can be used to perform similar operations as a Run pixel shader programs as provided in a direct 3D API.

In at least one embodiment, graphics processor 2510 additionally includes one or more memory management units (MMUs) 2520A-2520B, cache(s) 2525A-2525B, and circuit interconnect(s) 2530A-2530B. In at least one embodiment, one or more MMU(s) 2520A-2520B are provided for mapping virtual to physical addresses for graphics processor 2510, including for vertex processor 2505 and/or fragment processor(s). ) 2515A-2515N, which may reference vertex or image/texture data stored in memory in addition to vertex or image/texture data stored in one or more cache(s) 2525A-2525B. In at least one embodiment, one or more MMU(s) 2520A-2520B can be synchronized with other MMUs within the system, including one or more MMUs that interface with one or more application processor(s) 2405, image processor(s) 2415, and/or video processor (en) 2420 out 24 are associated so that each processor 2405-2420 can participate in a shared or unified virtual memory system. In at least one embodiment, circuit interconnect(s) 2530A-2530B enable graphics processor 2510 to connect to other IP cores within the SoC, either through an internal bus of the SoC or through a direct connection.

In at least one embodiment, graphics processor 2540 includes one or more MMU(s) 2520A-2520B, caches 2525A-2525B, and graphics processor 2510 circuit interconnects 2530A-2530B 25A . In at least one embodiment, graphics processor 2540 includes one or more shading core(s) 2555A-2555N (e.g., 2555A, 2555B, 2555C, 2555D, 2555E, 2555F through 2555N-1, and 2555N) that provide a unified shading core architecture in which a single core or type or core can execute all kinds of programmable shading code, including shading program code to implement vertex shading, fragment shading, and/or computational shading. In at least one embodiment, a number of shading kernels may vary. In at least one embodiment, the graphics processor 2540 includes an inter-core task manager 2545 that operates as a thread dispatcher to dispatch threads of execution to one or more shading cores 2555A-2555N, and a tiling unit 2558 for Accelerate tiling operations for tile-based rendering, in which rendering operations for a scene are partitioned in image space, for example to exploit local spatial coherence within a scene or to optimize the use of internal caches.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, inference and/or training logic 1515 in integrated circuit 25A and/or 25B may be used to infer or predict operations based at least in part on weight parameters obtained using training operations with neural networks, functions, and/or or neural network architectures or neural network use cases described herein.

26A-26B 12 show further example graphics processor logic consistent with the embodiments described herein. 26A FIG. 1 shows a graphics core 2600, which in at least one embodiment may include the graphics processor 2410 of FIG 24 may comprise and in at least one embodiment a unified shading kernel 2555A-2555N as in 25B can represent. 26B Figure 2630 shows a highly parallel general purpose graphics processing unit 2630, suitable for deployment on a multi-chip module, in at least one embodiment.

In at least one embodiment, the graphics core 2600 includes a shared instruction cache 2602, a texture unit 2618, and a cache/shared memory 2620 that are shared among the execution resources in the graphics core 2600. In at least one embodiment, graphics core 2600 may include multiple slices 2601A-2601N or partitions for each core, and a graphics processor may include multiple instances of graphics core 2600. Sections 2601A-2601N may include support logic including a local instruction cache 2604A-2604N, a thread scheduler 2606A-2606N, a thread dispatcher 2608A-2608N, and a set of registers 2610A-2610N. In at least one embodiment, sections 2601A-2601N may include a set of additional functional units (AFUs 2612A-2612N), floating point units (FPU 2614A-2614N), integer arithmetic logic units (ALUs 2616-2616N), address calculation units (ACU 2613A-2613N), double precision floating point units (DPFPU 2615A-2615N) and matrix processing units (MPU 2617A-2617N).

In at least one embodiment, FPUs 2614A-2614N can perform single-precision (32-bit) and half-precision (16-bit) floating-point operations, while DPFPUs 2615A-2615N can perform double-precision (64-bit) floating-point operations. In at least one embodiment, ALUs 2616A-2616N can perform variable-precision integer operations with 8-bit, 16-bit, and 32-bit precision and can be configured for mixed-precision operations. In at least one embodiment, MPUs 2617A-2617N may also be configured for mixed-precision matrix operations, including half-precision floating-point and 8-bit integer operations. In at least one embodiment, MPUs 2617-2617N may perform a variety of matrix operations to accelerate machine learning application frameworks, including support for accelerated general matrix to matrix multiplication , GEMM). In at least one embodiment, AFUs 2612A-2612N can perform additional logical operations not supported by floating point or integer units, including trigonometric operations (e.g., sine, cosine, etc.).

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, the inference and/or training logic 1515 in Graphics Core 2600 for inferring or predicting operations may be based, at least in part, on weight parameters obtained using training operations with neural networks, functions and/or neural network architectures, or as described herein Use cases of neural networks were calculated can be used.

26B Figure 12 shows a general purpose processing unit (GPGPU) 2630 configurable to perform highly parallel computational operations by an array of graphics processing units in at least one unit, in at least one embodiment. In at least one embodiment, the GPGPU 2630 can be connected directly to other instances of the GPGPU 2630 to create a multi-GPU cluster to improve training speed for deep neural networks. In at least one embodiment, GPGPU 2630 includes a host interface 2632 for connecting to a host processor enable. In at least one embodiment, host interface 2632 is a PCI Express interface. In at least one embodiment, the host interface 2632 may be a proprietary communication interface or communication structure. In at least one embodiment, the GPGPU 2630 receives commands from a host processor and uses a global scheduler 2634 to schedule the threads of execution associated with those commands onto a set of compute clusters 2636A-2636H. In at least one embodiment, compute clusters 2636A-2636H share a cache 2638. In at least one embodiment, cache 2638 may serve as a higher level cache for caches within compute clusters 2636A-2636H.

In at least one embodiment, GPGPU 2630 includes memory 2644A-2644B coupled to compute clusters 2636A-2636H via a set of memory controllers 2642A-2642B. In at least one embodiment, memory 2644A-2644B may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM). ), including graphics memory with double data rate (GDDR).

In at least one embodiment, compute clusters 2636A-2636H each include a set of graphics cores, such as graphics core 2600 of FIG 26A , which can include multiple types of integer and floating-point logic units that can perform arithmetic operations with a range of precisions that are also suitable for machine learning. For example, in at least one embodiment, at least a subset of floating point units in each of the compute clusters 2636A-2636H may be configured to perform 16-bit or 32-bit floating point operations, while another subset of floating point units may be configured to perform 64-bit floating point operations can be configured.

In at least one embodiment, multiple instances of the GPGPU 2630 can be configured to operate as a compute cluster. In at least one embodiment, the communication used by compute clusters 2636A-2636H for synchronization and data exchange varies from embodiment to embodiment. In at least one embodiment, multiple instances of GPGPU 2630 communicate through host interface 2632. In at least one embodiment, GPGPU 2630 includes an I/O hub 2639 that couples GPGPU 2630 to a GPU connection 2640 that connects directly to other instances of the GPGPU 2630. In at least one embodiment, the GPU link 2640 is coupled to a dedicated GPU-to-GPU bridge that enables communication and synchronization between multiple GPGPU 2630 instances. In at least one embodiment, GPU connection 2640 is coupled to a high-speed connection to send and receive data to other GPGPUs or parallel processors. In at least one embodiment, multiple instances of GPGPU 2630 reside in separate computing systems and communicate through a network device accessible through host interface 2632 . In at least one embodiment, GPU connection 2640 may be configured to allow connection to a host processor in addition to or as an alternative to host interface 2632 .

In at least one embodiment, the GPGPU 2630 can be configured to train neural networks. In at least one embodiment, GPGPU 2630 may be used within an inference platform. In at least one embodiment where GPGPU 2630 is used for inference, GPGPU may include fewer compute clusters 2636A-2636H than when GPGPU is used to train a neural network. In at least one embodiment, the storage technology associated with memory 2644A-2644B may differ between inference and training configurations, with higher bandwidth storage technologies being used for training configurations. In at least one embodiment, the inference configuration of the GPGPU 2630 may support specific instruction inference. For example, in at least one embodiment, an inference configuration may provide support for one or more 8-bit integer dot product instructions that may be used in inference operations for deployed neural networks.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. in minutes In at least one embodiment, the inference and/or training logic 1515 in graphics core 2600 can infer or predict operations based, at least in part, on weight parameters obtained using training operations with neural networks, functions and/or neural network architectures, or use cases described herein calculated by neural networks can be used.

27 12 is a block diagram depicting a computer system 2700 in accordance with at least one embodiment. In at least one embodiment, computing system 2700 includes a processing subsystem 2701 having one or more processors 2702 and system memory 2704 communicating over an interconnect path that may include a memory hub 2705 . In at least one embodiment, the memory hub 2705 may be a separate component within a chipset component or integrated into one or more processor(s) 2702 . In at least one embodiment, storage hub 2705 is coupled to I/O subsystem 2711 via connection 2706 . In at least one embodiment, I/O subsystem 2711 includes an I/O hub 2707 that may enable computing device 2700 to receive input from one or more input devices 2708 . In at least one embodiment, I/O hub 2707 may enable a controller for a display, which may be included in one or more processors 2702, to provide outputs to one or more display devices 2710A. In at least one embodiment, one or more display devices 2710A coupled to I/O hub 2707 may include a local, internal, or embedded display device.

In at least one embodiment, processing subsystem 2701 includes one or more parallel processors 2712 coupled to memory hub 2705 via bus or other connection 2713 . In at least one embodiment, communication link 2713 may be any number of standards-based communication link technologies or protocols, such as but not limited to PCI Express, or a proprietary communication interface or communication structure. In at least one embodiment, parallel processor(s) 2712 form a computational parallel or vector processing system that may include a large number of computational cores and/or processing clusters, such as a many integrated core (MIC) processor. . In at least one embodiment, parallel processor(s) 2712 form a graphics processing subsystem that can output pixels to one of display device(s) 2710A coupled via I/O hub 2707 . In at least one embodiment, parallel processor(s) 2712 may also include a controller and display interface (not shown) to enable direct connection to display device(s) 2710B.

In at least one embodiment, a system storage device 2714 may be coupled to I/O hub 2707 to provide a storage mechanism for computer system 2700. In at least one embodiment, an I/O switch 2716 may be used to provide an interface mechanism that enables connections between the I/O hub 2707 and other components, such as a network adapter 2718 and/or a wireless network adapter 2719, that may be used in can be integrated into the platform, and various other devices that can be added via add-on device(s) 2720. In at least one embodiment, network adapter 2718 may be an Ethernet adapter or other wired network adapter. In at least one embodiment, wireless network adapter 2719 may include one or more Wi-Fi, Bluetooth, near field communication (NFC), or other network devices that include one or more wireless radios.

In at least one embodiment, computing system 2700 may include other components not explicitly shown, including USB or other port connections, optical storage drives, video capture devices, and the like that may also connect to I/O hub 2707 . In at least one embodiment, communication paths connecting various components in 27 interconnect, be implemented using any suitable protocols, such as PCI (Peripheral Component Interconnect) based protocols (e.g. PCI-Express) or other bus or point-to-point communication interfaces and/or protocol(s) such as NV- Link high-speed connections or connection protocols.

In at least one embodiment, one or more parallel processor(s) 2712 includes circuitry optimized for graphics and video processing, including, for example, a 12 and 12 illustrate a graphics processing unit (GPU). In at least one embodiment, parallel processor(s) 2712 include circuitry optimized for general purpose processing. In at least one embodiment, components of computing system 2700 may be integrated with one or more other system elements on a single integrated circuit. For example, in at least one embodiment, one or more parallel processor(s) 2712, memory hub 2705, processor(s) 2702, and I/O hub 2707 may be integrated into a system-on-a-chip (SoC) integrated circuit. In at least one embodiment, components of computing system 2700 may be integrated into a single chassis to form a system in package (SIP) configuration. In at least one embodiment, at least a portion of the components of computing system 2700 may be integrated into a multi-chip module (MCM) that may be interconnected with other multi-chip modules to form a modular computing system.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, the inference and/or training logic 1515 in Graphics Core 2600 for inferring or predicting operations may be based, at least in part, on weight parameters obtained using training operations with neural networks, functions and/or neural network architectures, or as described herein Use cases of neural networks were calculated can be used.

28A 12 shows a parallel processor 2800 according to at least one embodiment. In at least one embodiment, various components of parallel processor 2800 may be implemented using one or more integrated circuit devices, such as programmable processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGA). The parallel processor 2800 illustrated in at least one embodiment is a variant of one or more in 27 shown parallel processor(s) 2712 according to an example embodiment.

In at least one embodiment, the parallel processor 2800 includes a parallel processing unit 2802. In at least one embodiment, the parallel processing unit 2802 includes an I/O unit 2804 that enables communication with other devices, including other instances of the parallel processing unit 2802. In at least one embodiment, I/O unit 2804 may be directly connected to other devices. In at least one embodiment, I/O unit 2804 is connected to other devices through a hub or switch interface, such as storage hub 2705. In at least one embodiment, connections between the storage hub 2705 and the I/O device 2804 form a communication link 2713. In at least one embodiment, the I/O device 2804 connects to a host interface 2806 and a storage crossbar 2816 where host interface 2806 receives commands directed to performing processing operations and memory crossbar 2816 receives commands directed to performing memory operations.

In at least one embodiment, when the host interface 2806 receives a command buffer via the I/O device 2804, the host interface 2806 may direct operations to a front end 2808 for execution of those commands. In at least one embodiment, the front end 2808 is coupled to a scheduler 2810 configured to dispatch commands or other work to a processing cluster array 2812 . In at least one embodiment, scheduler 2810 ensures processing cluster array 2812 is properly configured and in a valid state before dispatching tasks to processing cluster array 2812 of processing cluster array 2812 . In at least one embodiment, scheduler 2810 is implemented via firmware logic running on a microcontroller. In at least one embodiment, the scheduler 2810 implemented in a microcontroller is configurable to perform complex scheduling and work distribution operations at coarse and fine granularity, enabling fast pre-allocation and context switching of threads running on the processing array 2812. In at least one embodiment, the host software may expose workloads for scheduling on the array 2812 via one of a plurality of graphics processing doorbells. In at least one embodiment, workloads may then be automatically distributed to the processing array 2812 across the processing array 2812 by scheduler 2810 logic within a microcontroller that includes the scheduler 2810 .

In at least one embodiment, processing cluster array 2812 may include up to "N" processing clusters (e.g., cluster 2814A, cluster 2814B, up to cluster 2814N).

In at least one embodiment, each cluster 2814A-2814N of processing cluster array 2812 can execute a large number of concurrent threads. In at least one embodiment, scheduler 2810 may allocate work to clusters 2814A-2814N of processing cluster array 2812 using different scheduling and/or work distribution algorithms, which may vary depending on the workload involved for each program or computation type. In at least one embodiment, scheduling may be handled dynamically by scheduler 2810 or assisted in part by compiler logic during compilation of program logic configured for execution by processing cluster array 2812 . In at least one embodiment, different clusters 2814A-2814N of processing cluster array 2812 may be assigned to process different types of programs or to perform different types of calculations.

In at least one embodiment, the processing cluster array 2812 can be configured to perform various types of parallel processing operations. In at least one embodiment, the processing cluster array 2812 is configured to perform general parallel computing operations. For example, in at least one embodiment, the processing cluster array 2812 may include logic to perform processing tasks including filtering video and/or audio data, performing modeling operations including physical operations, and performing data transformations.

In at least one embodiment, processing cluster array 2812 is configured to perform parallel graphics processing operations. In at least one embodiment, the processing cluster array 2812 may include additional logic to support the execution of such graphics processing operations, including but not limited to texture scanning logic for performing texture operations, as well as tessellation logic and other vertex processing logic. In at least one embodiment, the processing cluster array 2812 can be configured to execute graphics processing-related shading programs such as vertex shading, tessellation shading, geometry shading, and pixel shading. In at least one embodiment, parallel processing unit 2802 may transfer data from system memory through I/O unit 2804 for processing. In at least one embodiment, the transferred data may be stored in on-chip memory (e.g., parallel processor memory 2822) during processing and then written back to system memory.

In at least one embodiment, when using the parallel processing unit 2802 to perform graphics processing, the scheduler 2810 can be configured to split a processing workload into approximately equally sized tasks to better distribute graphics processing operations across multiple clusters 2814A-2814N of the processing cluster array 2812 enable. In at least one embodiment, portions of the processing cluster array 2812 can be configured to perform different types of processing. For example, in at least one embodiment, a first part can be configured to perform vertex shading and topology generation, a second part can be configured to perform tessellation and geometry shading, and a third part can be configured to perform performs pixel shading or other screen space operations to produce a rendered image for display. In at least one embodiment, intermediate data generated by one or more of clusters 2814A-2814N may be stored in buffers to facilitate the transfer of intermediate data between clusters 2814A-2814N for further processing.

In at least one embodiment, processing cluster array 2812 may receive processing tasks to be performed via scheduler 2810 , which receives processing task definition commands from front end 2808 . In at least one embodiment, the processing tasks may include indices of the data to be processed, e.g., surface (patch) data, primitive data, vertex data, and/or pixel data, as well as state parameters and instructions that define how the data should be processed (e.g., which program to run shall be). In at least one embodiment, scheduler 2810 may be configured to fetch indices corresponding to tasks or to receive indices from front end 2808 . In at least one embodiment, the front end 2808 can be configured to ensure that the processing cluster array 2812 is on a valid state before initiating a workload specified by incoming command buffers (e.g., batch buffer, push buffer, etc.).

In at least one embodiment, each of one or more instances of parallel processing unit 2802 may be coupled to parallel processor memory 2822 . In at least one embodiment, parallel processor memory 2822 may be accessed via memory crossbar 2816, which may receive memory requests from processing cluster array 2812 as well as I/O unit 2804. In at least one embodiment, parallel processor memory 2822 may be accessed through a memory interface 2818, via memory crossbar 2816. In at least one embodiment, memory interface 2818 may include multiple partition units (e.g., partition unit 2820A, partition unit 2820B, via partition unit 2820N), each with may be coupled to a portion (e.g., memory unit) of parallel processor memory 2822. In at least one embodiment, a number of partition units 2820A-2820N are configured to correspond to a number of storage units such that a first partition unit 2820A has a corresponding first storage unit 2824A, a second partition unit 2820B has a corresponding storage unit 2824B, and an Nth partition unit 2820N has a corresponding Nth storage unit 2824N. In at least one embodiment, a number of partition units 2820A-2820N may not equal a number of storage devices.

In at least one embodiment, memory units 2824A-2824N may comprise various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory (SGRAM), including double data rate ( Graphics Double Data Rate, GDDR). In at least one embodiment, memory units 2824A-2824N may also include 3D stacked memory, including but not limited to high-bandwidth memory (HBM). In at least one embodiment, render targets, such as frame buffers or texture maps, may be stored across memory units 2824A-2824N such that partition units 2820A-2820N may write portions of each render target in parallel to utilize the available bandwidth of parallel processor memory 2822 to use efficiently. In at least one embodiment, a local instance of parallel processor memory 2822 may be eliminated in favor of a unified memory design that uses system memory in conjunction with local cache memory.

In at least one embodiment, each of clusters 2814A-2814N of processing cluster array 2812 may process data that is written to one of storage units 2824A-2824N in parallel processor memory 2822. In at least one embodiment, the memory crossbar 2816 can be configured so that an output of each cluster 2814A-2814N can be transferred to any partition unit 2820A-2820N or to another cluster 2814A-2814N that can perform additional processing operations on an output. In at least one embodiment, each cluster 2814A-2814N can communicate with storage interface 2818 via storage crossbar 2816 to read from and write to various external storage devices. In at least one embodiment, memory crossbar 2816 connects to memory interface 2818 to communicate with I/O device 2804, and connects to a local instance of parallel processor memory 2822, thereby connecting processing units within different processing clusters 2814A-2814N to system memory or another memory that is not local to the parallel processing unit 2802. In at least one embodiment, memory crossbar 2816 may use virtual channels to separate traffic flows between clusters 2814A-2814N and partition units 2820A-2820N.

In at least one embodiment, multiple instances of parallel processing unit 2802 may be provided on a single add-on card, or multiple add-on cards may be interconnected. In at least one embodiment, different instances of parallel processing unit 2802 may be configured to work together even if the different instances have different numbers of processing cores, different amounts of local parallel processor memory, and/or other configuration differences. For example, in at least one embodiment, some instances of parallel processing unit 2802 may include higher precision floating point units compared to other instances. In at least one embodiment, systems containing one or more instances of parallel processing unit 2802 or parallel processor 2800 may be implemented in a variety of configurations and form factors, including but not limited to desktop, laptop, or handheld personal computers, servers, workstations , game consoles and/or embedded systems.

28B FIG. 28 shows a block diagram of a partition unit 2820 according to at least one embodiment. In at least one embodiment, partition unit 2820 is an example of one of partition units 2820A-2820N 28A . In at least one embodiment, the partition unit 2820 includes an L2 cache 2821, a frame buffer interface 2825, and a ROP 2826 (raster operations unit). L2 cache 2821 is a read/write cache configured to perform load and store operations received from memory crossbar 2816 and ROP 2826. In at least one embodiment, read errors and urgent writeback requests are issued from L2 cache 2821 to framebuffer interface 2825 for processing. In at least one embodiment, updates may also be sent via framebuffer interface 2825 to a framebuffer for processing. In at least one embodiment, frame buffer interface 2825 is coupled to one of the storage units in parallel processor memory, such as storage units 2824A-2824N of FIG 28 (e.g. in parallel processor memory 2822).

In at least one embodiment, ROP 2826 is a processing unit that performs raster operations such as stencil, z-test, blending, and the like. In at least one embodiment, ROP 2826 then outputs processed graphics data, which is stored in graphics memory. In at least one embodiment, ROP 2826 includes compression logic to compress depth or color data that is written to memory and to decompress depth or color data that is read from memory. In at least one embodiment, the compression logic may be lossless compression logic that uses one or more of a variety of compression algorithms. The type of compression performed by ROP 2826 can vary based on statistical properties of the data being compressed. For example, in at least one embodiment, delta color compression is performed on depth and color data on a per Ktil basis.

In at least one embodiment, ROP 2826 is located in each processing cluster (eg, clusters 2814A-2814N of 28 ) instead of contained in partition unit 2820. In at least one embodiment, read and write requests for pixel data are transmitted across memory crossbar 2816 instead of pixel fragment data. In at least one embodiment, processed graphics data may be displayed on a display device, eg, one or more display devices 2710 27 , be forwarded for further processing by processor(s) 2702 or for further processing by one of the processing units within parallel processor 2800 28A to get redirected.

28C 12 shows a block diagram of a processing cluster 2814 within a parallel processing unit, according to at least one embodiment. In at least one embodiment, a processing cluster is an instance of one of processing clusters 2814A-2814N 28 . In at least one embodiment, the processing cluster 2814 can be configured to execute many threads in parallel, where the term "thread" refers to an instance of a particular program executing on a particular set of input data. In at least one embodiment, single-instruction, multiple-data (SIMD) instruction issue techniques are used to support the parallel execution of a large number of threads without providing multiple independent instruction units. In at least one embodiment, single-instruction multiple-thread (SIMT) techniques are used to support the parallel execution of a large number of commonly synchronized threads using a common instruction unit configured to issue instructions to a set of processing machines within each individual processing cluster.

In at least one embodiment, the operation of processing cluster 2814 may be controlled via a pipeline manager 2832 that distributes processing tasks to SIMT parallel processors. In at least one embodiment, pipeline manager 2832 receives instructions from scheduler 2810 from 28 and manages the execution of these instructions via a graphics multiprocessor 2834 and/or a texture unit 2836. In at least one embodiment, the graphics multiprocessor 2834 is an exemplary case of a SIMT parallel processor. However, in at least one embodiment, processing cluster 2814 may include different types of SIMT parallel processors of different architectures. In at least one embodiment, one or more instances of graphics multiprocessor 2834 may be included in a processing cluster 2814. In at least one embodiment, the graphics multiprocessor 2834 can process data, and a data crossbar 2840 can be used to distribute processed data to any of a number of possible destinations, including other shading engines. In at least one embodiment, the Pipeline manager 2832 facilitates the distribution of processed data by specifying destinations for processed data to be distributed across data crossbar 2840 .

In at least one embodiment, each graphics multiprocessor 2834 within the processing cluster 2814 may include an identical set of functional execution logic (e.g., arithmetic logic units, load memory units). In at least one embodiment, the functional execution logic can be configured in a pipelined fashion, where new instructions can be issued before the previous instructions complete. In at least one embodiment, the functional execution logic supports a variety of operations, including integer and floating point arithmetic, comparison operations, Boolean operations, bit shifting, and computation of various algebraic functions. In at least one embodiment, the same hardware with functional units can be used to perform different operations, and there can be any combination of functional units.

In at least one embodiment, the instructions submitted to processing cluster 2814 form a thread. In at least one embodiment, a set of threads executing across a set of parallel processing engines forms a thread group. In at least one embodiment, a thread group forms a program with different input data. In at least one embodiment, each thread within a thread group may be mapped to a different processing engine within a 2834 graphics multiprocessor. In at least one embodiment, a thread group may include fewer threads than a number of processing engines within a graphics multiprocessor 2834. In at least one embodiment, when a thread group includes fewer threads than a number of processing engines, one or more of the processing engines to be idle during the cycles in which this thread group is processed. In at least one embodiment, a thread group may also include more threads than a number of processing engines within the graphics multiprocessor 2834. In at least one embodiment, when a thread group includes more threads than the number of processing engines within the graphics multiprocessor 2834, the processing may be performed over consecutive clock cycles in at least one embodiment. In at least one embodiment, multiple groups of threads may execute concurrently on a 2834 graphics multiprocessor.

In at least one embodiment, the graphics multiprocessor 2834 includes internal cache memory for performing load and store operations. In at least one embodiment, the graphics multiprocessor 2834 may forego an internal cache memory and use cache memory (eg, L1 cache 2848) within the processing cluster 2814. In at least one embodiment, each graphics multiprocessor 2834 also has access to L2 caches within partition units (eg, partition units 2820A-2820N of 28 ) that are shared by all processing clusters 2814 and can be used to transfer data between threads. In at least one embodiment, graphics multiprocessor 2834 may also access off-chip global memory, which may include one or more local parallel processor memories and/or system memory. In at least one embodiment, any memory external to parallel processing unit 2802 may be used as global memory. In at least one embodiment, processing cluster 2814 includes multiple instances of graphics multiprocessor 2834 may share common instructions and data that may be stored in L1 cache 2848 .

In at least one embodiment, each processing cluster 2814 may include an MMU 2845 (memory management unit) configured to map virtual addresses to physical addresses. In at least one embodiment, one or more instances of MMU 2845 may reside within memory interface 2818 of 28 condition. In at least one embodiment, MMU 2845 includes a set of Page Table Entries (PTEs) used to map a virtual address to a physical address of a tile (more on tiling) and optionally a cache line index. In at least one embodiment, MMU 2845 may include address translation lookaside (TLB) buffers or caches, which may reside in a graphics multiprocessor 2834 or L1 cache or processing cluster 2814 . In at least one embodiment, the physical address is processed to distribute the surface data access location to allow for efficient interleaving of requests between partition units. In at least one embodiment, the cache line index may be used to determine whether a request for a cache line is a hit or miss.

In at least one embodiment, a processing cluster 2814 may be configured such that each graphics multiprocessor 2834 is coupled to a texture unit 2836 to perform texture mapping operations, such as determining texture sample locations, reading texture data, and filtering texture data. In at least one embodiment, texture data is read from an internal texture LI cache (not shown) or from an L1 cache within the graphics multiprocessor 2834 and fetched from an L2 cache, local parallel processor memory, or system memory as needed. In at least one embodiment, each graphics multiprocessor 2834 issues processed tasks to the data crossbar 2840, to make processed tasks available to another processing cluster 2814 for further processing, or to have processed tasks via the memory crossbar 2816 in an L2 cache, to be stored in local parallel processor memory or in system memory. In at least one embodiment, preROP 2842 (Pre-Raster Operations Unit) is configured to receive data from graphics multiprocessor 2834 and forward data to ROP units, which as described herein may reside on partition units (e.g., partition units 2820A- 2820N from 28 ). In at least one embodiment, the PreROP 2842 unit may perform color mixing optimizations, organize pixel color data, and perform address translations.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are here in connection with 15A and or 15B provided. In at least one embodiment, the inference and/or training logic 1515 in the system 18C to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

28D Figure 1 shows a graphics multiprocessor 2834 in accordance with at least one embodiment. In at least one embodiment, the graphics multiprocessor 2834 is coupled to the pipeline manager 2832 of the processing cluster 2814 . In at least one embodiment, the graphics multiprocessor 2834 has an execution pipeline that includes an instruction cache 2852, an instruction unit 2854, an address mapper 2856, a register file 2858, one or more GPGPU cores (General Purpose Graphics Processing Unit) 2862, and one or more loaders /memory units 2866 includes, but is not limited to. GPGPU cores 2862 and load/store units 2866 are coupled to cache memory 2872 and shared memory 2870 via a store and cache interconnect 2868 .

In at least one embodiment, instruction cache 2852 receives a stream of instructions to be executed from pipeline manager 2832. In at least one embodiment, instructions are cached in instruction cache 2852 and dispatched to instruction unit 2854 for execution. In at least one embodiment, the instruction unit 2854 may dispatch instructions as thread groups (e.g., warps), with each thread of the thread group being associated with a different execution unit within the GPGPU core 2862. In at least one embodiment, an instruction can access any local, shared, or global address space by specifying an address within a unified address space. In at least one embodiment, address mapping unit 2856 may be used to translate addresses in a uniform address space into a unique memory address accessible by load/store units 2866.

In at least one embodiment, register file 2858 provides a set of registers for functional units of graphics multiprocessor 2834. In at least one embodiment, register file 2858 provides temporary storage for operands associated with data paths of functional units (e.g., GPGPU cores 2862, load/store units 2866) of graphics multiprocessor 2834. In at least one embodiment, register file 2858 is partitioned among each of the functional units such that each functional unit has a dedicated portion of register file 2858 associated with it. In at least one embodiment, register file 2858 is partitioned into various warps executed by graphics multiprocessor 2834.

In at least one embodiment, GPGPU cores 2862 may each include floating point units (FPUs) and/or integer arithmetic logic units (ALUs) used to execute graphics multiprocessor 2834 instructions. GPGPU cores 2862 may be similar in architecture or different in architecture. In at least one embodiment, a a first part of the GPGPU cores 2862 comprises a single precision FPU and an integer ALU, while a second part of the GPGPU cores comprises a double precision FPU. In at least one embodiment, FPUs may implement the IEEE 754-2008 standard for floating point arithmetic or enable variable precision floating point arithmetic. In at least one embodiment, the graphics multiprocessor 2834 may additionally include one or more fixed function(s) or special purpose functional units to perform specific functions such as copying rectangles or pixel blending operations. In at least one embodiment, one or more GPGPU cores may also include fixed or special purpose functional logic.

In at least one embodiment, the GPGPU cores 2862 include SIMD logic capable of executing a single instruction on multiple sets of data. In at least one embodiment, GPGPU cores 2862 can physically execute SIMD4, SIMD8, and SIMD16 instructions and logically execute SIMD1, SIMD2, and SIMD32 instructions. In at least one embodiment, SIMD instructions for GPGPU cores may be generated at compile time by a shadowing compiler or may be generated automatically when executing programs written and compiled for SPMD (Single Program Multiple Data) or SIMT architectures. In at least one embodiment, multiple threads of a program configured for a SIMT execution model may be executed from a single SIMD instruction. For example, in at least one embodiment, eight SIMT threads performing the same or similar operations may execute in parallel through a single SIMD8 logic unit.

In at least one embodiment, memory and cache interconnect 2868 is an interconnection network that connects each functional unit of graphics multiprocessor 2834 to register file 2858 and to shared memory 2870 . In at least one embodiment, store and cache interconnect 2868 is a crossbar interconnect that allows load/store unit 2866 to implement load and store operations between shared memory 2870 and register file 2858. In at least one embodiment, the register file 2858 can operate at the same frequency as the GPGPU cores 2862 such that data transfer between the GPGPU cores 2862 and the register file 2858 occurs with very low latency. In at least one embodiment, shared memory 2870 may be used to enable communication between threads executing on functional units within graphics multiprocessor 2834. In at least one embodiment, cache memory 2872 may be used as a data cache to cache texture data between functional units and texture unit 2836, for example. In at least one embodiment, shared memory 2870 may also be used as a program managed cache. For at least one embodiment, threads executing on GPGPU cores 2862 may programmatically store data in shared memory in addition to auto-cached data stored in cache memory 2872 .

In at least one embodiment, a parallel processor or GPGPU as described herein is communicatively coupled to host/processor cores to perform graphics operations, machine learning, pattern analysis operations, and various general purpose GPU (GPGPU) functions accelerate. In at least one embodiment, the GPU may be communicatively coupled to the host processor/cores via a bus or other connection (e.g., a high-speed connection such as PCIe or NVLink). In at least one embodiment, the GPU may be integrated in the same package or chip as the cores and communicatively coupled to the cores via an internal processor bus/interconnect (i.e., internal to the package or chip). In at least one embodiment, regardless of how the GPU is connected, the processor cores may assign work to the GPU in the form of sequences of commands/instructions contained in a work descriptor. In at least one embodiment, the GPU then uses dedicated circuitry/logic to efficiently process these commands/instructions.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are here in connection with 15A and or 15B provided. In at least one embodiment, the inference and/or training logic 1515 in the system 18C for inferring or predicting operations based, at least in part, on weight parameters generated using training operations with neural networks, functions and/or architectures ronal networks or use cases of neural networks described herein can be used.

29 12 shows a multi-GPU computing system 2900 in accordance with at least one embodiment. In at least one embodiment, the multi-GPU computing system 2900 may include a processor 2902 coupled through a host interface switch 2904 to multiple general purpose graphics processing units (GPGPUs) 2906A-D. In at least one embodiment, host interface switch 2904 is a PCI Express switch device that couples processor 2902 to a PCI Express bus over which processor 2902 can communicate with GPGPUs 2906A-D. The GPGPUs 2906A-D can communicate with each other via a set of high-speed point-to-point connections between the GPUs 2916. In at least one embodiment, GPU-to-GPU connections 2916 to each of the GPGPUs 2906A-D make a dedicated GPU connection. In at least one embodiment, P2P GPU links 2916 allow direct communication between each GPGPU 2906A-D without requiring communication over the host interface bus 2904 to which the processor 2902 is attached. In at least one embodiment where GPU-to-GPU traffic is routed to P2P GPU connections 2916, host interface bus 2904 remains available for accessing system memory or communicating with other instances of the multi-GPU Computing system 2900, for example via one or more network devices. While in at least one embodiment the GPGPUs 2906A-D connect to the processor 2902 through the host interface switch 2904, in at least one embodiment the processor 2902 includes direct support for P2P GPU links 2916 and can interface directly with the GPGPUs 2906A -D to be connected.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are here in connection with 15A and or 15B provided. In at least one embodiment, the inference and/or training logic 1515 in the system 18C to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

30 10 is a block diagram of a graphics processor 3000, according to at least one embodiment. In at least one embodiment, graphics processor 3000 includes ring interconnect 3002, pipeline front end 3004, engine 3037, and graphics cores 3080A-3080N. In at least one embodiment, ring interconnect 3002 couples graphics processor 3000 to other entities, including other graphics processors or one or more general processor cores. In at least one embodiment, graphics processor 3000 is one of many processors integrated into a multi-core processing system.

In at least one embodiment, graphics processor 3000 receives batches of commands over ring interconnect 3002. In at least one embodiment, incoming commands are interpreted by command streamer 3003 in pipeline front end 3004. In at least one embodiment, graphics processor 3000 includes scalable execution logic to perform 3D geometry processing and media processing using graphics core(s) 3080A-3080N. In at least one embodiment for 3D geometry processing commands, command streamer 3003 delivers commands to geometry pipeline 3036. In at least one embodiment, command streamer 3003 delivers commands to a video front end 3034 running media engine 3037 for at least some media processing commands is coupled. In at least one embodiment, media engine 3037 includes a video quality engine (VQE) 3030 for video and image post-processing and a multi-format encoding/decoding engine (MFX) 3033 to provide hardware-accelerated encoding and decoding of media data. In at least one embodiment, geometry pipeline 3036 and media engine 3037 each create threads of execution for thread execution resources provided by at least one graphics core 3080A.

In at least one embodiment, graphics processor 3000 includes scalable thread execution resources with modular cores 3080A-3080N (sometimes referred to as core sections) each having multiple sub-cores 3050A-550N, 3060A-3060N (sometimes referred to as core subsections). In at least one embodiment, graphics processor 3000 may have any number of graphics cores 3080A-3080N. In at least one embodiment, the graphic includes Processor 3000 includes a graphics core 3080A having at least a first sub-core 3050A and a second sub-core 3060A. In at least one embodiment, graphics processor 3000 is a low-power processor with a single sub-core (eg, 3050A). In at least one embodiment, graphics processor 3000 includes multiple graphics cores 3080A-3080N, each including a set of first sub-cores 3050A-3050N and a set of second sub-cores 3060A-3060N. In at least one embodiment, each sub-core in first sub-cores 3050A-3050N includes at least a first set of execution units 3052A-3052N and media/texture scanners 3054A-3054N. In at least one embodiment, each sub-core in the second sub-cores 3060A-3060N includes at least a second set of execution units 3062A-3062N and samplers 3064A-3064N. In at least one embodiment, each sub-core 3050A-3050N, 3060A-3060N shares a set of shared resources 3070A-3070N. In at least one embodiment, shared resources include shared cache memory and pixel operation logic.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are here in connection with 15A and or 15B provided. In at least one embodiment, the inference and/or training logic 1515 in the system 18C to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

31 12 is a block diagram depicting a microarchitecture for a processor 3100, which may include logic circuitry for executing instructions, according to at least one embodiment. In at least one embodiment, processor 3100 may execute instructions, including x86 instructions, ARM instructions, specialized instructions for application specific integrated circuits (ASICs), etc. In at least one embodiment, processor 3110 may include registers for storing packed data, such as 64- Bit-wide MMXTM registers in microprocessors operating on MMX technology from Intel Corporation of Santa Clara, California. In at least one embodiment, MMX registers, which are available in both integer and floating point form, can operate with packed data items involving single instructions, multiple data ("SIMD"), and streaming SIMD extensions ("SSE"). In at least one embodiment, 128-bit wide XMM registers related to SSE2, SSE3, SSE4, AVX, or beyond (commonly referred to as “SSEx”) technology may contain such packed data operands. In at least one embodiment, processor 3110 may execute instructions to accelerate machine learning or deep learning algorithms, training, or inference.

In at least one embodiment, the processor 3100 includes an R in-order front end (“front end”) 3101 to fetch instructions to be executed and to prepare instructions to be used later in the processor pipeline. In at least one embodiment, the front end 3101 may include multiple entities. In at least one embodiment, an instruction prefetcher 3126 fetches instructions from memory and provides instructions to an instruction decoder 3128, which in turn decodes or interprets instructions. For example, in at least one embodiment, instruction decoder 3128 decodes a received instruction into one or more operations, referred to as "micro-instructions" or "micro-ops" (also referred to as "micro-ops" or "uops"), that may be executed by that machine . In at least one embodiment, instruction decoder 3128 decomposes an instruction into an opcode and corresponding data and control fields that can be used by the microarchitecture to perform operations in accordance with at least one embodiment. In at least one embodiment, a trace cache 3130 may assemble decoded uops into program-ordered sequences or traces in a uop queue 3134 for execution. In at least one embodiment, when the trace cache 3130 encounters a complex instruction, a microcode ROM 3132 provides the necessary Uops to complete the operation.

In at least one embodiment, some instructions may be converted into a single micro-op, while others require multiple micro-ops to perform a full operation. In at least one embodiment, instruction decoder 3128 may access microcode ROM 3132 to execute an instruction when more than four micro-operations are required to execute an instruction. In at least one embodiment, an instruction may be decoded into a small number of micro-ops for processing in instruction decoder 3128 . At least In at least one embodiment, an instruction may be stored in microcode ROM 3132 if a number of micro-ops are required to perform the operation. In at least one embodiment, trace cache 3130 refers to an Entry Point Programmable Logic Array ("PLA") for determining a correct microinstruction pointer for reading microcode sequences to complete one or more instructions from the Microcode ROM 3132, according to at least one embodiment. In at least one embodiment, after the microcode ROM 3132 finishes sequencing micro-ops for an instruction, the engine front end 3101 may resume fetching micro-ops from the sequence cache 3130 .

In at least one embodiment, out-of-order execution engine 3103 may prepare instructions for execution. In at least one embodiment, the out-of-order execution logic has a series of buffers to smooth and reorder instruction flow to optimize performance when the pipeline is shut down and execution is scheduled. The out-of-order execution engine 3103 includes, without limitation, an allocator/register renamer 3140, a memory uop queue 3142, an integer/floating point uop queue 3144, a memory scheduler 3146, a fast scheduler 3102, a slow /general floating-point scheduler ("slow/general FP scheduler") 3104, and a simple floating-point scheduler ("simple FP scheduler") 3106. In at least one embodiment, the fast scheduler 3102, the slow/general floating-point scheduler 3104 and the simple floating point scheduler 3106 also referred to herein collectively as "uop scheduler 3102, 3104, 3106". The Allocator/Register Renamer 3140 allocates machine buffers and resources each uop needs to execute. In at least one embodiment, allocator/register renamer 3140 renames logical registers to entries in a register file. In at least one embodiment, the allocator/register renamer 3140 also allocates an entry for each uop in one of two uop queues, the memory uop queue 3142 for memory operations and the integer/floating point op queue 3144 for non-memory operations, before the memory scheduler 3146 and the uop schedulers 3102, 3104, 3106. In at least one embodiment, the uop schedulers 3102, 3104, 3106 determine the uops required to complete their operation based on the readiness of their dependent input register operand sources and the availability of execution resources , when a uop is ready to run. In at least one embodiment, the fast scheduler 3102 of at least one embodiment can schedule on every half of the main clock cycle, while the slow/general floating point scheduler 3104 and the simple floating point scheduler 3106 can schedule once per main processor clock cycle. In at least one embodiment, hub port uop schedulers 3102, 3104, and 3106 may arbitrate to schedule uops for execution.

In at least one embodiment, execution block b11 includes, without limitation, an integer register file/bypass network 3108, a floating point register file/bypass network ("FP register file/bypass network") 3110, address generation units (" AGUs") 3112 and 3114, fast arithmetic logic units (ALUs) ("Fast ALUs") 3116 and 3118, a slow arithmetic logic unit ("Slow ALU") 3120, a floating point ALU ("FP") 3122, and a floating point move unit ("FP move") 3124. In at least one embodiment, integer register file/bypass network 3108 and floating point register file/bypass network 3110 are also referred to herein as "register files 3108, 3110". In at least one embodiment, AGUS 3112 and 3114, fast ALUs 3116 and 3118, slow ALU 3120, floating point ALU 3122, and floating point mover 3124 are also referred to herein as "execution units 3112, 3114, 3116, 3118 , 3120, 3122 and 3124”. In at least one embodiment, execution block b11 may include, without limitation, any number (including zero) and type of register files, bypass networks, address generation units, and execution units, in any combination.

In at least one embodiment, register files 3108, 3110 may be located between uop schedulers 3102, 3104, 3106 and execution units 3112, 3114, 3116, 3118, 3120, 3122, and 3124. In at least one embodiment, register file/bypass network 3108 performs integer operations. In at least one embodiment, floating point register file/bypass network 3110 performs floating point operations. In at least one embodiment, each of the register files 3108, 3110 may include, without limitation, a bypass network that may bypass or forward to new dependent uops just completed results that have not yet been written to the register file. In at least one embodiment, the register files 3108, 3110 can communicate with each other. In at least one embodiment, register file/bypass network 3108 may, without limitation, be two separate register files comprise a register file for low-order 32 bits of data and a second register file for high-order 32 bits of data. In at least one embodiment, the floating point register file/bypass network 3110 may include, without limitation, 128-bit wide entries since floating point instructions typically have operands that are 64 to 128 bits wide.

In at least one embodiment, execution units 3112, 3114, 3116, 3118, 3120, 3122, 3124 may execute instructions. In at least one embodiment, register files 3108, 3110 store integer and floating point data operand values that microinstructions are required to execute. In at least one embodiment, processor 3100 may include any number and combination of execution units 3112, 3114, 3116, 3118, 3120, 3122, 3124, without limitation. In at least one embodiment, floating point ALU 3122 and floating point mover 3124 may perform floating point, MMX, SIMD, AVX, and SSE or other operations, including special machine learning instructions. In at least one embodiment, floating point ALU 3122 may include, without limitation, a 64-bit x 64-bit floating point divider for performing division, square root, and remainder micro-ops. In at least one embodiment, instructions containing a floating point value may be processed using floating point hardware. In at least one embodiment, ALU operations may be committed to fast ALUs 3116,3118. In at least one embodiment, the fast ALUs 3116, 3118 can perform fast operations with an effective latency of half a clock cycle. In at least one embodiment, the most complex integer operations go to the slow ALU 3120, since the slow ALU 3120 may include, without limitation, integer execution hardware for long latency operations such as a multiplier, shifts, flag logic, and branch processing. In at least one embodiment, memory load/store operations may be performed by AGUS 3112, 3114. In at least one embodiment, fast ALU 3116, fast ALU 3118, and slow ALU 3120 can perform integer operations on 64-bit data operands. In at least one embodiment, fast ALU 3116, fast ALU 3118, and slow ALU 3120 can be implemented to support a variety of data bit sizes, including sixteen, thirty-two, 128, 256, etc. In at least one embodiment, floating point ALU 3122 and the Floating point mover 3124 can be implemented to support a range of operands with different bit widths. In at least one embodiment, floating point ALU 3122 and floating point mover 3124 can operate on 128-bit wide packed data operands in conjunction with SIMD and multimedia instructions.

In at least one embodiment, the uop schedulers 3102, 3104, 3106 schedule dependent operations before the parent load has finished executing. Since uops may be speculatively scheduled and executed in processor 3100, in at least one embodiment, processor 3100 may also include logic for handling misses in memory. In at least one embodiment, when there is a miss loading data in the data cache, there may be dependent operations in the pipeline during execution that exited the scheduler with temporarily bad data. In at least one embodiment, a retry mechanism tracks and re-executes instructions that use incorrect data. In at least one embodiment, dependent operations may need to be re-executed and independent operations allowed to complete. In at least one embodiment, the scheduler and retry mechanism of at least one embodiment of a processor may also be configured to capture instruction sequences for text string comparison operations.

In at least one embodiment, the term "register" may refer to storage locations in the processor that may be used as part of instructions to identify operands. In at least one embodiment, registers may be those that can be used from outside the processor (from a programmer's point of view). In at least one embodiment, registers may not be limited to any particular circuit type. Rather, in at least one embodiment, a register may store data, provide data, and perform the functions described herein. In at least one embodiment, the registers described herein may be implemented by circuitry within a processor using any number of different techniques, such as dedicated physical registers, dynamically allocated physical registers using register renaming, combinations of dedicated and dynamically allocated physical registers, etc. In In at least one embodiment, integer registers store 32-bit integer data. The register file of at least one embodiment also includes eight packed data multimedia SIMD registers.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, some or all of the inference and/or training logic 1515 may be included in EXE block 3111 and other memory or registers, shown or not shown. For example, in at least one embodiment, the training and/or inference techniques described herein may use one or more of the ALUs illustrated in EXE block 3111 . Additionally, weight parameters may be stored in on-chip or off-chip memory and/or in registers (shown or not) that configure ALUs of EXE block 3111 to implement one or more machine learning algorithms, neural network architectures, use cases or training techniques described here.

32 12 shows a deep learning application processor 3200 according to at least one embodiment. In at least one embodiment, deep learning applications processor 3200 uses instructions that, when executed by deep learning applications processor 3200, cause deep learning applications processor 3200 to perform some or all of the processes and techniques described in this disclosure. In at least one embodiment, deep learning application processor 3200 is an application specific integrated circuit (ASIC). In at least one embodiment, applications processor 3200 performs matrix multiplication operations that are either "hardwired" into hardware or performed as a result of the execution of one or more instructions, or both. In at least one embodiment, deep learning applications processor 3200 includes, without limitation, processing clusters 3210(1)-3210(12), inter-chip interconnects ("ICLs") 3220(1)-3220(12), inter-chip controllers (“ICCs”) 3230(1)-3230(2), Second Generation High Bandwidth Memory (“HBM2”) 3240(1)-3240(4), Memory Controller (“Mem Ctrlrs”) 3242(1)-3242 (4), a high-bandwidth memory ("HBM PHY") physical layer 3244(1)-3244(4), a management controller central processing unit (CPU) ) 3250, a Serial Peripheral Interface, Inter-Integrated Circuit and General Purpose Input/Output (SPI, I2C, GPIO) block 3260, a Peripheral Component Connection Express Control and Direct Memory Access block ( "PCIe Controller and DMA") 3270 and a sixteen-lane peripheral component connection express port ("PCI Express x 16") 3280.

In at least one embodiment, processing clusters 3210 may perform deep learning operations, including inference or prediction operations based on weight parameters calculated by one or more training techniques, including those described herein. In at least one embodiment, each processing cluster 3210 may include any number and type of processors, without limitation. In at least one embodiment, deep learning applications processor 3200 may include any number and type of processing clusters 3200. In at least one embodiment, the inter-chip connections 3220 are bi-directional. In at least one embodiment, inter-chip interconnects 3220 and inter-chip controllers 3230 enable multiple deep learning application processors 3200 to exchange information, including enablement information, resulting from the execution of one or more machine learning algorithms running in one or more neural networks are included. In at least one embodiment, deep learning application processor 3200 may include any number (including zero) and type of ICLs 3220 and ICCs 3230.

In at least one embodiment, the HBM2s 3240 are provided with a total of 32 gigabytes (GB) of memory. HBM2 3240(i) is assigned to both Controller 3242(i) and HBM PHY 3244(i). In at least one embodiment, any number of HBM2s 3240 can provide any type and amount of high-bandwidth memory and can be associated with any number (including zero) and type of memory controllers 3242 and HBM PHYs 3244 . In at least one embodiment, SPI, I2C, GPIO 3260, PCIe controller and DMA 3270, and/or PCIe 3280 may be replaced with any number and type of blocks that enable any number and type of communication standards in any technically appropriate manner.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, the deep learning application processor is used to train a machine learning model, such as a neural network, to predict or infer information provided to the deep learning application processor 3200 . At least In one embodiment, deep learning application processor 3200 is used to derive or predict information trained by another processor or system or by deep learning application processor 3200 based on a trained machine learning model (e.g., a neural network). In at least one embodiment, processor 3200 may be used to perform one or more of the neural network use cases described herein.

33 3300 is a block diagram of a neuromorphic processor 3300, according to at least one embodiment. In at least one embodiment, neuromorphic processor 3300 may receive one or more inputs from sources external to neuromorphic processor 3300. In at least one embodiment, these inputs may be transmitted to one or more neurons 3302 within neuromorphic processor 3300. In at least one embodiment, neurons 3302 and their components may be implemented using circuitry or logic that includes one or more arithmetic logic units (ALUs). In at least one embodiment, neuromorphic processor 3300 may include, without limitation, thousands or millions of instances of neurons 3302, but any suitable number of neurons 3302 may be used. In at least one embodiment, each instance of neuron 3302 may include a neuron input 3304 and a neuron output 3306. In at least one embodiment, neurons 3302 may generate outputs that may be transmitted to inputs of other instances of neurons 3302. For example, in at least one embodiment, neuron inputs 3304 and neuron outputs 3306 may be connected to each other via synapses 3308 .

In at least one embodiment, neurons 3302 and synapses 3308 may be interconnected such that neuromorphic processor 3300 operates to process or analyze information received from neuromorphic processor 3300. In at least one embodiment, neurons 3302 may emit an output pulse (or "fire" or "spike") when input received via neuron input 3304 exceeds a threshold. In at least one embodiment, neurons 3302 may sum or integrate the signals received at neuron inputs 3304. For example, in at least one embodiment, neurons 3302 may be implemented as leaky-integrate-and-fire neurons, where when a sum (referred to as "membrane potential") exceeds a threshold, neuron 3302 outputs (or "fires"). using a transfer function such as a sigmoid or threshold function. In at least one embodiment, a leaky integrate and fire neuron can sum the signals received at neuron inputs 3304 into a membrane potential and also apply a decay factor (or "leak") to reduce a membrane potential. In at least one embodiment, a leaky integrate-and-fire neuron may fire when multiple input signals are received at neuron inputs 3304 fast enough to cross a threshold (i.e., before a membrane potential decays too low to fire). In at least one embodiment, neurons 3302 may be implemented by circuitry or logic that receives inputs, integrates inputs into a membrane potential, and decays a membrane potential. In at least one embodiment, the inputs may be averaged, or any other suitable transfer function may be used. Additionally, in at least one embodiment, without limitation, neurons 3302 may include comparator circuitry or logic that generates an output spike at neuron output 3306 when the result of applying a transfer function to neuron input 3304 exceeds a threshold. In at least one embodiment, once neuron 3302 has fired, it may ignore previously received input information, for example, by resetting a membrane potential to 0 or some other suitable default value. In at least one embodiment, once the membrane potential has been reset to 0, neuron 3302 may resume normal operation after an appropriate period of time (or refractory period).

In at least one embodiment, neurons 3302 may be connected to each other via synapses 3308. In at least one embodiment, synapses 3308 may function to transmit signals from an output of a first neuron 3302 to an input of a second neuron 3302. In at least one embodiment, neurons 3302 may transmit information across more than one synapse 3308 instance. In at least one embodiment, one or more instances of neuron output 3306 may be connected to an instance of neuron input 3304 in the same neuron 3302 via an instance of synapse 3308 . In at least one embodiment, an instance of neuron 3302 that produces an output to be transmitted across an instance of synapse 3308 may be referred to as a "presynaptic neuron" with respect to that instance of synapse 3308. be designated. In at least one embodiment, a neuron 3302 that receives input transmitted across a synapse 3308 may be referred to as a "postsynaptic neuron" with respect to that synapse 3308. Because an instance of neuron 3302 can receive input from one or more instances of synapse 3308, and can also send outputs across one or more instances of synapse 3308, a single instance of neuron 3302 can therefore be both a "presynaptic neuron" and a "postsynaptic neuron". Neuron” in relation to various instances of synapse 3308 in at least one embodiment.

In at least one embodiment, neurons 3302 may be organized into one or more layers. Each instance of a neuron 3302 can have a neuron output 3306 that can fan out to one or more neuron inputs 3304 via one or more synapses 3308 . In at least one embodiment, neuron outputs 3306 of neurons 3302 in a first layer 3310 may be connected to neuron inputs 3304 of neurons 3302 in a second layer 3312. In at least one embodiment, layer 3310 may be referred to as a "feed-forward layer." In at least one embodiment, each instance of neuron 3302 in an instance of the first layer 3310 may fan out to each instance of neuron 3302 in the second layer 3312. In at least one embodiment, the first layer 3310 may be referred to as a fully connected feed-forward layer. In at least one embodiment, each instance of a neuron 3302 in a second layer 3312 instance may fan out to fewer than all of the neuron 3302 instances in a third layer 3314 . In at least one embodiment, the second layer 3312 may be referred to as a "sparsely connected feed-forward layer." In at least one embodiment, neurons 3302 in second layer 3312 may fan out to neurons 3302 in multiple other layers, including neurons 3302 in the (same) second layer 3312. In at least one embodiment, second layer 3312 may be referred to as a "recurring layer." “ be designated. The neuromorphic processor 3300 may include, without limitation, any suitable combination of recurring layers and feed-forward layers, including without limitation both sparsely connected feed-forward layers and fully connected feed-forward layers.

In at least one embodiment, neuromorphic processor 3300 may include, without limitation, a reconfigurable connection architecture or dedicated hard-wired connections for connecting synapse 3308 to neurons 3302. In at least one embodiment, neuromorphic processor 3300 may include, without limitation, circuitry or logic that allows synapses to be mapped to different neurons 3302 as needed based on neural network topology and neuron fan-in/out. For example, in at least one embodiment, synapses 3308 may be connected to neurons 3302 using an interconnect fabric, such as network-on-chip, or with dedicated connections. In at least one embodiment, synapse connections and their components may be implemented by circuitry or logic.

34 12 is a block diagram of a processing system, according to at least one embodiment. In at least one embodiment, the system 3400 includes one or more processors 3402 and one or more graphics processors 3408 and may be a uniprocessor desktop system, a multiprocessor workstation system, or a server system having a large number of processors 3402 or processor cores 3407 . In at least one embodiment, system 3400 is a processing platform integrated into a system-on-a-chip (SoC) integrated circuit for use in mobile, portable, or embedded devices.

In at least one embodiment, system 3400 may comprise a gaming console, including a gaming and media console, a mobile gaming console, a handheld gaming console, or an online gaming console, or may be integrated with a server-based gaming platform. In at least one embodiment, the system 3400 is a cell phone, a smart phone, a tablet computing device, or a mobile internet device. In at least one embodiment, processing system 3400 may also include, be coupled to, or integrated with a wearable device, such as a wearable smart watch device, smart eyewear device, augmented reality device, or virtual machine reality device. In at least one embodiment, processing system 3400 is a television or set top box device having one or more processors 3402 and a graphical interface generated by one or more graphics processors 3408 .

In at least one embodiment, one or more processors 3402 each include one or more processor cores 3407 for processing instructions that, when executed, perform system and application software operations. In at least one embodiment, each of one or more processor cores 3407 is configured to process a particular set of instructions 3409 . In at least one embodiment, instruction set 3409 may include complex instruction set computing (CISC), reduced instruction set computing (RISC), or very long instruction word (VLIW) computing ) enable. In at least one embodiment, processor cores 3407 may each process a different instruction set 3409, which may include instructions to facilitate emulation of other sets of instructions. In at least one embodiment, processor core 3407 may also include other processing devices, such as a digital signal processor (DSP).

In at least one embodiment, processor 3402 includes cache memory 3404. In at least one embodiment, processor 3402 may have a single internal cache or multiple internal cache levels. In at least one embodiment, the cache memory is shared between different processor 3402 components. In at least one embodiment, the processor 3402 also uses an external cache (e.g., a Level 3 (L3) cache or Last Level Cache (LLC)) (not shown) that is shared between the processor cores 3407 using known cache coherency techniques be able. In at least one embodiment, register file 3406 is additionally included in processor 3402, which may include various types of registers for storing various types of data (e.g., integer registers, floating point registers, status registers, and an instruction pointer register). In at least one embodiment, register file 3406 may include general purpose registers or other registers.

In at least one embodiment, processor(s) 3402 is/are coupled to interface bus(s) 3410 to transfer communication signals such as address, data, or control signals between processor 3402 and other components in system 3400. In at least one embodiment, interface bus 3410 may be a processor bus, e.g., a version of a Direct Media Interface (DMI) bus, in one embodiment. In at least one embodiment, interface 3410 is not limited to a DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In at least one embodiment, processor(s) 3402 includes an integrated memory controller 3416 and a platform control hub 3430. In at least one embodiment, the memory controller 3416 facilitates communication between a memory device and other components of the system 3400, while the platform control hub (PCH ) 3430 provides connections to I/O blocks via a local I/O bus.

In at least one embodiment, storage device 3420 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase change memory device, or another suitably performing memory device that serves as process memory. In at least one embodiment, device 3420 may operate as system memory for system 3400 to store data 3422 and instructions 3421 for use when one or more processors 3402 execute an application or process. In at least one embodiment, memory controller 3416 is also coupled to an optional external graphics processor 3412 that can communicate with one or more graphics processors 3408 in processors 3402 to perform graphics and media operations. In at least one embodiment, a display device 3411 may be coupled to processor(s) 3402 . In at least one embodiment, display device 3411 may include one or more internal display devices, such as a mobile electronic device or laptop, or an external display device connected via a display interface (e.g., DisplayPort, etc.). In at least one embodiment, the display device 3411 may comprise a head mounted display (HMD), e.g., a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

In at least one embodiment, platform control hub 3430 enables peripheral devices to be connected to memory controller 3420 and processor 3402 via a high-speed I/O bus. In at least one embodiment, the I/O peripherals include, but are not limited to, an audio controller 3446, a network controller 3434, a firmware interface 3428, a wireless transceiver (ie, transmitter/receiver) 3426, touch sensors 3425, a data storage device 3424 (eg, hard drive, flash memory, etc.). In at least one embodiment, it can Data storage device 3424 can be connected via a storage interface (eg, SATA) or via a peripheral bus, such as a peripheral component interconnect bus (eg, PCI, PCI Express). In at least one embodiment, touch sensors 3425 may include touch screen sensors, pressure sensors, or fingerprint sensors. In at least one embodiment, wireless transceiver 3426 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, or Long Term Evolution (LTE) transceiver. In at least one embodiment, firmware interface 3428 enables communication with system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, controller 3434 may enable network connection to a wired network. In at least one embodiment, a high performance network controller (not shown) is coupled to interface bus 3410. In at least one embodiment, controller 3446 is a multi-channel high definition audio controller. In at least one embodiment, the system 3400 includes an optional legacy I/O controller 3440 for coupling legacy devices (eg, Personal System 2 (PS/2)) to the system. In at least one embodiment, the platform controller hub 3430 can also be connected to one or more Universal Serial Bus (USB) controllers 3442 to connect input devices such as a keyboard and mouse 3443 combo, a camera 3444, or other USB input devices.

In at least one embodiment, an instance of memory controller 3416 and platform control hub 3430 may be integrated into a discrete external graphics processor, such as external graphics processor 3412. In at least one embodiment, platform control hub 3430 and/or memory controller 3416 may be external to processor(s) 3402 . For example, in at least one embodiment, system 3400 may include an external memory controller 3416 and a platform control hub 3430, which may be configured as a memory controller hub and peripheral controller hub within a system chipset that is in communication with processor(s) 3402.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, some or all of the inference and/or training logic 1515 may be integrated into the graphics processor 3400 . For example, in at least one embodiment, the training and/or inference techniques described herein may use one or more ALUs included in 3D pipeline 3412 . Furthermore, in at least one embodiment, the inferencing and/or training operations described herein may use logic other than that described in 15A or 15B logic shown are performed. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and/or registers (shown or not) that configure ALUs of graphics processor 3400 to implement one or more machine learning algorithms described herein , neural network architectures, use cases or training techniques.

35 10 is a block diagram of a processor 3500 having one or more processor cores 3502A-3502N, an integrated memory controller 3514, and an integrated graphics processor 3508, according to at least one embodiment. In at least one embodiment, processor 3500 may include additional cores up to and including additional core 3502N, represented by dashed boxes. In at least one embodiment, each of the processor cores 3502A-3502N includes one or more internal cache units 3504A-3504N. In at least one embodiment, each processor core also has access to one or more shared cache units 3506.

In at least one embodiment, internal cache units 3504A-3504N and shared cache units 3506 represent a cache memory hierarchy within processor 3500. In at least one embodiment, cache memory units 3504A-3504N may include at least one instruction and data cache level within each processor core and include one or more levels of mid-level shared cache, such as level 2 (L2), level 3 (L3), level 4 (L4), or other cache levels, with the highest cache level is classified as LLC before the external storage. In at least one embodiment, cache coherency logic maintains coherency between the various cache units 3506 and 3504A-3504N.

In at least one embodiment, processor 3500 may also include a set of one or more bus control units 3516 and a system service core 3510 . In at least one embodiment, one or more bus control units 3516 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system service core 3510 is for managing various processor components. In at least one embodiment, the system service core 3510 includes one or more integrated storage controllers 3514 for managing access to various external storage devices (not shown).

In at least one embodiment, one or more of the processor cores 3502A-3502N include support for simultaneous multi-threading. In at least one embodiment, system service core 3510 includes components for coordinating and operating cores 3502A-3502N during multi-threaded processing. In at least one embodiment, system service core 3510 may additionally include a power control unit (PCU) that includes logic and components for controlling one or more power states of processor cores 3502A-3502N and graphics processor 3508.

In at least one embodiment, processor 3500 additionally includes graphics processor 3508 for performing graphics processing operations. In at least one embodiment, the graphics processor 3508 is coupled to the shared cache units 3506 and the system service core 3510, which includes one or more integrated memory controllers 3514. In at least one embodiment, the system service core 3510 also includes the controller 3511 to control the output of the graphics processor to one or more coupled displays. In at least one embodiment, the controller of the display 3511 may also be a separate module that is coupled to the graphics processor 3508 via at least one connection, or integrated with the graphics processor 3508.

In at least one embodiment, a ring-based interconnect unit 3512 is used to couple internal components of processor 3500 together. In at least one embodiment, an alternative connection entity may be used, such as a point-to-point connection, a switched connection, or other techniques. In at least one embodiment, graphics processor 3508 is coupled to ring interconnect 3512 via an I/O interconnect 3513 .

In at least one embodiment, I/O connection 3513 represents at least one of several varieties of I/O connections, including an in-housing I/O connection that enables communication between various processor components and an embedded high-performance memory module 3518, such as a eDRAM module, lightened. In at least one embodiment, each of processor cores 3502A-3502N and graphics processor 3508 use embedded memory modules 3518 as a shared last level cache.

In at least one embodiment, processor cores 3502A-3502N are homogeneous cores that execute a common instruction set architecture. In at least one embodiment, processor cores 3502A-3502N are instruction set architecture (ISA) heterogeneous, with one or more of processor cores 3502A-3502N executing a common instruction set, while one or more other cores of processor cores 3502A-35-02N execute a subset of a execute a common instruction set or a different instruction set. In at least one embodiment, processor cores 3502A-3502N are heterogeneous in terms of microarchitecture, with one or more relatively higher power consumption cores coupled with one or more lower power consumption cores. In at least one embodiment, processor 3500 may be implemented on one or more chips or as an SoC integrated circuit.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, some or all of the inference and/or training logic 1515 may be integrated into the graphics processor 3510 . For example, in at least one embodiment, the training and/or inference techniques described herein may utilize one or more ALUs resident in the 3D Pipeline 3412, Graphics Core(s) 3515A, Shared Functional Logic(s) 3516 , the 3515B graphics core(s), the 3520 shared functional logic(s), or other logic in 35 are included. About that Additionally, in at least one embodiment, the inferencing and/or training operations described herein may use logic other than that described in 15A or 15B logic shown are performed. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and/or registers (shown or not) that configure ALUs of graphics processor 3510 to implement one or more machine learning algorithms described herein , network architectures, use cases or training techniques for neural networks.

36 Figure 12 is a block diagram of a 3600 graphics processor, which may be a discrete graphics processing unit or a graphics processor integrated into multiple processing cores. In at least one embodiment, graphics processor 3600 communicates with registers of graphics processor 3600 and with instructions input to memory via an I/O interface mapped to memory. In at least one embodiment, graphics processor 3600 includes a memory interface 3614 for memory access. In at least one embodiment, memory interface 3614 is an interface to local memory, one or more internal caches, one or more shared external caches, and/or system memory.

In at least one embodiment, graphics processor 3600 also includes a display controller 3602 for controlling the output of display data to a display device 3620. In at least one embodiment, display controller 3602 includes hardware for one or more overlay layers for device 3620 and a composite of multiple layers of video or user interface elements. In at least one embodiment, display device 3620 may be an internal or external display device. In at least one embodiment, display device 3620 is a head-mounted display device, such as a virtual reality (VR) display device or an augmented reality (AR) display device. In at least one embodiment, the graphics processor 3600 includes a video codec engine 3606 for encoding, decoding, or transcoding media to, from, or between one or more media encoding formats, including but not limited to MPEG (Moving Picture Experts Group) formats such as MPEG -2, AVC (Advanced Video Coding) formats such as H.264 /MPEG-4 AVC, as well as the Society of Motion Picture & Television Engineers (SMPTE) 421M/VC-1 and the Joint Photographic Experts Group (JPEG) formats such as JPEG and Motion JPEG (MJPEG) formats.

In at least one embodiment, graphics processor 3600 includes a block image transfer (BLIT) engine 3604 to perform two-dimensional (2D) rasterization operations including, for example, bit boundary block transfers. However, in at least one embodiment, the 2D graphics operations are performed using one or more Graphics Processing Engine (GPE) 3610 components. In at least one embodiment, GPE 3610 is a computational engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

In at least one embodiment, the GPE 3610 includes a 3D pipeline 3612 for performing 3D operations, such as rendering three-dimensional images and scenes with processing functions that operate on 3D primitive shapes (e.g., rectangle, triangle, etc.). The 3D pipeline 3612 includes programmable and fixed functional elements that perform various tasks and/or spawn threads of execution on a 3D/media subsystem 3615. While 3D pipeline 3612 may be used to perform media operations in at least one embodiment, GPE 3610 also includes a media pipeline 3616 that may be used to perform media operations such as B. video post-processing and image enhancement is used.

In at least one embodiment, media engine 3616 includes fixed functional units or programmable logic units for performing one or more specialized media operations, such as video decoding acceleration, video de-interlacing, and video encoding acceleration instead of or for video codec engine 3606. In at least one embodiment Additionally, the media pipeline 3616 includes a thread creation unit for creating threads for execution on the 3D/media subsystem 3615. In at least one embodiment, the created threads perform computations for media operations on one or more graphics execution units running in the 3D /Media subsystem 3615 are included.

In at least one embodiment, 3D/media subsystem 3615 includes logic for executing threads spawned by 3D pipeline 3612 and media pipeline 3616 . in minutes In at least one embodiment, 3D pipeline 3612 and media pipeline 3616 send thread execution requests to 3D/media subsystem 3615, which includes thread dispatch logic for arbitrating and dispatching various requests to available thread execution resources. In at least one embodiment, the execution resources include an array of graphics execution units for processing 3D and media threads. In at least one embodiment, 3D/media subsystem 3615 includes one or more internal caches for thread instructions and data. In at least one embodiment, subsystem 3615 also includes shared memory, including registers and addressable memory, to share data between threads and to store output data.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, some or all of the inference and/or training logic 1515 may be integrated into the graphics processor 3600. For example, in at least one embodiment, the training and/or inference techniques described herein may use one or more ALUs included in 3D pipeline 3612 . Furthermore, in at least one embodiment, the inferencing and/or training operations described herein may use logic other than that described in 15A or 15B logic shown are performed. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and/or registers (shown or not) that configure ALUs of graphics processor 3600 to implement one or more machine learning algorithms described herein, network architectures, use cases or training techniques for neural networks.

37 3710 is a block diagram of a graphics processing engine 3710 of a graphics processor, according to at least one embodiment. In at least one embodiment, the graphics processing engine (GPE) 3710 is a version of the 36 GPE 3610 shown as shown. In at least one embodiment, media pipeline 3616 is optional and may not be included in GPE 3710 explicitly. In at least one embodiment, a separate media and/or image processor is coupled to GPE 3710.

In at least one embodiment, the GPE 3710 is coupled to or includes an instruction streamer 3703 that provides an instruction stream for the 3D pipeline 3612 and/or the media pipelines 3616 . In at least one embodiment, instruction streamer 3703 is coupled to memory, which may be system memory or one or more internal caches and shared caches. In at least one embodiment, instruction streamer 3703 receives instructions from memory and sends instructions to 3D pipeline 3612 and/or media pipeline 3616. In at least one embodiment, the instructions are instructions, primitives, or micro-operations that are retrieved from a circular buffer in which instructions for the 3D pipeline 3612 and the media pipeline 3616 are stored. In at least one embodiment, a ring buffer may additionally include batch instruction buffers storing batches of multiple instructions. In at least one embodiment, instructions for 3D pipeline 3612 may also include references to data stored in memory, such as, but not limited to, vertex and geometry data for 3D pipeline 3612 and/or image data and storage objects for media pipeline 3616. In In at least one embodiment, the 3D pipeline 3612 and media pipeline 3616 process commands and data by executing operations or sending one or more threads of execution to a graphics core array 3714 . In at least one embodiment, graphics core array 3714 includes one or more blocks of graphics cores (e.g., graphics core(s) 3715A, graphics core(s) 3715B), where each block includes one or more graphics cores. In at least one embodiment, each graphics core includes a set of graphics execution resources that includes general and graphics-specific execution logic for performing graphics and computational operations, as well as texture processing with fixed functionality and/or machine learning and artificial intelligence acceleration logic, including inference and /or training logic 1515 in 15A and 15B .

In at least one embodiment, 3D pipeline 3612 includes logic with fixed functionality and programmable logic to implement one or more shading programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders, or other shading programs. by processing commands and sending threads of execution to graphics core array 3714. In at least one embodiment, graphics core array 3714 provides a unified block of execution resources for use in processing shader programs men ready. In at least one embodiment, the general execution logic (e.g., execution units) within the graphics core(s) 3715A-3715B of the graphics core array 3714 includes support for various 3D API shader languages and can execute multiple simultaneous threads of execution associated with multiple shaders are.

In at least one embodiment, the graphics core array 3714 also includes execution logic to perform media functions such as video and/or image processing. In at least one embodiment, in addition to the graphics processing operations, the execution units include general purpose logic that is programmable to perform parallel general purpose computing operations.

In at least one embodiment, output data generated by threads executing on the graphics core array 3714 may output data to memory in a unified return buffer (URB) 3718 . The URB 3718 can store data for multiple threads. In at least one embodiment, the URB 3718 may be used to send data between different threads executing on the graphics core array 3714. In at least one embodiment, URB 3718 may be used in addition to synchronization between threads on graphics core array 3714 and fixed functional logic within shared functional logic 3720.

In at least one embodiment, graphics core array 3714 is scalable such that graphics core array 3714 includes a variable number of graphics cores, each having a variable number of execution units based on a target and performance level of GPE 3710. In at least one embodiment, execution resources are dynamically scalable such that execution resources can be enabled or disabled as needed.

In at least one embodiment, graphics core array 3714 is coupled to shared functional logic 3720 that includes multiple resources shared by the graphics cores in graphics core array 3714 . In at least one embodiment, shared functions performed by shared function logic 3720 are embodied in hardware logic units that provide specific additional functions to graphics core array 3714 . In at least one embodiment, shared functional logic 3720 includes, but is not limited to, sampler 3721, math 3722, and inter-thread communication (ITC) 3723 logic. In at least one embodiment, one or more caches 3725 are included in or coupled to shared functional logic 3720 .

In at least one embodiment, a shared function is used when the demand for a specialized function is insufficient for inclusion in graphics core array 3714 . In at least one embodiment, a single instantiation of a specialized function is used in shared function logic 3720 and shared by other execution resources within graphics core array 3714 . In at least one embodiment, specific shared functions within shared function logic 3720 that are widely used by graphics core array 3714 may be included in shared function logic 3716 within graphics core array 3714 . In at least one embodiment, shared functional logic 3716 within graphics core array 3714 may include some or all of the logic within shared functional logic 3720 . In at least one embodiment, all logic elements within shared functional logic 3720 may be duplicated in shared functional logic 3716 of graphics core array 3714 . In at least one embodiment, shared functional logic 3720 is eliminated in favor of shared functional logic 3716 within graphics core array 3714 .

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, some or all of the inference and/or training logic 1515 may be integrated into the graphics processor 3710 . For example, in at least one of the example embodiments, training and/or inference techniques described herein, one or more of the ALUs included in the 3D pipeline 3612, the graphics core(s) 3715A, the common functional logic 3716, the graphics core(s). (en) 3715B, the common functional logic 3720 or other logic in 37 are included. Furthermore, in at least one embodiment, the inferencing and/or training operations described herein may use logic other than that described in 15A or 15B logic shown are performed. At least In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and/or registers (shown or not shown) that configure ALUs of graphics processor 3710 to implement one or more machine learning algorithms described herein, Execute network architectures, use cases or training techniques of neural networks.

38 10 is a block diagram of the hardware logic of a graphics processor core 3800, according to at least one embodiment described herein. In at least one embodiment, the graphics processor core 3800 is included in a graphics core array. In at least one embodiment, the graphics processor core 3800, sometimes referred to as a core section, may be one or more graphics cores within a modular graphics processor. In at least one embodiment, graphics processor core 3800 is exemplary of a graphics core portion, and a graphics processor as described herein may include multiple graphics core portions based on target power and performance envelopes. In at least one embodiment, each graphics core 3800 may include a fixed functional block 3830 coupled to multiple sub-cores 3801A-3801F, also referred to as subsections, which comprise modular blocks of common and fixed functional logic.

In at least one embodiment, fixed functionality block 3830 includes a fixed functionality geometry pipeline 3836 that can be shared among all sub-cores in graphics processor 3800, for example in lower performance and/or lower power consumption graphics processor implementations. In at least one embodiment, the fixed functionality geometry pipeline 3836 includes a 3D fixed functions pipeline, a video front end unit, a thread creator and thread dispatcher, and a unified return buffer manager that manages unified return buffers.

In at least one embodiment, the fixed functionality block 3830 also includes a graphics SoC interface 3837, a graphics microcontroller 3838, and a media pipeline 3839. The graphics SoC interface 3837 is an interface between the graphics core 3800 and other processor cores within of a system on an integrated circuit on a chip. In at least one embodiment, the 3838 graphics microcontroller is a programmable subprocessor that is configurable to manage various functions of the 3800 graphics processor, including thread dispatcher, scheduler, and pre-emption. In at least one embodiment, media pipeline 3839 includes logic to facilitate decoding, encoding, pre-processing, and/or post-processing of multimedia data, including image and video data. In at least one embodiment, media pipeline 3839 implements media operations via requests for computation or sampling logic within sub-cores 3801-3801F.

In at least one embodiment, the SoC interface 3837 allows the graphics core 3800 to communicate with general application processor cores (e.g., CPUs) and/or other components within a SoC, including memory hierarchy elements such as a shared last-level cache, system RAM, and /or embedded on-chip or on-package DRAM. In at least one embodiment, the SoC interface 3837 may also enable communication with devices with fixed functions within a SoC, such as camera image processing pipelines, and enable the use and/or implementation of global memory building blocks (atoms) used by the graphics core 3800 and CPUs can be shared within a SoC. In at least one embodiment, the SoC interface 3837 may also implement power management controls for the graphics core 3800 and enable an interface between a clock domain of the graphics core 3800 and other clock domains within a SoC. In at least one embodiment, SoC interface 3837 enables receiving command buffers from a command streamer and a global thread dispatcher configured to provide commands and commands to each of one or more graphics cores within a graphics processor. In at least one embodiment, commands and instructions may be sent to media pipeline 3839 when media operations are to be performed, or to a fixed function geometry pipeline (e.g., fixed function geometry pipeline 3836, fixed function geometry pipeline 3814). , when graphics processing operations are to be performed.

In at least one embodiment, the graphics microcontroller 3838 can be configured to perform various scheduling and management tasks for the graphics core 3800. In at least one embodiment, graphics microcontroller 3838 may perform graphics and/or computational load scheduling for various parallel graphics engines within arrays 3802A-3802F and 3804A-3804F of embodiments ing unit (EU) within sub-cores 3801A-3801F. In at least one embodiment, host software running on a CPU core of a SoC including the graphics core 3800 can dispatch workloads to one of a plurality of graphics processor doorbells that invoke a scheduling operation on an appropriate graphics engine. In at least one embodiment, the scheduling operations include determining the workload to be executed next, transferring a workload to an instruction streamer, anticipating existing workloads running on an engine, monitoring a workload's progress, and notifying the host software when a workload completes. In at least one embodiment, the graphics microcontroller 3838 may also enable low-power or idle states for the graphics core 3800 by allowing the graphics core 3800 to reset registers within the graphics core 3800 during low-power state transitions independent of an operating system and/or graphics driver software save and restore on a system.

In at least one embodiment, graphics core 3800 may have more or less than the depicted sub-cores 3801A-3801F, up to N modular sub-cores. In at least one embodiment, for each set of N sub-cores, the graphics core 3800 may also include shared function logic 3810, shared and/or cache memory 3812, a fixed functionality geometry pipeline 3814, and additional fixed function logic 3816 to accelerate various graphics and computational operations include. In at least one embodiment, shared functional logic 3810 may include logic units (e.g., scanners, math, and/or inter-thread communication logic) that may be shared by any of the N sub-cores within graphics core 3800 . Shared and/or cache memory 3812 may be a last level cache for N sub-cores 3801A-3801F within graphics core 3800 and may also serve as shared memory accessible by multiple sub-cores. In at least one embodiment, fixed functionality geometry pipeline 3814 may be included within fixed functional block 3830 in place of fixed functionality geometry pipeline 3836 and may include the same or similar logical units.

In at least one embodiment, graphics core 3800 includes additional fixed functionality logic 3816 that may include various fixed functionality acceleration logic for use by graphics core 3800 . In at least one embodiment, the additional logic with fixed functionality 3816 includes an additional geometry pipeline for use only in positional shading. In position-only shading there are at least two geometry pipelines, while in a full geometry pipeline within the fixed functionality geometry pipeline 3816, 3836 and a cull pipeline, which is a additional geometry pipeline that may be included in additional fixed functionality logic 3816 . In at least one embodiment, the fade-out pipeline is a lightweight version of a full geometry pipeline. In at least one embodiment, a full pipeline and a hide pipeline may run different instances of an application, with each instance having a separate context. In at least one embodiment, only positional shading can suppress long fading runs for discarded triangles, allowing shading to complete earlier in some cases. For example, in at least one embodiment, the hide pipeline logic within the additional fixed function logic can perform 3816 position shading in parallel with a main application and produce critical results generally faster than a full pipeline because the hide pipeline retrieves the position attribute of vertices and shades without perform rasterization and rendering of pixels into a frame buffer. In at least one embodiment, the hiding pipeline may use generated critical results to compute visibility information for all triangles, regardless of whether those triangles are hidden. In at least one embodiment, a complete pipeline (which in this case may be referred to as a rendering pipeline) may use the visibility information to skip hidden triangles and shade only the visible triangles, which are eventually passed to a rasterization phase.

In at least one embodiment, the additional fixed function logic 3816 may also include machine learning acceleration logic, such as fixed function matrix multiplication logic, for implementations involving machine learning optimizations or inference.

In at least one embodiment, within each graphics sub-core 3801A-3801F includes a set of execution resources that can be used to run graphics, media, and compute perform operations in response to requests from the graphics pipeline, media pipeline, or shading programs. In at least one embodiment, the graphics sub-cores 3801A-3801F include multiple EU arrays 3802A-3802F, 3804A-3804F, thread distribution and inter-thread communication logic (TD/IC) 3803A-3803F, a 3D (e.g., texture 3805A-3805F scanner, 3806A-3806F media scanner, 3807A-3807F shading processor, and 3808A-3808F shared local memory (SLM). The EU arrays 3802A-3802F, 3804A-3804F each include multiple execution units, which are general purpose graphics processing units capable of performing floating point and integer/fixed point logic operations in support of graphics, media, or computational operations , including graphics, media, or compute shader programs. In at least one embodiment, the 3803A-3803F TD/IC logic performs local thread dispatch and thread control operations for execution units within a sub-core and facilitates communication between threads executing on execution units of a sub-core. In at least one embodiment, the 3D scanner 3805A-3805F can read texture or other 3D graphics data into memory. In at least one embodiment, the 3D scanner may read texture data differently based on a configured scan state and texture format associated with a given texture. In at least one embodiment, media scanner 3806A-3806F may perform similar operations based on a type and format associated with media data. In at least one embodiment, each graphics sub-core 3801A-3801F may alternately include a unified 3D and media scanner. In at least one embodiment, threads executing on execution units within each sub-core 3801A-3801F can use shared local memory 3808A-3808F within each sub-core so that threads executing within a thread group can use a shared pool of On-chip memory can run.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, some or all of the inference and/or training logic 1515 may be integrated into the graphics processor 3810. For example, in at least one embodiment, the training and/or inference techniques described herein may utilize one or more ALUs resident in 3D Pipeline 3810, Graphics Microcontroller 3838, Fixed Functionality Geometry Pipeline 3814 and 3836, or other logic in 35 are included. In addition, in at least one embodiment, the inferencing and/or training operations described herein may be performed with a method other than that described in 15A or 15B shown logic are carried out. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and/or registers (shown or not) that configure ALUs of graphics processor 3800 to implement one or more machine learning algorithms described herein , architectures, use cases or training techniques of neural networks.

39A-39B 12 show thread execution logic 3900 comprising an array of processing elements of a graphics processor core, according to at least one embodiment. 39A FIG. 12 shows at least one embodiment in which thread execution logic 3900 is used. 39B 12 shows exemplary internal details of an execution unit according to at least one embodiment.

As in 39A As shown, the thread execution logic 3900 includes, in at least one embodiment, a shader processor 3902, a thread dispatcher 3904, the instruction cache 3906, a scalable array of execution units that includes a plurality of execution units 3908A-3908N, a scanner 3910, a data cache 3912, and a Data port 3914 includes. In at least one embodiment, a scalable array of execution units may be dynamically scaled by adding one or more execution units (e.g., any of execution units 3908A, 3908B, 3908C, 3908D through 3908N-1 and 3908N) based on the computational needs of a workload can be enabled or disabled. In at least one embodiment, scalable execution units are interconnected by an interconnect fabric that connects to each of the execution units. In at least one embodiment, thread execution logic 3900 includes one or more connections to memory, such as: e.g., to system memory or cache memory, via one or more of instruction cache 3906, data port 3914, scanner 3910, and execution units 3908A-3908N. In at least one embodiment, each execution unit (eg, 3908A) is a self-contained programmable general purpose processing unit capable of executing multiple simultaneous hardware threads while processing multiple data items in parallel for each thread. In In at least one embodiment, the array of execution units 3908A-3908N is scalable to include any number of individual execution units.

In at least one embodiment, execution units 3908A-3908N are used primarily to execute shading programs. In at least one embodiment, shader processor 3902 may process various shader programs and dispatch threads of execution associated with shader programs via thread dispatcher 3904 . In at least one embodiment, thread dispatcher 3904 includes logic for arbitrating thread initiation requests from graphics and media pipelines and instantiating requested threads on one or more execution units in execution units 3908A-3908N. For example, in at least one embodiment, a geometry pipeline may dispatch vertex, tessellation, or geometry shaders to thread execution logic for processing. In at least one embodiment, thread dispatcher 3904 may also process thread creation requests from executing shaders at runtime.

In at least one embodiment, execution units 3908A-3908N support an instruction set that includes native support for many standard 3D graphics shading instructions such that shading programs from graphics libraries (e.g., Direct 3D and OpenGL) execute with minimal translation. In at least one embodiment, the execution units support vertex and geometry processing (e.g., vertex programs, geometry programs, vertex shaders), pixel processing (e.g., pixel shaders, fragment shaders), and general processing (e.g., computations). - and Media Shader). In at least one embodiment, each of the execution units 3908A-3908N, comprising one or more arithmetic logic units (ALUs), is capable of executing multiple instruction SIMD (single instruction multiple data) execution, and the multi-threaded Operation enables an efficient execution environment despite higher latency memory accesses. In at least one embodiment, each hardware thread within each execution unit has a dedicated high bandwidth register file and an associated independent thread state. In at least one embodiment, execution occurs multiple times per clock on pipelines capable of performing integer, single and double precision floating point operations, SIMD branches, logical operations, transcendent operations, and other miscellaneous operations. In at least one embodiment, while waiting for data from memory or one of the shared functions, dependency logic within execution units 3908A-3908N causes a waiting thread to sleep until the requested data is returned. In at least one embodiment, while a waiting thread is idle, hardware resources may be used to process other threads. For example, in at least one embodiment, during a delay associated with a vertex shader operation, an execution unit may execute operations for a pixel shader, fragment shader, or other type of shader program including another vertex shader.

In at least one embodiment, each execution unit in execution units 3908A-3908N operates on arrays of data elements. In at least one embodiment, a number of data items is the "run size" or number of channels for an instruction. In at least one embodiment, an execution channel is a logical unit of execution for data element access, masking, and flow control within instructions. In at least one embodiment, a number of channels may be independent of a number of physical arithmetic logic units (ALUs) or floating point units (FPUs) for a particular graphics processor. In at least one embodiment, execution units 3908A-3908N support integer and floating point data types.

In at least one embodiment, an instruction set of an execution unit includes SIMD instructions. In at least one embodiment, different data items may be stored as a packed data type in a register, and the execution unit processes different items based on the data size of the items. For example, in at least one embodiment, when operating with a 256-bit wide vector, 256 bits of a vector are stored in a register and an execution unit operates on a vector as four separate 64-bit packed data elements (quad-word (QW )), eight separate 32-bit packed data items (Double Word (DW) size data items), sixteen separate 16-bit packed data items (Word (W) size data items), or thirty-two separate 8-bit data items (Double Word (W) size data items). size bytes (B)). However, in at least one embodiment, different vector widths and register sizes are possible.

In at least one embodiment, one or more execution units may be combined into a merged execution unit 3909A-3909N with thread control logic (3907A-3907N) common to all merged EUs. In at least one embodiment, multiple EUs may be merged into an EU group. In at least one embodiment, each EU in a merged EU group can be configured to run a separate SIMD hardware thread. The number of EUs in a merged EU group may vary according to different embodiments. In at least one embodiment, different SIMD widths can be implemented per EU, including but not limited to SIMD8, SIMD16, and SIMD32. In at least one embodiment, each merged graphics execution unit 3909A-3909N includes at least two execution units. For example, in at least one embodiment, the merged execution unit 3909A includes a first EU 3908A, a second EU 3908B, and control of the thread 3907A that is common to the first EU 3908A and the second EU 3908B. In at least one embodiment, thread control logic 3907A controls the threads executing on merged graphics execution unit 3909A such that each EU can execute within merged execution units 3909A-3909N using a common instruction pointer register.

In at least one embodiment, thread execution logic 3900 includes one or more internal instruction caches (e.g., 3906) to cache thread instructions for execution units. In at least one embodiment, one or more data caches (e.g., 3912) are included to cache thread data during thread execution. In at least one embodiment, a sampler 3910 is included to provide texture sampling for 3D operations and media sampling for media operations. In at least one embodiment, sampler 3910 includes specialized texture or media sampling functionality to process texture or media data during the sampling process before sampled data is provided to an execution unit.

During execution, in at least one embodiment, graphics and media pipelines send thread initiation requests to thread execution logic 3900 via thread creation and distribution logic. In at least one embodiment, once a set of geometric objects has been processed and rasterized into pixel data, Pixel processor logic (e.g. pixel shading logic, fragment shading logic, etc.) is called within the shading processor 3902 to calculate other output information and cause the results to be written to output surfaces (e.g. color buffers, depth buffers, stencil buffers, etc.). In at least one embodiment, a pixel shader or fragment shader calculates values of various vertex attributes to be interpolated across a rasterized object. In at least one embodiment, the pixel processor logic within the shading processor 3902 then executes a pixel or fragment shading program provided by an API (Application Programming Interface). In at least one embodiment, shader processor 3902 dispatches threads via thread dispatcher 3904 to an execution unit (e.g., 3908A) for execution of a shader program. In at least one embodiment, shading processor 3902 uses texture sampling logic in sampler 3910 to access texture data in memory-associated texture maps. In at least one embodiment, arithmetic operations on texture data and input geometry data calculate pixel color data for each geometric fragment or exclude one or more pixels from further processing.

In at least one embodiment, data port 3914 provides a memory access mechanism for thread execution logic 3900 to output processed data to memory for further processing on a graphics processor output pipeline. In at least one embodiment, data port 3914 includes or is coupled to one or more cache memories (e.g., data cache 3912) to cache data for memory access via a data port.

As in 39B As shown, a graphics execution unit 3908, in at least one embodiment, may include an instruction fetch unit 3937, a general register file array (GRF) 3924, an architecture register file array (ARF) 3926, a thread arbiter 3922, a transmit unit 3930, a branch unit 3932, a set of SIMD floating point units (FPUs) 3934, and in at least one embodiment a set of dedicated integer SIMD ALUs 3935. In at least one embodiment, register files GRF 3924 and ARF 3926 include a set of general register files and Architectural register files associated with each concurrent hardware thread that may be active in graphics execution unit 3908. In at least one embodiment, per-thread architecture state is maintained in ARF 3926 while data used during thread execution is stored in GRF 3924 . In at least one embodiment, the execution state of each thread, including the instruction pointers for each thread, may be held in ARF 3926 in thread-specific registers.

In at least one embodiment, the graphics execution unit 3908 has an architecture that is a combination of Simultaneous Multi-Threading (SMT) and Fine-Grain Interleaved Multi-Threading (IMT). In at least one embodiment, the architecture has a modular configuration that can be fine-tuned at design time based on a target number of simultaneous threads and a number of registers per execution unit, where execution unit resources are allocated to the logic needed to execute multiple simultaneous threads is used.

In at least one embodiment, graphics execution unit 3908 may concurrently issue multiple instructions, each of which may be different instructions. In at least one embodiment, the thread arbiter 3922 of the graphics execution unit thread 3908 may send instructions to one of the dispatch unit 3930, branch unit 3942, or SIMD FPU(s) 3934 for execution. In at least one embodiment, each thread of execution can access 128 general purpose registers in the GRF 3924, with each register capable of storing 32 bytes that can be accessed as a SIMD 8-element vector of 32-bit data elements. In at least one embodiment, each execution unit thread has access to 4K bytes within the GRF 3924, although the execution units are not so limited and in other embodiments more or fewer register resources may be provided. In at least one embodiment, up to seven threads may be executing concurrently, although the number of threads per execution unit may also vary by embodiment. In at least one embodiment, where seven threads can access 4K bytes, the GRF 3924 can store a total of 28K bytes. In at least one embodiment, flexible addressing modes may allow registers to be addressed together to effectively build wider registers or to represent dashed rectangular block data structures.

In at least one embodiment, memory operations, scanner operations, and other higher latency system communications are dispatched via "send" commands executed by the messaging sending unit 3930 . In at least one embodiment, branch instructions are sent to a dedicated branch unit 3932 to facilitate SIMD divergence and eventual convergence.

In at least one embodiment, graphics execution unit 3908 includes one or more SIMD floating point units (FPU(s)) 3934 for performing floating point operations. In at least one embodiment, FPU(s) 3934 also support integer math. In at least one embodiment, FPU(s) 3934 may perform SIMD up to M 32-bit floating point (or integer) operations, or SIMD up to 2M 16-bit integer or 16-bit floating point operations. In at least one embodiment, at least one of the FPU(s) is equipped with extended math functions to support high throughput transcendent math functions and 64-bit double-precision floating-point operations. In at least one embodiment, there is also a set of 8-bit integer SIMD ALUs 3935 that can be specifically optimized to perform operations associated with machine learning computations.

In at least one embodiment, arrays of multiple instances of graphics core array 3908 may be instantiated in a graphics sub-core grouping (e.g., subsection). In at least one unit of execution unit 3908, in at least one embodiment, instructions may be executed through a variety of execution channels. In at least one embodiment, each thread executing on graphics execution unit 3908 executes in a different channel.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, some or all of the inference and/or training logic 1515 may be incorporated into the execution logic 3900. Furthermore, in at least one embodiment, inference and/or training of operations described herein may use logic other than in the 15A or 15B logic shown can be used. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and/or registers (shown or not) that configure ALUs of execution logic 3900 to execute one or more machine learning algorithms described herein , architectures, use cases or training techniques for neural networks.

40 12 shows a parallel processing unit (“PPU”) 4000 according to at least one embodiment. In at least one embodiment, PPU 4000 is configured with machine-readable code that, when executed by PPU 4000, causes PPU 4000 to perform some or all of the processes and techniques described in this disclosure. In at least one embodiment, PPU 4000 is a multithreaded processor implemented on one or more integrated circuit devices and using multithreading as a latency reduction technique designed to execute computer-readable instructions (also referred to as machine-readable instructions or just instructions) on multiple threads to process in parallel. In at least one embodiment, a thread refers to a thread of execution and is an instantiation of an instruction set configured to be executed by PPU 4000. In at least one embodiment, PPU 4000 is a graphics processing unit ("GPU") configured to implement a graphics rendering pipeline for processing three-dimensional ("3D") graphics data to generate two-dimensional ("2D") image data for the generate a display on a display device, such as a liquid crystal display ("LCD") device. In at least one embodiment, PPU 4000 is used to perform computations such as linear algebra operations and machine learning operations. 40 Figure 13 illustrates an example parallel processor for purposes of illustration only and is intended as a non-limiting example of processor architectures contemplated by this disclosure, where any suitable processor may be substituted and/or supplemented.

In at least one embodiment, one or more PPUs 4000 are configured to accelerate high performance computing ("HPC"), data centers, and machine learning applications. In at least one embodiment, PPU 4000 is configured to accelerate systems and applications for deep learning, including but not limited to the following: autonomous vehicle platforms, deep learning, high-fidelity speech, image, text recognition systems, intelligent video analytics, molecular simulations, Drug discovery, disease diagnosis, weather forecasting, big data analysis, astronomy, molecular dynamics simulation, financial modeling, robotics, factory automation, real-time language translation, online search optimizations and personalized user recommendations, and much more.

In at least one embodiment, the PPU 4000 includes, without limitation, an input/output ("I/O") unit 4006, a front-end unit 4010, a scheduler unit 4012, a work distribution unit 4014, a hub 4016, a crossbar ("Xbar ") 4020, one or more general processing clusters ("GPCs") 4018, and one or more partition units ("memory partition units") 4022. In at least one embodiment, the PPU 4000 is connected to a host processor or other PPUs 4000 via one or more High Speed GPU Connections (“GPU Connections”) 4008 connected. In at least one embodiment, PPU 4000 is connected via interconnect 4002 to a host processor or other peripheral devices. In at least one embodiment, PPU 4000 is coupled to local storage, which includes one or more storage devices (“storage”) 4004 . In at least one embodiment, memory devices 4004 include, without limitation, one or more Dynamic Random Access Memory ("DRAM") devices. In at least one embodiment, one or more DRAM devices are configured and/or configurable as high-bandwidth memory (“HBM”) memory subsystems, with multiple DRAM chips stacked in each device.

In at least one embodiment, GPU high-speed link 4008 may refer to a wired, multi-lane communication link used by systems to scale and incorporate one or more PPUs 4000 in combination with one or more central processing units ("CPUs") that Cache coherency between PPUs 4000 and CPUs as well as CPU mastering supported. In at least one embodiment, data and/or commands are transmitted through the GPU high-speed interconnect unit 4008 via the hub 4016 to/from other units of the PPU 4000, such as one or more copy engines, video encoders, video decoders, power management units, and other components. in the 40 cannot be shown explicitly.

In at least one embodiment, I/O unit 4006 is configured to receive communications (e.g., commands, data) from a host processor (not in 40 shown) transmits and receives over system bus 4002. In at least one embodiment, I/O unit 4006 communicates with the host processor directly over system bus 4002 or through one or more intermediary devices such as a memory bridge. In at least one embodiment, I/O unit 4006 may communicate with one or more other processors, such as one or more PPUs 4000, via system bus 4002. In at least one embodiment, I/O unit 4006 implements a Peripheral Component Interconnect Express ("PCIe") interface for communication over a PCIe bus. In at least one embodiment, I/O unit 4006 implements interfaces for communicating with external devices.

In at least one embodiment, I/O unit 4006 decodes packets received over system bus 4002. In at least one embodiment, at least some packets represent instructions configured to cause PPU 4000 to perform various operations. In at least one embodiment, I/O unit 4006 transmits decoded commands to various other units of PPU 4000 as specified by commands. In at least one embodiment, commands are transmitted to the front end unit 4010 and/or the hub 4016 or other units of the PPU 4000 such as one or more copy engines, a video encoder, a video decoder, a power management unit, and so forth. (not expressly in 40 shown). In at least one embodiment, I/O unit 4006 is configured to route communications between and between different PPU 4000 logical units.

In at least one embodiment, a program executed by the host processor encodes an instruction stream in a buffer that provides PPU 4000 workloads for processing. In at least one embodiment, a workload includes instructions and data to be processed by those instructions. In at least one embodiment, the buffer is an area in memory that is accessible (e.g., read/write) by both the host processor and the PPU 4000 - a host interface unit may be configured to handle memory requests that exceed system bus 4002 from I/O unit 4006 via system bus 4002 accesses the buffer in a system memory connected to system bus 4002. In at least one embodiment, the host processor writes an instruction stream to the buffer and then sends a pointer to the beginning of the instruction stream to PPU 4000 so that front end unit 4010 receives pointers to one or more instruction streams and manages one or more instruction streams, instructions reads from command streams and forwards commands to various units of PPU 4000.

In at least one embodiment, the front-end unit 4010 is coupled to the scheduler unit 4012, which configures various GPCs 4018 to process tasks defined by one or more instruction streams. In at least one embodiment, the scheduler unit 4012 is configured to track status information on various tasks managed by the scheduler unit 4012, where status information may indicate which of the GPCs 4018 a task is associated with, whether the task is active or inactive, the priority level associated with the task is associated, etc. In at least one embodiment, scheduler unit 4012 schedules execution of a variety of tasks on one or more GPCs 4018.

In at least one embodiment, scheduler engine 4012 is coupled to work distribution engine 4014, which is configured to distribute tasks to GPCs 4018 for execution. In at least one embodiment, the work distribution unit 4014 keeps track of a number of scheduled tasks received from the scheduler unit 4012, and the work distribution unit 4014 maintains a pending task pool and an active task pool for each of the GPCs 4018. In at least one embodiment, the pending task pool includes a number of slots (e.g., 32 slots) containing tasks assigned for processing by a particular GPC 4018; the active task pool may include a number of slots (e.g., 4 slots) for tasks that are being actively processed by the GPCs 4018 such that when one of the GPCs 4018 completes execution of a task, that task is removed from the active task pool for GPC 4018 is preempted and one of the other tasks is selected from the pool of pending tasks and scheduled to run on GPC 4018. In at least one embodiment, when an active task is idle on GPC 4018, e.g. while waiting for a data dependency to be resolved, the active task is evicted from GPC 4018 and returned to the pending task pool while another task is removed from the pool of pending tasks is selected and scheduled to run on GPC 4018.

In at least one embodiment, the work distribution unit 4014 communicates with one or more GPCs 4018 via the XBar 4020. In at least one embodiment, the XBar 4020 is an interconnection network that couples many units of the PPU 4000 to other units of the PPU 4000 and can be configured to the work distribution unit 4014 is paired with a particular GPC 4018. In at least one embodiment, one or more other units of PPU 4000 may also be connected to XBar 4020 via hub 4016 .

In at least one embodiment, the tasks are managed by the scheduler engine 4012 and dispatched to one of the GPCs 4018 by the work distribution engine 4014 . GPC 4018 is configured to process tasks and produce results. In at least one embodiment, the results may be used by other tasks within GPC 4018, forwarded to another GPC 4018 via XBar 4020, or stored in memory 4004. In at least one embodiment, results may be written to memory 4004 via partition units 4022 that implement a memory interface for reading and writing data to/from memory 4004 . In at least one embodiment, results may be transmitted to another PPU 4004 or CPU via the GPU high speed link 4008 . In at least one embodiment, PPU 4000 includes, without limitation, a number U of partition units 4022 equal to the number of separate and distinct storage devices 4004 coupled to PPU 4000 . In at least one embodiment, partition unit 4022 is used in conjunction with 42 described in more detail herein.

In at least one embodiment, a host processor executes a driver kernel that implements an application programming interface ("API") that allows one or more applications executing on the host processor to schedule operations for execution on PPU 4000. In at least one embodiment, multiple computing applications are executed concurrently by PPU 4000, and PPU 4000 provides isolation, quality of service ("QoS"), and independent address spaces for multiple computing applications. In at least one embodiment, an application generates commands (e.g., in the form of API calls) that cause the driver kernel to generate one or more tasks for execution by PPU 4000, and the driver kernel issues tasks to one or more data streams , which are processed by the PPU 4000. In at least one embodiment, each task includes one or more groups of related threads, which may be referred to as a warp. In at least one embodiment, a warp includes a plurality of related threads (eg, 32 threads) that may execute in parallel. In at least one embodiment, cooperating threads may refer to a plurality of threads that contain instructions to perform tasks and that exchange data over a shared memory. In at least one embodiment, threads and cooperating threads are described in accordance with at least one embodiment in connection with 42 described in more detail.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, a deep learning application processor is used to train a machine learning model, such as a neural network, to predict or infer the information provided to PPU 4000 . In at least one embodiment, deep learning application processor 4000 is used to infer or predict information based on a trained machine learning model (e.g., a neural network) trained by another processor or system or by PPU 4000. In at least one embodiment, PPU 4000 may be used to perform one or more of the neural network use cases described herein.

41 4 shows a general processing cluster (“GPC”) 4100 according to at least one embodiment. In at least one embodiment, GPC 4100 is GPC 4018 off 40 . In at least one embodiment, each GPC 4100 includes, without limitation, a number of hardware units for processing tasks, and each GPC 4100 includes, without limitation, a pipeline manager 4102, a pre-raster operations unit ("PROP") 4104, a rasterization engine 4108 , a work distribution crossbar (“WDX”) 4116, a memory management unit (“MMU”) 4118, one or more data processing clusters (“DPCs”) 4106, and any suitable combination of elements.

In at least one embodiment, the operation of GPC 4100 is controlled by pipeline manager 4102 . In at least one embodiment, pipeline manager 4102 manages the configuration of one or more DPCs 4106 for processing GPC 4100 assigned tasks. In at least one embodiment, pipeline manager 4102 configures at least one of one or more DPCs 4106 to implement at least a portion of a graphics rendering pipeline. In at least one embodiment, DPC 4106 is configured to run a shading program on a streaming programmable multiprocessor ("SM") 4114 . In at least one embodiment, pipeline manager 4102 is configured to forward packets received from a work distribution unit to appropriate logical units within GPC 4100, and some packets may be sent to hardware units with fixed functionality in PROP 4104 and/or the grid Engine 4108 may be forwarded, while other packets may be forwarded to DPCs 4106 for processing by primitive engine 4112 or SM 4114. In at least one embodiment, the pipeline manager 4102 configures at least one of the DPCs 4106 to implement a neural network model and/or a computational pipeline.

In at least one embodiment, PROP unit 4104 is configured to forward the data generated by engine 4108 and DPCs 4106 to a raster operations ("ROP") unit in partition unit 4022, described above in connection with 40 is described in more detail. In at least one embodiment, the PROP unit 4104 is configured to perform color mixing optimizations, organize pixel data, perform address translations, and more. In at least one embodiment, raster engine 4108 includes, without limitation, a set of fixed functionality hardware units configured to perform various raster operations, and raster engine 4108 includes, without limitation, a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine rasterization engine, a tile coalescing engine, and any suitable combination thereof. In at least one embodiment, the setup engine receives transformed vertices and generates plane equations associated with a geometric primitive defined by vertices; Plane equations are passed to the coarse raster engine to generate coverage information (eg, an x,y coverage mask for a tile) for the primitive; the output of the coarse raster engine is sent to the culling engine, where fragments associated with the primitive that fail a z-test are hidden, and to a clipping engine, where fragments that fall outside a Visibility cone are cut off. In at least one embodiment, the fragments that survive clipping and culling are passed to the fine raster engine to generate attributes for pixel fragments based on the plane equations generated by the setup engine. In at least one embodiment, the output of engine 4108 includes fragments that are processed by any suitable entity, such as a fragment shader implemented in DPC 4106 .

Each DPC 4106 included in GPC 4100 consists, in at least one embodiment, of, without limitation, an M-Pipe Controller ("MPC") 4110; the primitive engine 4112; one or more SMs 4114 and any suitable combination thereof. In at least one embodiment, MPC 4110 controls the operation of DPC 4106 and forwards packets received from pipeline manager 4102 to appropriate entities in DPC 4106 . In at least one embodiment, packets associated with a vertex are forwarded to primitive engine 4112, which is configured to fetch vertex attributes associated with the vertex from memory; in contrast, packets associated with a shader program may be forwarded to SM 4114.

In at least one embodiment, without limitation, SM 4114 includes a programmable streaming processor configured to process tasks represented by a number of threads. In at least one embodiment, SM 4114 is multi-threaded and configured to execute the plurality of threads (e.g., 32 threads) from a particular set of threads concurrently and uses a single-threaded (SIMD) architecture. Instruction, Multiple-Data"), where each thread in a group of threads (e.g., a warp) is configured to process a different set of data based on the same instruction set. In at least one embodiment, all threads in a group of threads execute the same instructions. In at least one embodiment, SM 4114 implements a single-instruction, multiple-thread (SIMT) architecture, where each thread in a group of threads is configured to process a different data set based on the same instruction set, but with individual threads in a group of threads are allowed to diverge during execution. In at least one embodiment, a program counter, call stack, and execution state are maintained for each warp, enabling concurrency between warps and serial execution within warps becomes when the threads diverge within a warp. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, allowing equal concurrency between all threads within and between warps. In at least one embodiment, execution state is maintained for each individual thread, and threads executing the same instructions can be converged and executed in parallel for better efficiency. At least one embodiment of SM 4114 is described in more detail herein.

In at least one embodiment, MMU 4118 provides an interface between GPC 4100 and the memory partition unit (eg, partition unit 4022 of 40 ) and MMU 4118 provides virtual address to physical address translation, memory protection, and arbitration of memory requests. In at least one embodiment, MMU 4118 provides one or more translation lookaside buffers ("TLBs") for translation of virtual addresses to physical addresses in memory.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, a deep learning application processor is used to train a machine learning model, such as a neural network, to predict or infer information provided to GPC 4100 . In at least one embodiment, GPC 4100 is used to infer or predict information based on a trained machine learning model (eg, a neural network) trained by another processor or system or by GPC 4100 . In at least one embodiment, GPC 4100 may be used to perform one or more of the neural network use cases described herein.

42 12 shows a memory partition unit 4200 of a parallel processing unit ("PPU"), according to at least one embodiment. In at least one embodiment, the memory partition unit 4200 includes, without limitation, a raster operations unit ("ROP") 4202, an L2 cache ("L2") 4204, a memory interface 4206, and any suitable combination thereof . Memory interface 4206 is coupled to memory. Memory interface 4206 may implement 32, 64, 128, 1024 bit data buses or the like for high speed data transfer. In at least one embodiment, the PPU U includes memory interfaces 4206, one memory interface 4206 per pair of partition units 4200, with each pair of partition units 4200 connected to a corresponding device. For example, in at least one embodiment, the PPU may be coupled to up to Y memory devices, such as high-bandwidth memory stacks or double graphics data rate memory, version 5, synchronous dynamic random access memory ("GDDR5 SDRAM").

In at least one embodiment, memory interface 4206 implements a high-bandwidth second generation memory interface ("HBM2"), where Y equals half of U . In at least one embodiment, the HBM2 memory stacks reside in the same package as the PPU, providing significant power and area savings compared to traditional GDDR5 SDRAM systems. In at least one embodiment, each HBM2 stack includes, without limitation, four memory devices and Y equals 4, with each HBM2 stack including two 128-bit channels per device for a total of 8 channels and a data bus width of 1024 bits. In at least one embodiment, the memory supports Single-Error Correcting Double-Error Detecting ("SECDED") Error Correction Code ("ECC") to protect the data. ECC provides higher reliability for computing applications that are sensitive to data corruption.

In at least one embodiment, the PPU implements a multi-level memory hierarchy. In at least one embodiment, memory partition unit 4200 supports unified memory to provide a single unified virtual address space for central processing unit ("CPU") and PPU memory, thereby enabling data sharing across virtual memory systems. In at least one embodiment, the frequency of accesses by a PPU to memory residing on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU that is accessing pages more frequently. In at least one embodiment, GPU high speed interconnect 4008 supports address translation services that allow the PPU to directly access a CPU's page tables and allow the PPU full access to CPU memory.

In at least one embodiment, the copy engines transfer data between multiple PPUs or between PPUs and CPUs. In at least one embodiment, copy engines can generate page faults for addresses not mapped into page tables, and memory partition unit 4200 then services page faults by mapping addresses into page tables, after which the copy engine performs the transfer. In at least one embodiment, memory for multiple copy engine operations is pinned (i.e., non-swappable) between multiple processors, thereby significantly reducing available memory. In at least one embodiment with hardware page faults, addresses can be passed to the copy engines regardless of whether memory pages are resident, and the copy process is transparent.

Data from memory 4004 of 40 or other system memory are retrieved from the memory partition unit 4200 and stored in the on-chip L2 cache 4204 shared between different GPCs according to at least one embodiment. Each memory partition unit 4200 includes, without limitation, at least a portion of the L2 cache associated with a corresponding memory device, in at least one embodiment. In at least one embodiment, lower level caches are implemented in various entities within GPCs. In at least one embodiment, each of the SMs 4114 may implement a level 1 ("L1") cache, where the L1 cache is private memory associated with a particular SM 4114, and data is fetched from the L2 cache 4204 and stored in each of the L1 caches for processing in SMs 4114 functional units. In at least one embodiment, L2 cache 4204 is coupled to memory interface 4206 and XBar 4020.

The ROP unit 4202, in at least one embodiment, performs graphical rasterization operations related to pixel colors, such as color compression, pixel blending, and more. ROP unit 4202, in at least one embodiment, performs a depth check in conjunction with rasterization engine 4108 and receives a depth for a sample location associated with a pixel fragment from rasterization engine 4108's masking engine. In at least one embodiment, the depth is tested against a corresponding depth in a depth buffer for a sample location associated with the fragment. In at least one embodiment, if the fragment passes the depth test for the sample location, the ROP unit 4202 updates the depth buffer and transmits the result of the depth test to the raster engine 4108. Note that the number of partition units 4200 depends on the number of GPCs can differ and therefore each ROP unit 4202 can be coupled to each of the GPC in at least one embodiment. In at least one embodiment, ROP unit 4202 keeps track of packets received from different GPCs and determines which packets a result generated by ROP unit 4202 is forwarded via XBar 4020 to.

43 12 shows a streaming multiprocessor ("SM") 4300, according to at least one embodiment. In at least one embodiment, SM 4300 is the SM of 41 . In at least one embodiment, SM 4300 includes, without limitation, instruction cache 4302; one or more scheduler units 4304; a register file 4308; one or more processing cores (“Cores”) 4310; one or more Special Function Units (“SFUs”) 4312; one or more load/store units ("LSUs") 4314; an interconnection network 4316; a shared memory/level 1 ("L1") cache 4318; and any suitable combination thereof. In at least one embodiment, a work distribution unit distributes tasks for execution across general processing clusters ("GPCs") of parallel processing units ("PPUs"), with each task associated with a particular data processing cluster ("DPC") within a GPC and, if the task is associated with a shader program , assigned to one of the SMs 4300. In at least one embodiment, scheduler unit 4304 receives tasks from the work distribution unit and manages instruction scheduling for one or more thread blocks associated with SM 4300. In at least one embodiment, scheduler unit 4304 schedules thread blocks for execution as parallel thread warps, with each thread block having at least one warp associated therewith. In at least one embodiment, each warp executes threads. In at least one embodiment, scheduler unit 4304 manages a variety of different thread blocks, assigns the warps to different thread blocks, and then dispatches instructions from a variety of different cooperative groups to different functional units (e.g., processing cores 4310, SFUs 4312, and LSUs 4314) during each clock cycle. .

In at least one embodiment, cooperative groups can refer to a programming model for organizing groups of communicating threads that allows developers to specify the granularity at which the threads communicate, allowing for the design of richer, more efficient parallel decompositions allows. In at least one embodiment, cooperative launch APIs support synchronization between thread blocks for executing parallel algorithms. In at least one embodiment, applications of conventional programming models are provided with a single, simple construct for synchronizing cooperating threads: a barrier across all threads of a thread block (eg, the syncthreads( ) function). However, in at least one embodiment, programmers can define groups of threads with a granularity smaller than that of a thread block and synchronize within the defined groups to provide greater performance, design flexibility, and software reuse in the form of collective group-wide functional interfaces enable. In at least one embodiment, cooperative groups allow programmers to explicitly define groups of threads at sub-block (ie, as small as a single thread) and multi-block granularities and perform collective operations such as synchronizing threads in a cooperative group. The programming model supports clean composition across software boundaries, allowing libraries and utility functions to be safely synchronized within their local context without making assumptions about convergence. In at least one embodiment, the cooperative set primitives enable new patterns of cooperative parallelism, including, without limitation, producer-consumer parallelism, opportunistic parallelism, and global synchronization across an entire lattice of thread blocks.

In at least one embodiment, dispatch unit 4306 is configured to dispatch instructions to one or more functional units, and scheduler unit 4304 includes, without limitation, two dispatch units 4306 that allow two different instructions from the same chain to be scheduled during each clock cycle. In at least one embodiment, each scheduler unit 4304 includes a single distribution unit 4306 or additional distribution units 4306.

In at least one embodiment, each SM 4300 includes a register file 4308 that provides a set of registers for SM 4300 functional units. In at least one embodiment, register file 4308 is partitioned among each of the functional units such that each functional unit is allocated a specific portion of register file 4308. In at least one embodiment, register file 4308 is partitioned among various warps executed by SM 4300, and register file 4308 is provided as temporary storage for operands associated with functional unit data paths. In at least one embodiment, each SM 4300 includes, without limitation, a plurality of L processing cores 4310. In at least one embodiment, SM 4300 includes, without limitation, a large number (e.g., 128 or more) of different processing cores 4310. In at least one embodiment, each processing core 4310 includes in at least one embodiment, without limitation, a fully pipelined, single-precision, double-precision, and/or mixed-precision processing unit that includes, without limitation, a floating point arithmetic logic unit and an integer arithmetic logic unit. In at least one embodiment, floating point arithmetic logic units implement the IEEE 754-2008 standard for floating point arithmetic. In at least one embodiment, the processing cores 4310 include, without limitation, 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

Tensor cores are configured to perform matrix operations, according to at least one embodiment. In at least one embodiment, processing cores 4310 include one or more tensor cores. In at least one embodiment, the tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inference. In at least one embodiment, each tensor core operates on a 4x4 matrix and performs a matrix multiplication and accumulation operation D = A X B + C, where A, B, C, and D are 4x4 matrices.

In at least one embodiment, the inputs A and B to the matrix multiplication are 16-bit floating point matrices and the accumulation matrices C and D are 16-bit floating point or 32-bit floating point matrices. In at least one embodiment, the tensor cores operate on 16-bit floating point input data with 32-bit floating point accumulation. In at least one embodiment, 16-bit floating point multiplication uses 64 operations and results in a full precision product, which is then combined using 32-bit floating point addition with other intermediate products is accumulated for a 4x4x4 matrix multiplication. Tensor cores are used to perform much larger two-dimensional or higher-dimensional matrix operations built from these smaller elements, in at least one embodiment. In at least one embodiment, an API, such as the CUDA 9 C++ API, provides special matrix loading, matrix multiplication and accumulation, and matrix storage operations to efficiently use tensor cores from a CUDA C++ program. In at least one embodiment, the CUDA-level warp-level interface accepts 16x16 arrays spanning all 32 threads of the warp.

In at least one embodiment, each SM 4300 includes, without limitation, M SFUs 4312 that perform specific functions (e.g., attribute scoring, reciprocal square root, and the like). In at least one embodiment, SFUs 4312 include, without limitation, a tree traversal unit configured to traverse a hierarchical tree data structure. In at least one embodiment, without limitation, SFUs 4312 include a texture unit configured to perform texture map filtering operations. In at least one embodiment, the texture units are configured to load texture maps (e.g., a 2D array of texels) from memory and sample texture maps to generate sampled texture values for use in shading programs executed by SM 4300. In at least one embodiment, the texture maps are stored in shared memory/L1 cache 4318 . In at least one embodiment, the texture units implement texture operations, such as filtering operations, using mip-maps (e.g., texture maps with varying levels of detail), in accordance with at least one embodiment. In at least one embodiment, each SM 4300 includes, without limitation, two texture units.

Each SM 4300 includes, without limitation, N LSUs 4314 that implement load and store operations between shared memory/L1 cache 4318 and register file 4308, according to at least one embodiment. Each SM 4300 includes, without limitation, interconnection network 4316 connecting each of the functional units to register file 4308 and LSU 4314 to register file 4308, and shared memory/L1 cache 4318, according to at least one embodiment. In at least one embodiment, interconnection network 4316 is a crossbar that can be configured to connect each of the functional units to each of the registers in register file 4308 and the LSUs 4314 to register file 4308 and locations in shared memory/L1 cache 4318 connects.

In at least one embodiment, shared memory/L1 cache 4318 is an array of on-chip memory that enables data storage and communication between SM 4300 and the primitive engine, and between threads within SM 4300, according to at least one embodiment an embodiment. In at least one embodiment, shared memory/L1 cache 4318 is, without limitation, 128 KB of storage capacity and is located in the path from SM 4300 to the partition unit. In at least one embodiment, shared memory/L1 cache 4318 is used to cache reads and writes. In at least one embodiment, one or more of shared memory/L1 cache 4318, L2 cache, and memory is backing memory.

Combining the functionalities of data cache and shared memory in a single block of memory provides improved performance for both types of memory access, in accordance with at least one embodiment. In at least one embodiment, a capacity is or is usable as a cache by programs that do not use shared memory, e.g., if the shared memory is configured to use half the capacity, texture and load /memory operations use the remaining capacity. Integration with a shared memory/L1 cache 4318 allows the shared memory/L1 cache 4318 to act as a high-throughput conduit for a stream of data while providing high-bandwidth, low-latency access to frequently reused data, according to FIG at least one embodiment. In at least one embodiment, when configured for general parallel computations, a simpler configuration can be used compared to graphics processing. In at least one embodiment, fixed functionality graphics processing units are bypassed, providing a much simpler programming model. In a general configuration for parallel computations, in at least one embodiment, the work distribution unit allocates and distributes blocks of threads directly to the DPCs. In at least one step in at least one embodiment, threads in a block execute the same program using a unique thread ID in the computation to ensure that each thread produces unique results, with SM 4300 for program execution and performing computations, shared memory/L1 cache 4318 for inter-thread communication, and LSU 4314 for reading and writing global memory via shared memory/L1 -Cache 4318 and the memory partition unit are used. In at least one embodiment, when configured for general purpose parallel computations, SM 4300 writes instructions that scheduler unit 4304 can use to start new work on DPCs.

In at least one embodiment, the PPU is in a desktop computer, a laptop computer, a tablet computer, servers, supercomputers, a smartphone (e.g., a wireless handheld device), a personal digital assistant ("PDA"), a digital camera , a vehicle, a head-mounted display, a portable electronic device, and more. In at least one embodiment, the PPU is formed on a single semiconductor substrate. In at least one embodiment, the PPU resides in a system-on-a-chip (“SoC”) along with one or more other devices such as additional PPUs, memory, a reduced instruction set CPU (“RISC”), a memory management unit (“MMU ’), a digital-to-analog converter (‘DAC’) and the like.

In at least one embodiment, the PPU may be included on a graphics card that includes one or more memory devices. The graphics card can be configured to connect to a PCIe slot on a desktop computer motherboard. In at least one embodiment, the PPU may be an integrated graphics processing unit ("iGPU") included in the motherboard chipset.

Inference and/or training logic 1515 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1515 are in connection with here 15A and or 15B provided. In at least one embodiment, a deep learning application processor is used to train a machine learning model, such as a neural network, to predict or infer information provided to SM 4300 . In at least one embodiment, SM 4300 is used to infer or predict information based on a trained machine learning model (eg, a neural network) trained by another processor or system or by SM 4300 . In at least one embodiment, SM 4300 may be used to perform one or more of the neural network use cases described herein.

In at least one embodiment, a single semiconductor platform may refer to a single unitary semiconductor-based integrated circuit or chip. In at least one embodiment, multi-chip modules may be used with increased connectivity that simulate on-chip operation and provide significant improvements over the use of a traditional central processing unit ("CPU") and bus implementation. In at least one embodiment, different modules can also be arranged separately or in different combinations of semiconductor platforms as desired by the user.

In at least one embodiment, computer programs are stored in main memory 2104 and/or secondary storage in the form of machine-readable executable code or computer control logic algorithms. Computer programs, when executed by one or more processors, enable system 2100 to perform various functions, in accordance with at least one embodiment. Storage 2104, memory, and/or any other storage are possible examples of computer-readable media. In at least one embodiment, secondary storage may refer to any suitable storage device or system, such as a hard disk drive and/or removable storage drive, including a floppy disk drive, a magnetic tape drive, a compact disk drive, a DVD drive, a recording device, a USB flash memory (Universal Serial Bus), etc. In at least one embodiment, the architecture and/or functionality of various previous figures relating to CPU 2102, parallel processing system 2112, an integrated circuit that may provide at least a portion of the capabilities of both CPU 2102, parallel processing system 2112, a chipset (e.g., a group integrated circuits designed and sold as a unit to perform related functions, etc.) and any suitable combination of integrated circuits.

In at least one embodiment, the architecture and/or functionality of various preceding figures are presented in the context of a general computing system, a circuit board system, implemented a game console system for entertainment purposes, an application specific system and more. In at least one embodiment, the computing device 2100 may take the form of a desktop computer, a laptop, a tablet computer, servers, supercomputers, a smartphone (eg, a wireless handheld device), a personal digital assistant ("PDA"), a digital camera, vehicle, head mounted display, handheld electronic device, cell phone, television, workstation, gaming console, embedded system and/or any other type of logic.

In at least one embodiment, the parallel processing system 2112 includes, without limitation, a plurality of parallel processing units ("PPUs") 2114 and associated memory 2116. In at least one embodiment, the PPUs 2114 are interconnected 2118 and a switch 2120 or multiplexer connected to a host processor or other peripheral devices. In at least one embodiment, the parallel processing system 2112 distributes compute tasks across the PPUs 2114, which may be parallelizable, e.g., as part of distributing compute tasks across multiple GPU (Graphics Processing Unit) thread blocks. In at least one embodiment, memory (e.g., for read and/or write access) is shared and made accessible across some or all PPUs 2114, although such shared memory incurs performance penalties relative to the use of local memory and registers resident in a PPU 2114 are resident. In at least one embodiment, the operation of the PPUs 2114 is synchronized through the use of an instruction such as _syncthreads(), where all threads execute in a block (e.g., across multiple PPUs 2114) to reach a certain point in code execution before continuing.

Other variations are within the spirit of the present disclosure. Thus, while the disclosed techniques are susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof have been shown in the drawings and described in detail above. It should be understood, however, that the disclosure is not intended to be limited to the particular form(s) disclosed, but on the contrary is intended to cover all modifications, alternative constructions, and equivalents as may come within the spirit and scope of the disclosure. as defined in the appended claims.

The use of the terms "a" and "the" and similar references in connection with the description of disclosed example embodiments (particularly in connection with the following claims) should be construed to include both singular and plural where used herein unless otherwise indicated or clearly contradicted by the context, and not as a definition of a term. The terms "comprise", "have", "include" and "contain" shall be construed as having unlimited terms (i.e. "including but not limited to") unless otherwise specified. The term "connected," when referring to physical connections and remaining unaltered, shall be construed as partially or fully included, attached, or connected to each other, even if anything in between. The mention of ranges of values in this document is for a brief method of individually referencing each individual value that falls within the range, unless otherwise specified herein, and each individual value is incorporated into the specification as if recited individually would. The use of the term "set" (e.g. "a set of items") or "subset" is intended to mean a non-empty set comprising one or more elements, unless otherwise indicated or the context indicates otherwise. Further, unless otherwise indicated or contradicted by context, the term "subset" of a corresponding set does not necessarily mean a true subset of the corresponding set, but subset and corresponding set may be the same.

Subjunctive language, such as phrases of the form "at least one of A, B and C" or "at least one of A, B and C", unless expressly stated otherwise or otherwise clearly contradicted by the context, is otherwise understood to mean as the context generally uses to represent that item, term, etc., can be either A or B or C or any non-empty subset of the set of A and B and C. For example, in an illustrative example of a three-member clause, subjunctive clauses "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following clauses: {A}, {B} , {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Such a subjunctive language should therefore generally not imply that at least one of A, at least one of B and at least one of C must be present for certain exemplary embodiments. Also, unless otherwise indicated or otherwise indicated by context, the term "plurality" denotes a state of plurality (e.g., "a plurality of elements" denotes multiple elements). The number of elements in a multitude is at least two, but may be more if so specified either explicitly or by context. Further, unless otherwise stated or otherwise clear from the context, the phrase "based on" means "based at least in part on" and not "based solely on."

Operation of the processes described herein may be performed in any suitable order, unless otherwise indicated herein or unless context clearly indicates the contrary. In at least one embodiment, a process such as the processes described herein (or variants and/or combinations thereof) is executed under the control of one or more computer systems configured with executable instructions and executed as code (e.g., executable instructions, one or more computer programs, or one or more applications) running collectively on one or more processors, by hardware, or combinations thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, e.g., in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, a computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagated transient electrical or electromagnetic transmission), but includes non-volatile data storage circuitry (e.g., buffers, cache, and queuing) within transceivers for volatile includes signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media storing executable instructions (or other memory for storing executable instructions) that, when executed (i.e., as a result their execution) by one or more processors of a computer system cause the computer system to perform the operations described herein. Set of non-transitory computer-readable storage media, in at least one embodiment, includes multiple non-transitory computer-readable storage media, and one or more individual non-transitory storage media of multiple non-transitory computer-readable storage media lacks all code, while multiple non-transitory computer-readable storage media collectively store all code. In at least one embodiment, executable instructions are executed such that different instructions are executed by different processors - e.g., a non-transitory computer-readable storage medium stores instructions, and a main central processing unit ("CPU") executes some of the instructions, while a graphics processing unit ("GPU") executes others executes commands. In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

Accordingly, in at least one embodiment, computer systems are configured to implement one or more services that individually or collectively perform operations of the processes described herein, and such computer systems are configured with appropriate hardware and/or software that enable operations to be performed. Furthermore, a computer system that implements at least one embodiment of the present disclosure is a single device, and in another embodiment, a distributed computer system that includes multiple devices that operate differently such that the distributed computer system performs the operations described herein and a single device does not performs all operations.

The use of examples or exemplary language (e.g., "such as") is intended solely to better illustrate embodiments of the disclosure and should not be construed as a limitation on the scope of the disclosure unless otherwise noted. Nothing in the specification should be construed as identifying any non-claimed element as essential to the practice of the disclosure.

All references cited herein, including publications, patent applications and patents, are hereby incorporated by reference to the same extent as if each reference were individually and specifically stated as incorporated by reference and reproduced in its entirety.

In the specification and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It is to be understood that these terms are not intended as synonyms for each other. Rather, in particular examples, the terms "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with one another. "Coupled" can also mean that two or more elements are not in direct contact with each other, but can still cooperate or interact with each other.

Unless expressly stated otherwise, it is to be understood that throughout the specification, terms such as "processing", "computing", "calculating", "determining" or the like refer to the actions and/or processes of a computer or computer system or similar electronic computing device that manipulate and/or transform data represented as physical, e.g. electronic, quantities in the registers and/or memories of the computing system into other data that are similar to physical quantities in the memories, registers or other such devices for storage , transmission or display of information.

Similarly, the term “processor” may refer to any device or part of a device that processes electronic data from registers and/or memory and converts that electronic data into other electronic data that is stored in registers and/or memory can become. As non-limiting examples, "processor" can be a CPU or a GPU. A "computing platform" may include one or more processors. As used herein, "software" processes may include, for example, software and/or hardware entities that perform work over time, such as tasks, threads, and intelligent handlers. Also, each process can refer to multiple processes to execute commands in sequence or in parallel, continuously or intermittently. The terms "system" and "method" are used interchangeably herein in that system can embody one or more methods and methods can be considered as a system.

This document may refer to the acquisition, acquisition, receipt, or input of analog or digital data into a subsystem, computer system, or computer-implemented machine. The process of obtaining, capturing, receiving, or inputting analog and digital data can be performed in a variety of ways, such as receiving data as a parameter of a function call or an application programming interface call. In some implementations, the process of obtaining, capturing, receiving, or inputting analog or digital data may be accomplished by transmitting data over a serial or parallel interface. In another implementation, the process of obtaining, gathering, receiving, or inputting analog or digital data may be performed by the transmission of data over a computer network from the providing entity to the receiving entity. It may also refer to the provision, output, transmission, broadcast, or presentation of analog or digital data. In various examples, the process of providing, outputting, transferring, sending, or representing analog or digital data may be accomplished by transferring data as an input or output parameter of a function call, as a parameter of an application programming interface, or as an interprocess communication mechanism.

Although the above discussion provides example implementations of the described techniques, other architectures may be used to implement the described functionality and are intended to be within the scope of this disclosure. In addition, although specific descriptions of responsibilities have been defined above for purposes of discussion, various roles and responsibilities may be distributed and divided in different ways depending on the circumstances.

Claims (27)

Computer-implemented method, comprising: determining (1406), from an image of a hand (102, 106, 110, 114, 802-817), a first pose (702) of the hand; translating (1408) the first pose (702) of the hand (102, 106, 110, 114, 802-817) into a corresponding second pose of a robot hand (104, 108, 112, 116, 204, 704, 818-833), determining (1410) a set of motions that converts the robot hand (104, 108, 112, 116, 204, 704, 818-833) from a first pose of the robot hand (104, 108, 112, 116, 204, 704, 818- 833) switched to the second pose; causing (1412) the robotic hand (104, 108, 112, 116, 204, 704, 818-833) to perform the set of motions; and kinematic adjustment (314) of the robotic hand (104, 108, 112, 116, 204, 704, 818-833) by minimizing a cost function which is based at least on vectors derived from primary fingers (406, 408, 410) of the hand (102, 106, 110, 114, 802-817) pointing to the thumb (412) of the hand (102, 106, 110, 114, 802-817), and vectors pointing between the primary fingers (406, 408, 410) of the hand (102, 106, 110, 114, 802-817). Computer-implemented method claim 1 , further comprising: obtaining (1402) the image from a depth camera; determining (1404) a point cloud of the hand (102, 106, 110, 114, 802-817) from the image; and determining (1406) the first pose (702) of the hand (102, 106, 110, 114, 802-817) from the point cloud. Computer-implemented method claim 2 , wherein: the depth camera is an RGB depth camera (302), a radar imager, a medical imaging system, or a LIDAR system. A computer-implemented method according to any one of the preceding claims, wherein: the first pose (702) of the hand (102, 106, 110, 114, 802-817) includes information identifying the position of each of five fingers (404, 406, 408, 410, 412) of the hand (102, 106, 110 , 114, 802-817); and the first pose (702) of the hand (102, 106, 110, 114, 802-817) includes information identifying one or more positions of joints of the hand (102, 106, 110, 114, 802-817). A computer-implemented method according to any one of the preceding claims, wherein: translating the first pose (702) to the second pose by performing at least one kinematic realignment (314) of joint angles; and the joint angles are determined based at least in part on the first pose (702). A computer-implemented method according to any one of the preceding claims, wherein the set of motions is determined using Riemannian motion rules. A computer-implemented method as claimed in any preceding claim, wherein the robotic hand (104, 108, 112, 116, 204, 704, 818-833) is a movable robotic hand, a robotic gripper or a probe. A computer-implemented method according to any one of the preceding claims, wherein: the robotic hand (104, 108, 112, 116, 204, 704, 818-833) includes one or more touch sensors (216, 218, 220, 222) that provide tactile sensory information; and the information from the tactile sensor(s) (216, 218, 220, 222) comprises a 2-dimensional array of force values for a finger of the robot hand (104, 108, 112, 116, 204, 704, 818-833). System comprising: one or more processors (2102); and a computer-readable memory (2104) storing executable instructions that, when executed by one or more processors (2102), cause the system (2100) to: determining (1406), from an image of a limb (102, 106, 110, 114, 802-817) performing a task, a first pose (702) of the limb (102, 106, 110, 114, 802-817) ; Determining (1408) a second pose for a robot gripper (104, 108, 112, 116, 204, 704, 818-833) based at least in part on the first pose (702) of the limb (102, 106, 110, 114, 802 -817) based; determining (1410) a set of motions that will move the robot gripper (104, 108, 112, 116, 204, 704, 818-833) from a first pose of the robot gripper (104, 108, 112, 116, 204, 704, 818- 833) switched to the second pose; performing (1412) the set of movements to position the robot gripper (104, 108, 112, 116, 204, 704, 818-833) in the second pose to position the robot gripper (104, 108, 112, 116, 204, 704, 818-833) to perform the task; and kinematically adjusting (314) the robot gripper (104, 108, 112, 116, 204, 704, 818-833) by minimizing a cost function based at least on vectors derived from primary fingers (406, 408, 410) of the limb (102, 106, 110, 114, 802-817) pointing to the thumb (412) of the limb (102, 106, 110, 114, 802-817), and vectors pointing between the primary fingers (406, 408, 410) of the limbs (102, 106, 110, 114, 802-817). system after claim 9 wherein the executable instructions further cause the system to: generate (1404) a point cloud from the image of the limb (102, 106, 110, 114, 802-817); and determining (1406) the first pose (702) from the point cloud. system after claim 9 or 10 , where the set of motions is performed in accordance with Riemannian motion rules. system according to one of the claims 9 until 11 , wherein the limb (102, 106, 110, 114, 802-817) is a human hand or foot. system according to one of the claims 9 until 12 wherein the executable instructions cause the system (2100) to further apply a scaling factor to the first pose (702) of the limb to scale the second pose of the robot gripper (104, 108, 112, 116, 204, 704, 818-833). determine. system according to one of the claims 9 until 13 , wherein the first pose (702) of the limb specifies a segmentation of the hand (102, 106, 110, 114, 802-817) and a set of joint angles. system after Claim 14 , wherein the second pose specifies a set of target joint angles and a position of the robot gripper (104, 108, 112, 116, 204, 704, 818-833). system after claim 15 , wherein the second pose is determined using a subset of the first pose (702). A computer-readable medium storing instructions that, when executed by one or more processors (2102) of a computer system (2100), cause the computer system (2100) to: determining (1406), from an image of a human performing a task, a first pose (702) of a limb (102, 106, 110, 114, 802-817) of the human; translating (1408) the first pose (702) of the limb (102, 106, 110, 114, 802-817) into a corresponding second pose of a robot hand (104, 108, 112, 116, 204, 704, 818-833); determining (1410) a set of motions that the robotic hand (104, 108, 112, 116, 204, 704, 818-833) requires from a first pose of the robotic hand (104, 108, 112, 116, 204, 704, 818- 833) switch to the second pose; causing (1412) the robotic hand (104, 108, 112, 116, 204, 704, 818-833) to perform the set of motions; and kinematically re-enacting (314) the robotic hand (104, 108, 112, 116, 204, 704, 818-833) by minimizing a cost function based at least on vectors derived from primary fingers (406, 408, 410) of the limb (102, 106, 110, 114, 802-817) pointing to the thumb (412) of the limb (102, 106, 110, 114, 802-817), and vectors pointing between the primary fingers (406, 408, 410) of the limbs (102, 106, 110, 114, 802-817). computer-readable medium Claim 17 , wherein the instructions further cause the computer system (2100) to: obtain (1402) the image from a depth camera; determining (1404) a point cloud of the limb (102, 106, 110, 114, 802-817) from the image; and determining (1406) the first pose (702) of the limb (102, 106, 110, 114, 802-817) from the point cloud. computer-readable medium Claim 18 , wherein the depth camera is an RGB depth camera (302), a radar imager, a medical imaging system, or a LIDAR system. Computer-readable medium according to any of claims 17 until 19 wherein: the first pose (702) of the limb (102, 106, 110, 114, 802-817) includes information identifying a position of each of five fingers (404, 406, 408, 410, 412) of a hand; and the first pose (702) of the limb (102, 106, 110, 114, 802-817) includes information identifying one or more positions of joints of the hand. Computer-readable medium according to any of claims 17 until 20 wherein: the first pose (702) of the limb (102, 106, 110, 114, 802-817) includes information identifying a position of each of five fingers (404, 406, 408, 410, 412) of a hand; and the first pose (702) of the limb includes information identifying one or more positions of joints of the hand. Computer-readable medium according to any of claims 17 until 21 wherein: the first pose (702) is translated into the second pose by performing at least one kinematic realignment (314) of joint angles; and the joint angles are determined based at least in part on the first pose (702). Computer-readable medium according to any of claims 17 until 22 wherein: the robotic hand (104, 108, 112, 116, 204, 704, 818-833) includes one or more touch sensors (216, 218, 220, 222) that provide tactile sensory information; and the tactile sensor(s) (216, 218, 220, 222) information comprises a 2-dimensional array of force values for a finger of the robotic hand (104, 108, 112, 116, 204, 704, 818-833). Computer-readable medium according to any of claims 17 until 23 , wherein the robot hand (104, 108, 112, 116, 204, 704, 818-833) is a movable robot hand, a robot gripper or a probe. Computer-readable medium according to any of claims 17 until 24 wherein the first pose (702) is translated into the second pose by performing at least one kinematic adjustment (314) of joint angles; and the joint angles are determined based at least in part on the first pose (702). Computer-readable medium according to any of claims 17 until 25 , where the first pose of the robot hand (104, 108, 112, 116, 204, 704, 818-833) is the current pose of the robot hand (104, 108, 112, 116, 204, 704, 818-833). A robot comprising: an arm comprising one or more movable elements connected via one or more servomotors; a robotic link connected to the arm; one or more processors (2102); and the computer-readable medium of any one of claims 17 until 26 , which is connected to one or more processors (2102).

Images