DE102020127508B4 - Pose tracking of objects in hand - Google Patents

Abstract

A computer-implemented method comprising:obtaining (1302) touch sensor information from a robotic member (104, 108, 112, 116, 204, 504, 618-633) manipulating an object in the real world;generating (1304) a plurality of simulations of the robotic member (104, 108, 112, 116, 204, 504, 618-633) manipulating the object, each simulation having different poses for the object; determining (1308) a plurality of costs, each cost of the plurality of corresponds to a cost of a respective simulation of the plurality of simulations and each cost of the plurality of costs is based at least in part on differences between the touch-sensor information and simulated touch-sensor information generated by the respective simulation of the plurality of simulations;identifying (1310) a single simulation of the plurality of simulations based at least in part on the cost;determining (1312) a pose of the object in the real world based at least in part on a pose of the object in the identified individual simulation; and providing the pose of the object to a robotic control system (702) that controls a robot to perform a task based at least in part on the pose of the object.

Description

Technical part

At least one embodiment relates to training and simulating robots to perform and facilitate tasks. For example, at least one embodiment relates to training and simulating robots that use artificial intelligence according to various emerging techniques described herein.

background

Training and simulating robots to perform tasks accurately can consume significant memory, time, or computational resources. Training a robot control system to track the pose of an object held and manipulated by a robotic hand is a challenge for vision-based systems for tracking an object's pose, since the object is heavily obscured while the robotic hand is holding it. Such occlusion reduces the amount of data that can be used in determining the movements a robot should perform, creating the risk that the task at hand will be performed incorrectly and/or inefficiently, potentially damaging the object or other objects in the environment will be damaged. Such tracking is particularly complex because objects sometimes slip or otherwise move during the process, resulting in changes in the object's orientation that go undetected and therefore not accounted for. The amount of memory, time, or computational resources used to accurately train and simulate robots can be improved.

With this in mind, it is an object of the present invention to track the pose of an object being held and manipulated by a robotic hand. This object is solved by the subject matter of the computer-implemented method of claim 1, the system of claim 9, and the computer-readable medium of claim 17.

pamphlet DE 10 2020 124 285 A1 discloses a vision-based teleoperation of a mobile robotic system. A human operator controls a robotic arm and robotic gripper by simulating a set of desired motions with the human hand. In at least one example, one or more images of the operator's hand are captured and analyzed to determine a set of hand poses. In at least one example, the set of hand poses is translated into a corresponding set of robot gripper poses. In at least one example, a set of motions that perform the set of robot gripper poses is determined and the robot is instructed to perform the set of motions.

pamphlet DE 11 2019 001 507 T5 discloses techniques for controlling a robotic picking arm using estimated seal quality metrics. A plurality of possible contact points for holding an item using a suction device of the robotic picking arm based on captured images of the item and an n-dimensional surface model of the item are determined. An expected seal quality metric for a first of the possible contact points is determined by processing the n-dimensional surface model of the article and the physical properties of the robotic picking arm's suction device. Based on the expected seal quality metric, examples may determine whether to pick the item from the storage bin by holding the item at the first possible contact point using the suction device of the robotic picking arm.

pamphlet DE 10 2016 009 030 B4 discloses a machine learning apparatus that learns an operation of a robot for picking up, through a hand-held unit, any one of a plurality of workpieces laid down in an arbitrary manner including a loosely heaped state, and comprising: a state variable observer, that observes a state variable that represents a state of the robot and that includes data output from a three-dimensional measuring device that retrieves a three-dimensional image for each workpiece, a work result retrieval unit that receives a result of a pick-up operation of the robot to pick up the workpiece by the hand unit and a learning unit which, in connection with the state variable of the robot and the result of the pickup operation, when receiving the output from the state variable observing unit and the output from the operation result retrieving unit, a Learns the manipulated variable, which includes command data for instructing the robot to carry out the picking operation of the workpiece.

pamphlet DE 10 2013 113 459 B4 discloses an intuitive gripper control for a multi-axis robotic gripper. The gripper, which has a sensory matrix containing a plurality of sensors, performs selected gripping poses with respect to a component in the appropriate method to thereby grip the component in response to a grip command signal from the controller. The controller includes a touch-sensitive screen or other interactive graphical user interface (GUI) that generates a trajectory signal in response to input from a user. Sensory maps provide calibrated thresholds for the chosen grasping pose for each sensor included in the sensory matrix. The controller transmits the grip command signal to the gripper in response to receiving the trajectory signal from the GUI. The GUI may display a trajectory wheel with icons including a hub corresponding to a neutral pose of the robot gripper and icons corresponding to gripping poses arranged around a perimeter of the trajectory wheel.

pamphlet DE 60 2005 006 126 T2 discloses a method and apparatus for off-line programming of multiple interacting robots. For example, a system for offline programming of multiple interacting robots includes a computer for offline programming and verification of program code for multiple interacting robots and a robot controller connected to the computer to receive a download of at least one of the program codes for execution. Multiple interacting robots can be controlled by the robot controller.

pamphlet EP 3 771 522 A1 discloses a method and manipulation system for manipulating an object by a robot with vector fields.

pamphlet US 9,321,176 B1 discloses systems and methods for designing a robotic grip based on a demonstrated grip. In one example, planning a robotic grip on an object involves determining a grip type that would be used by a human to grasp the object, determining a position and orientation of the human's thumb relative to the object, and planning the robotic grip based on the specific grip type and thumb position and orientation.

character list

• 1 Figure 12 shows examples of teleoperations for various tasks, according to at least one embodiment;
• 2 12 shows an example of a robot with touch sensors, according to at least one embodiment;
• 3 Figure 12 shows an example of a system that tracks objects in real time, in accordance with at least one embodiment;
• 4 Figure 12 shows an example of estimating a hand pose, according to at least one embodiment;
• 5 12 shows an example of a pose of a human hand and a robotic gripper performing a corresponding pose, according to at least one embodiment;
• 6 12 shows an example of poses of a human hand and corresponding poses of a robot gripper, according to at least one embodiment;
• 7 Figure 12 shows an example of a framework (programming framework) for tracking object poses in the hand, according to at least one embodiment;
• 8th 12 shows an example of a comparison of optimizers, according to at least one embodiment;
• 9 Figure 12 shows an example of an algorithm used by a system, in accordance with at least one embodiment;
• 10 shows a first example of results of ablation studies, according to at least one embodiment;
• 11 12 shows a second example of ablation study results, according to at least one embodiment;
• 12 Figure 12 shows an example of real-world test results, in accordance with at least one embodiment;
• 13 Figure 12 shows an example of a method that, as a result of being executed by a computer system, determines the pose of an object being manipulated by a robotic hand equipped with touch sensors;
• 14A shows an inference and/or training logic according to at least one embodiment;
• 14B shows an inference and/or training logic according to at least one embodiment;
• 15 shows a training and the use of a neural network, according to at least one embodiment;
• 16 Figure 12 shows an example of a data center system, according to at least one embodiment;
• 17A shows an example of an autonomous vehicle, according to at least one embodiment;
• 17B shows an example of camera positions and fields of view for the autonomous vehicle 17A , according to at least one embodiment;
• 17C FIG. 12 is a block diagram showing an example system architecture for the autonomous vehicle of FIG 17A shows, according to at least one embodiment;
• 17D is a diagram showing a system for communication between cloud-based servers and the autonomous vehicle of the 17A shows, according to at least one embodiment;
• 18 Figure 12 is a block diagram showing a computer system according to at least one embodiment;
• 19 Figure 12 is a block diagram showing a computer system according to at least one embodiment;
• 20 Figure 12 shows a computer system, according to at least one embodiment;
• 21 Figure 12 shows a computer system, according to at least one embodiment;
• 22A Figure 12 shows a computer system, according to at least one embodiment;
• 22B Figure 12 shows a computer system, according to at least one embodiment;
• 22C Figure 12 shows a computer system, according to at least one embodiment;
• 22D Figure 12 shows a computer system, according to at least one embodiment;
• 22E and 22F show a common programming model, according to at least one embodiment;
• 23 Figure 12 shows example integrated circuits and associated graphics processors, according to at least one embodiment;
• 24A and 24B 12 show example integrated circuits and associated graphics processors, according to at least one embodiment;
• 25A and 25B Figure 12 shows further example graphics processor logic, in accordance with at least one embodiment;
• 26 Figure 12 shows a computer system, according to at least one embodiment;
• 27A Figure 12 shows a parallel processor, according to at least one embodiment;
• 27B Figure 12 shows a partition unit, according to at least one embodiment;
• 27C 12 shows a processing cluster, according to at least one embodiment;
• 27D 12 shows a graphics multiprocessor, according to at least one embodiment;
• 28 shows a multi-graphics processing unit
• GPU), according to at least one embodiment;
• 29 12 shows a graphics processor, according to at least one embodiment;
• 30 12 is a block diagram showing a processor microarchitecture for a processor, according to at least one embodiment;
• 31 12 shows a processor for deep learning applications, according to at least one embodiment;
• 32 12 is a block diagram showing an example neuromorphic processor, according to at least one embodiment;
• 33 12 shows at least parts of a graphics processor, according to one or more embodiments;
• 34 12 shows at least parts of a graphics processor, according to one or more embodiments;
• 35 12 shows at least parts of a graphics processor, according to one or more embodiments;
• 36 3610 is a block diagram of a graphics processing engine 3610 of a graphics processor, according to at least one embodiment;
• 37 12 shows a block diagram of at least parts of a graphics processor core, according to at least one embodiment;
• 38A and 38B 12 show thread execution logic 3800 comprising an array of processing elements of a graphics processor core, according to at least one embodiment;
• 39 Figure 12 shows a parallel processing unit (“PPU”), according to at least one embodiment;
• 40 12 shows a general processing cluster ("GPC"), according to at least one embodiment;
• 41 Figure 12 shows a memory partition unit of a parallel processing unit (“PPU”), according to at least one embodiment; and
• 42 12 shows a streaming multiprocessor, in accordance with at least one embodiment.

Detailed description

The present document describes a system and method for estimating the pose of an object while the object is being manipulated by a robotic hand, claw or manipulator. In various examples, when an object is held by a robot, image-based pose estimation systems can produce inaccurate pose estimates due to object occlusions. In at least one embodiment, a robotic hand is equipped with touch (tactile) sensors, and as an object is manipulated by the robotic hand, the sensor signals generated by the touch sensors are used to better estimate the object's pose. In some situations, dynamic effects such as slipping can occur during active manipulation, increasing the difficulty of estimating the pose. In at least one embodiment, a physical model of the object is used to enhance modeling of robot-object interactions.

In at least one embodiment, a graphics processing unit ("GPU") accelerated physics engine with derivative-free, sample-based optimizers tracks the poses of an object-in-hand with feedback from contacts during manipulation. In at least one embodiment, physics simulation is used as a forward model for robot-object interactions, and the techniques described herein optimize together for the state and parameters of the simulations so that the simulations can more accurately estimate the real world.

In at least one embodiment, the techniques described herein explicitly model the dynamics of robot-object interactions for object pose tracking and optimize for the simulation parameters during pose tracking. In various instances, these features allow the system to track object pose under complex dynamic behaviors, such as translational and torsional slippage due to inertial and external forces, and breakup and recovery development of the contact. Additionally, by using the GPU-accelerated physics engine, these techniques can often be applied in real-time (30 Hz) using the GPU.

Several of the working examples demonstrate promising applications of GPU-accelerated physics simulation for robotics. For example, in some embodiments, the speed of the physics engine allows for expensive, high-contact simulations and sample-based optimization techniques that rely on data from many simultaneous simulations, all in real time on the same machine, which is often difficult with CPU-based simulations. Various embodiments can be used as an aid in estimating an object's pose in the hand and reducing the limitation many researchers face in skilled (mobile, manual) manipulation, i.e. that the object is typically placed so that it can only be part is visible from a camera, which limits the range of manipulation tasks that can be examined.

Teleoperation can endow inanimate robotic systems with highly developed logical skills, intuition and creativity. However, teleoperation solutions for robots with a high degree of actuation (“DoA”) and multiple fingers can be complex. In at least one embodiment, a system is developed that allows full control of a high DoA robotic system by merely observing the bare human hand. The system can enable operators to solve a variety of complex manipulation tasks that go beyond simple pick-and-place (pick and place) operations. In various embodiments, the system can be implemented by one or more systems, as in 13-40 described/illustrated.

Tracking an object's pose while being held and manipulated by a robotic hand can be difficult for vision-based methods due to significant occlusions. The techniques described here use GPU-accelerated parallel robotic simulations and derivative-free, sample-based optimizers to track the poses of in-hand objects with feedback from contacts during manipulation. In some examples, a physics simulation is used as a forward model for robot-object interactions, and the algorithm collectively optimizes the state and parameters of the simulations to better match those of the real world. At least one embodiment runs in real-time (30 Hz) on the GPU and achieves an average point cloud distance error of 6 mm in simulation experiments and 13 mm in the real world.

In at least one embodiment, performing dexterous manipulation policies benefits from a robust estimation of the pose of the hand-held object. However, in many implementations, tracking the hand-held object pose is still a challenge due to significant occlusions. In such implementations, work requiring poses of hand-held objects can be limited to experiments where the object is mostly visible or that use multiple cameras, or where the hand-to-object transformation is fixed or known. In some examples, the problem of visual occlusions is reduced by looking more closely at an estimate of object poses from contacts or tactile feedback, often using particle filters and knowledge of object geometry and contact positions. In at least one embodiment, these techniques can be applied to a static gripping situation where an object is stationary and gripped. In at least one embodiment, these techniques are extended to object pose tracking during in-hand manipulation, which requires modeling of complex object-hand contact dynamics.

In order to enable in-hand tracking of objects during manipulation by a robot, in at least one embodiment a GPU-accelerated high-precision physics simulator is combined as a forward dynamics model with a sample-based optimization framework for tracking object poses with contact feedback. as in 7 shown. In at least one embodiment, a concurrent set of simulations is initialized with the initial states of a real robot and the real object's initial pose, which may be obtained by a vision-based pose registration algorithm that assumes the object is not initially in occlusion. In at least one embodiment, the initial poses of the simulated objects are slightly shifted, reflecting the uncertainty of the vision-based pose registration algorithm. In at least one embodiment, the GPU-accelerated physics simulator runs many simultaneous simulations in real-time on a single GPU. While a particular policy, in at least one embodiment, directs the real robot to approach, grab, and manipulate the object, the system executes the same robot control commands on the simu alienated robots. In at least one embodiment, observations of the real robot and the simulated robots are collected, including, for example, size and direction of contacts on the robot hand's contact sensors. In at least one embodiment, a sample-based optimization algorithm periodically updates the states and parameters of the simulations according to a cost function that captures how well each simulation's observations match those of the real world. Additionally, in some embodiments, the algorithm updates simulation parameters such as mass and friction to further improve the dynamics models of the real-world simulations. In at least one embodiment, the object pose estimated at any given time is the pose of the robot-object system.

In at least one embodiment, a total of 24 trajectories of hand manipulation with three different objects in simulation and in the real world were collected to evaluate the proposed algorithm. In at least one embodiment, a Kuka IIWA7 arm was used with the 4-finger Wonik Robotics Allegro hand as the end effector, each finger equipped with a SynTouch BioTac touch sensor. In at least one embodiment, the object manipulation trajectories are human demonstrations collected via a teleoperation system that tracks a hand. In at least one embodiment, since object poses are available as ground truth data in the simulation, detailed ablation studies are performed in simulation experiments to investigate the properties of the proposed algorithm. In at least one embodiment, for real-world experiments, a vision-based algorithm is used to obtain the object pose in the first and last frames of the collected trajectories where the object is not in occlusion. In at least one embodiment, the pose in the first frame is used to initialize the simulations, and the pose in the last frame is used to assess the accuracy of the proposed contact-based algorithm.

In various examples, a position of an object in the hand is identified using only vision, by first segmenting the robot or human hand in an image before estimating the pose. However, at larger occlusions, the performance of vision-only approaches may degrade. Some embodiments use tactile feedback to help estimate object pose. Tactile perception can object properties, such. material and pose, and also provide feedback during object manipulation.

In at least one embodiment, experiments with dynamics models and particle filtering techniques show that adding noise to the applied forces, rather than the underlying dynamics, provides more accurate tracking results. In at least one embodiment, tactile feedback is combined with a vision-based object tracker to track trajectories of objects during planar sliding tasks, and in another embodiment, Incremental Smoothing and Mapping (iSAM) is applied to provide global visual Combine pose estimates with local contact pose measurements.

In at least one embodiment, a robotic hand grasps an object and locates the object's pose without moving. Some examples use point contact locations and some examples use a full tactile map to extract local geometry information around the contacts.

In at least one embodiment, contact position feedback is used for the pose estimation, and some implementations use some variant of Bayesian or particle filters. In some embodiments, the filtering is performed collectively across visual features, hand-joint positions, force-torque measurements, and binary contact modes. Some techniques can also be applied to estimating the pose when the object is not held by the robot hand using force sensors.

In at least one embodiment, tactile maps are used for pose estimation, with some examples using large, low-resolution tactile arrays to detect contacts in a grid, while other examples using high-resolution tactile sensors attached to the robot's fingertips. In at least one embodiment, the system looks for similar local areas on an object's surface to locate the object relative to contact position, and other systems fuse GelSight data with a point cloud perceived by a depth sensor before performing pose estimation.

In some embodiments, object pose tracking is implemented in the hand during object manipulation, which is more challenging than when the object is static. In at least one embodiment, an algorithm that combines contact locations with Dense Articulated Real-time Tracking ("DART") is used. In at least one embodiment, the algorithm fuses contact locations with visual color features, joint locations, and force-torque measurements. In at least one embodiment, the algorithm is sensitive to the initialization of object poses, particularly when the object appears small in the depth image. In at least one embodiment, the techniques described herein do not require access to robust visual features during manipulation, but instead use a physics simulator to model both the kinematics and dynamics of the robot-object system.

In various examples, robotic teleoperation can find applications in search and rescue, space travel, medicine, and applied machine learning. The motivation for a teleoperation capability may be to enable a robotic system to solve complex tasks by harnessing human cognition, creativity, and responsiveness via a human-machine interface (“HMI”). In one embodiment, this system is a gloveless solution to control a multi-fingered, high-motion robotic system to perform a variety of gripping and manipulation tasks. In some examples, depth cameras and various graphics processing units (“GPU”) can be used along with deep learning and optimization to create a nifty teleoperation system with a minimal footprint. In some examples, a variety of physical tasks can be performed with visual feedback alone. Therefore, this system can use the human ability to plan, move and predict the consequences of physical actions through vision alone, which can be a sufficient condition for solving a variety of tasks.

In various embodiments, the developed system enables such a skilful robot manipulation through depth observations of the bare human hand with several cameras. In some examples, the system may be a fully vision-based, gloveless teleoperation system that dexterously moves a highly maneuverable robotic hand-arm system through direct imitation. The system can also perform a number of tasks that require fine manipulation and dexterity in particular (e.g., pulling paper money out of a wallet and picking up two dice with four fingers at the same time, as in 1 shown).

1 12 shows an example of a teleoperation for various tasks, according to at least one embodiment. In one example, a robotic gripper 104 grips a cylinder using a gripping pose based on a pose of a human hand 102. In another example, a robotic gripper 108 grips a cube using a gripping pose based on a human hand 106. In another example, a robotic gripper 112 grips a cup using a gripping pose based on a human hand 110. In another example, a robot gripper 116 grips a wallet using a gripping pose based on a human hand 104.

The teleoperation setup may include a robotic system and an adjacent arena for human operators, as in 2 shown. 2 shows an example of a robot with tactile sensors according to at least one embodiment. In at least one embodiment, a robot 202 has a robot gripper 204 that is used to grip objects using tactile sensors. In at least one embodiment, a set of cameras 206, 208, 210, and 212 are used to view the robot's 202 workspace. In at least one embodiment, gripper 204 includes a set of tactile sensors 216, 218, 220, and 222 that provide sensory information to a control computer system. In at least one embodiment, the tactile sensors may be covered with a friction material to enhance the robot's ability to grasp an object.

In some embodiments, as in 2 As shown, the robotic system may be a KUKA LBR iiwa7 R800 series arm with a Wonik Robotics Allegro hand fitted with four SynTouch BioTac tactile sensors on the fingertips and a 3M TB641 grip tape applied to the inner surfaces of the phalanges and palm , where the rubbery surfaces of both the BioTac sensors and 3M tape can improve hand friction, while the BioTacs themselves can generate 23 signals that can later be used to learn sensorimotor control using demonstrations . The human sphere of action can be a black-clad table surrounded by four calibrated and time-synchronized cameras, such as Intel RealSense RGB D cameras, which can be spatially arranged to cover a working area of 80 cm × 55 cm × 38 cm. In some examples, the cameras can be right next to the robot to improve line of sight and visual proximity since teleoperation relies entirely on human vision and spatial reasoning. It should be noted that 2 is intended as an embodiment, and in various embodiments the system may include any robotic system using any robotic component (e.g., various types of robotic arms, hands, tactile sensors, handles, other sensors, cameras, and/or variations thereof) in any suitable environment can.

To create a natural-feeling teleoperation system, an imitation-type paradigm can be used. The mere movement of a human hand - pose and finger configuration - can be constantly observed and measured by a visual perception module. The movement of the human hand can then be forwarded to the robot system in such a way that the copied movement is self-explanatory. This approach can allow a human operator to curve and arrange their fingers, form grips, reorient their palms, and translate, with the robotic system following in a similar fashion. In at least one embodiment, the system relies heavily on Dense Articulated Real Time Tracking ("DART"), which can form the backbone of tracking the pose and joint angles of the human hand. The full system architecture and component connections are in 3 shown in one embodiment.

3 12 shows an example of a system that tracks objects in real time, according to at least one embodiment. In at least one embodiment, the system operates using three threads, which are independent processes running on one or more processors of a computer system. In at least one embodiment, one or more images of a hand are obtained from RGB depth ("RGB-D") cameras 302 . The images are processed by a Pointnet: Stage 1 304, a Pointnet: Stage 2 306, and a Jointnet 308 to generate a hand pose for the hand in the images. In at least one embodiment, a movable hand model 310 and the hand pose are processed with DART 312 and kinematic realignment (tracking, tracking, retargeting, retargeting, kinematic retargeting) 314 to generate a corresponding hand pose for a robotic gripper. In at least one embodiment, a control thread applies the Riemannian motion guidelines 318 to the gripper's hand pose and the resulting information is used to control the robot 320 .

In at least one embodiment, DART is used to continuously track the pose and joint angle of a human hand. In at least one embodiment, DART uses a moving model of the hand that is registered from an input point cloud. A human hand model can be obtained and converted into a single mesh model. Using CAD (Computer Aided Design) software, the fingers of the mesh model can be partitioned into their respective proximal, medial and distal limbs and restored as separate meshes along with an associated Extensible Markup Language (XML) file describing their kinematic arrangement be exported. In total, the human hand model may have 20 pivot joints: four joints per finger with one abduction joint and three flexion joints.

In at least one embodiment, DART is a model-based tracker that performs non-linear optimization and initialization (e.g., from the previous frame or a first guess). In some examples, if this initialization is not within the convergence basin, the tracker may not converge to the correct solution. In various embodiments, when tracking the human hand model with point cloud data, the hand model can often snap to incorrect local minima, resulting in tracking errors every few minutes. Therefore, in order to be able to reliably track the human hand over long periods of time - which is needed for teleoperation - it may be desirable to have reliable presets for the hand pose of a human hand, a clean segmentation of the hand and a multi-camera studio to prevent the hand model from snapping to unexpected local minima. In various embodiments, one method for generating preset hand poses is to train a neural network on a large data set of human hand poses generated by camera images.

In at least one embodiment, data collection is initiated with DART and without handpose constraints, thereby training a first network to produce handpose constraints. As a result, DART and the most recently trained neural network can generate increasing amounts of data. In In at least one embodiment, the network is constantly updated with the latest datasets to produce better and better specifications for DART, which can ultimately expand the range in which DART can function without failure. In some examples, the hand pose neural network may be a PointNet-based architecture that works directly with fused point cloud data obtained by back-projecting depth images from extrinsically calibrated depth cameras into a single global reference frame with annotations provided by DART. Since the merged point cloud contains both the points on the table and the human body and arm in various exemplary embodiments, it may be essential in various cases to first locate the hand. Points can be removed from the table by fitting a plane and passing the remaining points containing the arm and human body to PointNet, which can both locate the hand and provide the hand pose. PointNet can be based on estimating hand pose using a voting-based regression scheme on the 3D positions of specified key points on the hand, a technique that may involve spatial softmax commonly used in 2D key point localization. In various embodiments, PointNet can be trained to predict 3D coordinates of 23 key points specified on the hand - four joint key points for each of the five fingers and three key points on the back of the hand for estimating hand pose. The loss function may be the Euclidean distance between the predicted points and the key points of ground truth. Additionally, an auxiliary segmentation loss may be included to obtain hand segmentation. For efficiency, any input point cloud can be uniformly subsampled down to a fixed size of 8192 × 3 before being fed into PointNet. In at least one embodiment, a reasonable estimate of hand pose and segmentation may be achieved, and high quality predictions may not yet be obtained for the 20 common keypoints on the fingers. In at least one embodiment, the uniform undersampling used in the input may indicate that the points on the fingers are not densely sampled, so refinement may be required in a second stage, where the points on the hand are subsampled from the original raw point cloud be resampled to account for the pose and segmentation of the first stage. In at least one embodiment, the second stage can be trained on the same loss functions, but can instead use only the hand sampled points to accurately predict the 23 key points. In at least one embodiment, random noise may be added to the second stage hand pose to provide robustness to any inaccuracies in the first stage hand pose. 4 Figure 12 illustrates the second level of refinement within the system, according to at least one embodiment. In at least one embodiment, both stages of PointNet can be trained on 100K point clouds collected over a batch of 30-45 minutes each for a total of 7-8 hours by running DART to provide annotation for key points, joint angles, and segmentation. In at least one embodiment, a third neural network may be trained that maps the key points predicted by PointNet to the corresponding joint angles to provide finger joint angle specifications. This neural network, which can be referred to as a JointNet, can be a two-layer fully connected network that takes inputs of size 23×3 and predicts a 20-dimensional vector of joint angles for fingers.

In at least one embodiment, the neural networks are trained on data collected from multiple human hands, thereby ensuring accurate pose matching for this system and allowing reasonable targets for DART. In some embodiments, hand tracking may work better for hands that are geometrically close to the human DART hand model.

wobei q_h, q_a die Winkel des Modells der menschlichen Hand bzw. der Allegro-Hand sein können, r_i ∈ R³ der Vektor, der vom Ursprung eines Koordinatensystems zu einem anderen zeigt, ausgedrückt im Ursprungskoordinatensystem sein kann (siehe 7). Darüber hinaus ist in mindestens einem Ausführungsbeispiel $d_i=\Vert r_i\left(q_h\right)\Vert \text{ und }{r}_i^{}\left(q_h\right)=\frac{r_i\left(q_h\right)}{\Vert r_i\left(q_h\right)\Vert }.$

Die Umschaltgewichtsfunktion s(d_i) kann wie folgt definiert sein: $$s\left(d_i\right)=\{\begin{array}{c}\mathrm{1,}{d}_i>\in \\ \mathrm{200,}{d}_i\le \in \wedge r_i\left(q_h\right)\in S_1\\ \mathrm{400,}{d}_i\le \in \wedge r_i\left(q_h\right)\in S_2\end{array}$$

wobei S₁ Vektoren sein können, die von einem Primärfinger (Zeige-, Mittel-, Ringfinger) ausgehen und auf den Daumen zeigen, und S₂ m Vektoren zwischen zwei Primärfingern sein können, wenn beide Primärfinger assoziierte Vektoren ∈ S₁ haben (z.B. beide Primärfinger werden mit dem Daumen projiziert). In mindestens einem Ausführungsbeispiel ist die Distanzfunktion ƒ(d_i) E R definiert als: $$ƒ\left(d_i\right)=\{\begin{array}{c}\mathrm{\beta }{d}_i,d_i>\in \\ {\mathrm{\eta }}_1,d_i\le \in \wedge r_i\left(q_h\right)\in S_1\\ {\mathrm{\eta }}_2,d_i\le \in \wedge r_i\left(q_h\right)\in S_2\end{array}$$

wobei β = 1.6 ein Skalierungsfaktor sein kann, η₁ = 1 × 10^-4 m ein Abstand zwischen einem primären Finger und dem Daumen sein kann, und η₂ = 3 × 10^-2 m kann ein Mindestabstand zwischen zwei primären Fingern sein kann, wenn beide primären Finger mit dem Daumen projiziert werden. In mindestens einem Ausführungsbeispiel stellen diese Projektionen sicher, dass die Kontakte zwischen den primären Fingern und dem Daumen nahe beieinander liegen, ohne dass es in einem Präzisionsgriff zu Kollisionen der primären Finger kommt. In mindestens einem Ausführungsbeispiel kann dies bei Ungenauigkeiten der visuellen Fingerverfolgung besonders nützlich sein. In einigen Beispielen können die Vektoren r_i nicht nur Abstand und Richtung von einem Aufgabenraum zum anderen erfassen, sondern ihr Ausdruck in lokalen Koordinaten kann weiter Informationen darüber enthalten, wie die Koordinatensysteme und damit die Fingerspitzen zueinander orientiert sind. In mindestens einem Ausführungsbeispiel können die Koordinatensysteme des Modells der menschlichen Hand daher äquivalente Koordinatensysteme auf dem Allegro-Modell mit Ähnlichkeit in Orientierung und Platzierung haben. Die in 5 gezeigten Vektoren können einen minimalen Satz bilden, der das gewünschte Verhalten beim Neuausrichten (engl. retargeting) erzeugt. In einigen Ausführungsbeispielen kann γ = 2,5 × 10^-3 ein Gewicht bei der Regularisierung der Allegro-Winkel auf Null sein (äquivalent zur vollständig geöffneten Hand). In mindestens einem Ausführungsbeispiel trägt dieser Term dazu bei, die Redundanz in der Lösung zu reduzieren und sicherzustellen, dass die Hand nie in ungewöhnliche Minima gerät, von denen sie sich nur schwer wieder erholen kann (z.B. wenn sich die Finger in die Handfläche einbetten). In mindestens einem Ausführungsbeispiel werden in 6 verschiedene Zuordnungen von der menschlichen Hand 602-617 zu einer Allegro-Roboterhand 618-633 gezeigt, wie sie durch kinematisches Neuausrichten erzeugt werden.In at least one embodiment, teleoperation of a robotic hand that is kinematically different than the human hand may require a module that can map the observed human wrists to the robotic wrists, which in some embodiments may be referred to as allegro wrists. 5 FIG. 5 shows an example of a pose 502 of a human hand and a robotic gripper 504 performing a corresponding pose, according to at least one embodiment. There can be many different approaches to kinematic realignment. For example, a module according to at least one embodiment may be used to match the palm to fingertip and medial joint positions and directionality of the proximal phalanx and distal thumb phalanx. In at least one embodiment, the optimized mapping may be mapped using human depth images such that a depth network can take a depth image and output joint angles. In at least one embodiment, motion tracking is also used. motion retargeting) is used. For example, a deep recurrent neural network can be trained unsupervised to track movements between skeletons. In at least one embodiment, the system utilizes the fingertip task space metric since distal areas may be of the highest priority in grasping and manipulation tasks as measured by their contact prevalence, level of innervation, and increased controllability for fine hand manipulation capability. In at least one embodiment, the joint axes and locations may differ between two hands, and therefore metrics that directly compare the joint angles between the two hands cannot be used. In at least one embodiment, both the distance and the direction between the fingertips are considered to detect and optimize the positioning of the fingertips. Specifically, in at least one embodiment, the cost function for the kinematic realignment can be chosen as follows: $$C\left(q_H,q_a\right)=\frac{1}{2}{\displaystyle {\sum }_{i=0}^{N}s\left({\mathrm{i.e}}_i\right)}{\Vert {\mathrm{right}}_i\left(q_a\right)-f\left({\mathrm{i.e}}_i\right){\mathrm{right}}_i^{}\left(q_H\right)\Vert }^2+g{\Vert q_a\Vert }^2$$

where q _h , q _a can be the angles of the model of the human hand and the Allegro hand, respectively, _ri ∈ R ³ can be the vector pointing from the origin of one coordinate system to another, expressed in the origin coordinate system (see 7 ). In addition, in at least one embodiment ${\mathrm{i.e}}_i=\Vert {\mathrm{right}}_i\left(q_H\right)\Vert \text{and}{\mathrm{right}}_i^{}\left(q_H\right)=\frac{{\mathrm{right}}_i\left(q_H\right)}{\Vert {\mathrm{right}}_i\left(q_H\right)\Vert }.$

The switching weight function s(d _i ) can be defined as follows: $$s\left({\mathrm{i.e}}_i\right)=\{\begin{array}{c}\mathrm{1,}{\mathrm{i.e}}_i>\in \\ \mathrm{200,}{\mathrm{i.e}}_i\le \in \wedge {\mathrm{right}}_i\left(q_H\right)\in {\mathrm{<figure-callout\; id="1"\; label="S"\; filenames="DE102020127508B4\_0025.png,DE102020127508B4\_0026.png"\; state="\{\{state\}\}">S</figure-callout>}}_1\\ \mathrm{400,}{\mathrm{i.e}}_i\le \in \wedge {\mathrm{right}}_i\left(q_H\right)\in {\mathrm{<figure-callout\; id="2"\; label="S"\; filenames="DE102020127508B4\_0026.png,DE102020127508B4\_0039.png"\; state="\{\{state\}\}">S</figure-callout>}}_2\end{array}$$

where S ₁ can be vectors emanating from a primary finger (index, middle, ring finger) and pointing to the thumb, and S ₂ can be m vectors between two primary fingers if both primary fingers have associated vectors ∈ S ₁ (e.g. both Primary fingers are projected with the thumb). In at least one embodiment, the distance function ƒ(d _i ) ER is defined as: $$ƒ\left({\mathrm{i.e}}_i\right)=\{\begin{array}{c}\mathrm{\beta }{\mathrm{i.e}}_i,{\mathrm{i.e}}_i>\in \\ {\mathrm{<figure-callout\; id="1"\; label="n"\; filenames="DE102020127508B4\_0025.png,DE102020127508B4\_0026.png"\; state="\{\{state\}\}">n</figure-callout>}}_1,{\mathrm{i.e}}_i\le \in \wedge {\mathrm{right}}_i\left(q_H\right)\in S_1\\ {\mathrm{<figure-callout\; id="2"\; label="n"\; filenames="DE102020127508B4\_0026.png,DE102020127508B4\_0039.png"\; state="\{\{state\}\}">n</figure-callout>}}_2,{\mathrm{i.e}}_i\le \in \wedge {\mathrm{right}}_i\left(q_H\right)\in {\mathrm{<figure-callout\; id="2"\; label="S"\; filenames="DE102020127508B4\_0026.png,DE102020127508B4\_0039.png"\; state="\{\{state\}\}">S</figure-callout>}}_2\end{array}$$

where β = 1.6 can be a scaling factor, η ₁ = 1 × 10 ^-4 m can be a distance between a primary finger and the thumb, and η ₂ = 3 × 10 ^-2 m can be a minimum distance between two primary fingers, when both primary fingers are projected with the thumb. In at least one embodiment, these projections ensure that the contacts between the primary fingers and the thumb are close together without collisions of the primary fingers in a precision grip. In at least one embodiment, this can be particularly useful for inaccuracies in visual finger tracking. In some examples, the vectors r _i can not only capture distance and direction from one task space to another, but their expression in local coordinates can further contain information about how the coordinate systems and thus the fingertips are oriented to each other. Thus, in at least one embodiment, the coordinate systems of the human hand model may have equivalent coordinate systems on the Allegro model with similarity in orientation and placement. In the 5 The vectors shown can form a minimal set that produces the desired retargeting behavior. In some embodiments, γ=2.5×10 ^-3 may be a weight in regularizing the allegro angles to zero (equivalent to the fully open hand). In at least one embodiment, this term helps reduce redundancy in the solution and ensures that the hand never goes into unusual minima from which it is difficult to recover (eg, when fingers become embedded in the palm). In at least one embodiment, in 6 various mappings from the human hand 602-617 to an Allegro robotic hand 618-633 are shown as produced by kinematic realignment.

In at least one embodiment, the above functionality is minimized in real-time using the Sequential Least Squares Quadratic Programming ("SLSQP") algorithm. In at least one embodiment, the routine is initiated with the Allegro joint angles set to zero, and any solution thereafter can be completed with the previous solution to be initiated. In at least one embodiment, the forward kinematic calculations are performed between the different coordinate systems of both the human hand model and the Allegro hand model. In at least one embodiment, a first-order low-pass filter is applied to the raw transfer joint angles to remove high-frequency noise associated with human hand tracking and to smooth discrete events, such as the projection algorithm, that induce step response changes in the realigned angles.

Riemannian motion guidelines ("RMPs"), in one embodiment, are real-time motion generation methods that compute acceleration fields from potential functional gradients and corresponding Riemannian metrics. RMPs can combine multi-priority Cartesian trajectory generation and collision avoidance behavior in a cohesive frame of reference. In at least one embodiment, they are used to control the Cartesian pose of the Allegro palm using the observed pose of a human hand while avoiding arm or palm collisions with the table or operator using collision planes. In at least one embodiment, with these objectives in mind, the RMPs generated arm-joint target trajectories that are sent at 200 Hz to the arm's torque impedance controller. In at least one embodiment, the kinematically adjusted Allegro angles are sent at 30 Hz to the arm's torque impedance controller. In at least one embodiment, a teleoperation instance is initialized by registering the studio cameras with the robot base coordinate system via an initial, static robot pose and the initial observation of the human hand. In at least one embodiment, the axes of the human hand model and the axes of the robotic end effector are approximately aligned such that the direction of motion between human hand motion and robotic motion is preserved.

Overall, the system can be used reliably to solve a variety of tasks with different levels of difficulty. In some examples, the ability to perform these tasks demonstrates that the system may have the dexterity, precision and power grips, multi-finger grip and non-grip manipulations, finger motions of the hand, and compound hand manipulations (e.g., two-finger grips with simultaneous manipulation with the remaining fingers).

In at least one embodiment, the system may enable a highly mobile hand-arm system to find a motorized solution to a variety of manipulation tasks by translating observed human hand and finger movements into robotic arm and finger movements. In at least one embodiment, multiple tasks, such as pulling paper money from a wallet and opening a cardboard box in a plastic container, can be so complex that developing a robotic solution by hand or directly applying learning methods is likely unsolvable. Solving these and other tasks by the executed robot can allow these solutions to be generated on-demand for many demonstrations. In addition, creating these solutions on the system itself can read, access, and store the various tactile signals in the robot's fingertips, various commanded and measured joint position and velocity signals through the hand and arm, various torque commands throughout the system, and any camera feeds associated with the system. In at least one embodiment, this rich source of data may be used in conjunction with task demonstrations to solve complex, multi-step, long-term tasks.

In one embodiment, a system is developed to track objects in the hand during manipulation by a robot. In various embodiments, the system can be implemented by one or more systems, as in 13 - 41 described/illustrated. As in 7 As shown, the system may include a high-fidelity GPU-accelerated physics simulator as a forward dynamics model with a sample-based optimization framework to track object poses with contact feedback. In at least one embodiment, a simultaneous set of simulations is initialized with the initial states of a real robot and the initial pose of a real object, which can be obtained from a vision-based pose registration algorithm, assuming that the object is not initially occluded. In at least one embodiment, the initial poses of the simulated objects are slightly perturbed, reflecting the uncertainty of the vision-based pose registration algorithm. In at least one embodiment, the GPU-accelerated physics simulator runs many simultaneous simulations in real-time on one single GPU off. In at least one embodiment, a predetermined policy is used that controls the real robot to approach, grasp, and hand manipulate the object, and the same robot control commands are executed on the simulated robots. In at least one embodiment, observations of the real robot and the simulated robots are collected that include terms such as the magnitude and direction of contacts on the robot hand's touch sensors. In at least one embodiment, a sample-based optimization algorithm is used that periodically updates the states and parameters of the simulations according to a cost function that captures how well each simulation's observations match those of the real world. Additionally, in at least one embodiment, the algorithm also updates simulation parameters such as mass and friction to further improve the real-world dynamics models of the simulations. At any given time, the object pose estimate may be the pose of the robot object system.

In various embodiments, to evaluate the proposed algorithm, a total of 24 trajectories for in-hand manipulation with three different objects in simulation and in the real world can be collected, although any number of trajectories can be collected. In at least one embodiment, a robotic arm such as the Kuka IIWA7 arm is used with the 4-finger Wonik Robotics Allegro hand as the end effector, each finger equipped with a SynTouch BioTac touch sensor. In at least one embodiment, the object manipulation trajectories are human demonstrations collected via a hand-tracking teleoperation system. In various embodiments, based on the object poses of the ground truths in the simulation, detailed ablation studies are performed in simulation experiments to evaluate the properties of the proposed algorithm. In at least one embodiment, a vision-based algorithm is used to obtain the object pose in the first and last frames of the collected trajectories where the object is not in occlusion. In at least one embodiment, the pose in the first frame is used to initialize the simulations, and the pose in the last frame is used to assess the accuracy of the proposed contact-based algorithm.

7 FIG. 12 shows an embodiment of a framework for tracking poses of objects in the hand. In at least one embodiment, robot controls 702 are sent to a GPU-accelerated physics simulator that runs 708 many robot simulations in parallel, each with different physics parameters and altered (perturbed) object poses. In at least one embodiment, costs based on observations, such as Feedback from contacts from the real world and from the simulations, is passed to a sample-based, derivative-free optimizer 704 that periodically updates the states and parameters of all simulations to better match the real world. In at least one embodiment, the pose of the simulation with the lowest cost is selected as the current object pose estimate 706 at any given time.

In one embodiment, a system tracks the pose of an object being hand-held by a robotic manipulator during object manipulation. In some embodiments, for the time that can be represented by t, an object pose can be defined as p _t E SE(3) and a physical dynamics model can be defined as s _t+1 = ƒ(s _t ,u _t ,θ). where s _t can be the real world state (position and velocities of rigid bodies and of joint angles in moving bodies), u _t ∈ ℝ ^M can be the controller of the robot (desired joint positions can be used as action space), and θ ∈ ℝ ^N can be the fixed parameters of the simulation (eg, mass and friction).

In various embodiments, for a simulation model f that exactly matches reality with perfect initializations of p ₀ , s ₀ and θ, pose estimation may only require returning the sequence of actions u _t applied to the robot in the simulation. However, since the forward models can be imperfect and the pose initializations can be noisy, feedback from observations can improve the pose estimation.

(die sich an den Fingerspitzen befinden können), die Kraftvektoren der abgetasteten Kontakte $c_t^{\left(l\right)}\in {ℝ}^3$

$$

der Einheitsvektor in Richtung des translatorischen Schlupfes auf der Berührungsfläche $d_t^{\left(l\right)}\in {ℝ}^2,$

und der binären Richtung des rotatorischen Schlupfes auf der Berührungsfläche $r_t^{\left(l\right)}\in \left\{\mathrm{0,1}\right\},$

wobei l den l-ten Berührungssensor indizieren kann. In mindestens einem Ausführungsbeispiel wird zum allgemeinen Bestimmen der Posenabschätzung in der Hand anhand der aktuellen und vergangenen Beobachtungen o_1:t, der Steuerung des Roboters u_1:t und der initialen Pose p₀ die wahrscheinlichste aktuelle Objektpose p_t ermittelt.In some embodiments, D may be defined as the number of joints of the robot and L as the number of its touch sensors. An observation vector o _t can be defined as the concatenation of the robot's joint position configuration q _t ∈ ℝ ^D , the position and rotation of the robot's touch sensors $P_t^{\left(l\right)}\in {ℝ}^3,R_t^{\left(l\right)}\in SO\left(3\right)$

(which may be at the fingertips), the force vectors of the sensed contacts $c_t^{\left(l\right)}\in {ℝ}^3$

$$

the unit vector in the direction of translational slip the touch surface ${\mathrm{i.e}}_t^{\left(l\right)}\in {ℝ}^2,$

and the binary direction of the rotational slip on the mating surface ${\mathrm{right}}_t^{\left(l\right)}\in \left\{0.1\right\},$

where l can indicate the lth touch sensor. In at least one embodiment, to generally determine the pose estimate in the hand, the most probable current object pose p _t is determined based on the current and past observations o _1:t , the controller of the robot u _1:t , and the initial pose p ₀ .

In various embodiments, a GPU-accelerated physics simulator can be used as a forward dynamics model to simulate many robotic object environments simultaneously to track object pose in hand, and derivative-free, sample-based optimizers can be used to share the state and parameters of the simulations tune to improve tracking performance. 9 Figure 12 shows one embodiment of an algorithm that can be used.

und $o_t^{\left(i\right)}$

definiert werden, und die Beobachtungen der Grundwahrheit können als $o_t^{\left(gt\right)}$

definiert werden. In verschiedenen Ausführungsbeispielen kann bei einer Kostenfunktion C die aktuell beste Posenabschätzung zum Zeitpunkt t die Pose der i-ten Simulation, $p_t^{\left(i*\right)},$

sein, wobei die i-te Simulation diejenige sein kann, die die niedrigsten Durchschnittskosten über ein vergangenes Zeitfenster T verursacht: $$\begin{array}{c}{C}_i=\frac{1}{T}{\displaystyle \sum _{\mathrm{\Delta }t=0}^{T-1}\text{C}\left(o_{\left(t-\mathrm{\Delta }t\right)}^{\left(i\right)},o_{\left(t-\mathrm{\Delta }t\right)}^{\left(gt\right)}\right)}\\ i*=\text{arg min }{C}_i\end{array}$$

First, an estimate of an initial object pose can be obtained via a vision-based object pose estimator. The pose estimator can be expected to provide a reliable initial pose estimate p ₀ when the robot is not in contact with the object and when the object is not occluded, e.g. B. before grasping. With the initial object pose estimate and the robot configuration, K simultaneous simulations can then be initialized, and at each time step the real robot actions u _t can be copied to all K simulations. In various examples, the object's pose can change as the hand makes contact, and this can be modeled by the simulator. In various embodiments, the object pose and the observation of the i th simulation can be used as $p_t^{\left(i\right)}$

and $O_t^{\left(i\right)}$

be defined, and the observations of ground truth can be defined as $O_t^{\left(Gt\right)}$

To be defined. In various exemplary embodiments, given a cost function C, the currently best pose estimate at time t can be the pose of the i-th simulation, $p_t^{\left(i*\right)},$

where the i-th simulation can be the one that causes the lowest average cost over a past time window T: $$\begin{array}{c}{C}_i=\frac{1}{T}{\displaystyle \sum _{\mathrm{\Delta }t=0}^{T-1}\text{C}\left(O_{\left(t-\mathrm{\Delta }t\right)}^{\left(i\right)},O_{\left(t-\mathrm{\Delta }t\right)}^{\left(Gt\right)}\right)}\\ i*=\text{bad min}{C}_i\end{array}$$

The costs can be used to periodically update the simulations and their parameters, which can allow for improved consistency with the real robot object system.

In some embodiments, a cost function may be used that correlates sufficiently with differences between object poses during in-hand manipulation such that a lower cost corresponds to a better estimate of the poses. The cost function can be represented by the following symbolic mathematical equation: $$\begin{array}{l}C\left(O_{\left(t-\Delta t\right)}^{\left(i\right)},O_{\left(t-\Delta t\right)}^{\left(Gt\right)}\right)\\ \begin{array}{l}=w_1\Vert q_t^{\left(i\right)}-q_t^{\left(Gt\right)}\Vert 2\hfill \\ +{\displaystyle \sum _{i=1}^L(w_2}\Vert P_t^{\left(i,l\right)}-P_t^{\left(Gt,l\right)}\Vert 2+w_3\left|\Delta \left(R_t^{\left(i,l\right)},R_t^{\left(Gt,l\right)}\right)\right|+w_4\left(1-a_{\left(i,l\right)}\right)+w_5(a_{\left(i,l\right)}\hfill \\ \begin{array}{l})\left|\Delta M\left(c_t^{\left(i,l\right)},c_t^{\left(Gt,l\right)}\right)\right|+w_6\left(a_{\left(i,l\right)}\right)\left|\Delta \phi \left(c_t^{\left(i,l\right)},c_t^{\left(Gt,l\right)}\right)\right|+w_7\left(1-{\beta }_{\left(i,l\right)}\right)+w_{\mathrm{8th}}\left({\beta }_{\left(i,l\right)}\right)|\\ \Delta \phi \left({\mathrm{i.e}}_t^{\left(i,l\right)},{\mathrm{i.e}}_t^{\left(Gt,l\right)}\right)|+w_9\left(1-g_{\left(i,l\right)}\right)+w_{10}\left(g_{\left(i,l\right)}\right)\left|\left({\mathrm{right}}_t^{\left(i,l\right)}-{\mathrm{right}}_t^{\left(Gt,l\right)}\right)\right|)\end{array}\hfill \end{array}\end{array}$$

For the first term in the cost function, a comparison of q _t s between the simulated and the real robots can make sense even if they have the same u _t , since q _t depends on the collision constraints imposed by the current object pose in contact with the robot hand imposed may be different, which may make it physically impossible for a joint to achieve a commanded target angle.

In at least one embodiment, a touch sensor is considered in contact when its force magnitude is greater than a certain threshold. In at least one embodiment, α _(i,l) equals 1 if the binary contact state of the l th touch sensor of the ith simulation matches that of the real touch sensor, and equals 0 otherwise. In at least one embodiment, β _{(i,l )} equal to 1 if the l th touch sensor of the ith simulation agrees with the real touch sensor in whether or not the sensor experiences translational slip, 0 otherwise; γ _(i,l) is the same for rotating slip.

an. In mindestens einem Ausführungsbeispiel werden die Gewichte der Kostenterme, w_is, so bestimmt, dass der entsprechende mittlere Betrag jedes Terms ungefähr auf 1 normiert ist.In at least one embodiment, for any two vectors, ΔM(•,•) gives the difference in their magnitudes and Δφ(•,•) gives the angle between them. In at least one embodiment, for any two rotations R _a and R _b , Δ(R _a ,R _b ) gives the angle of the axis angle representation of $R_a^{-1}{R}_b$

on. In at least one embodiment, the weights of the cost terms, w _i s, are determined such that the corresponding mean magnitude of each term is approximately normalized to unity.

In various embodiments, two sources of uncertainty may arise in the simulation's estimation of the object pose: 1) the initial pose estimate p ₀ from the vision-based pose estimator may be noisy, and 2) there may be a mismatch between the simulated and real dynamics, caused in part by imperfect modeling and partly due to the unknown physical parameters θ of the real world.

zentriert ist, und die Anzahl der Simulationen K erhöht wird. In verschiedenen Ausführungsbeispielen, wenn K beliebig groß ist, kann es mit hoher Wahrscheinlichkeit sein, dass die wahre Ausgangspose in der Menge der Simulationen ausreichend repräsentiert ist, und eine gut entworfene Kostenfunktion die richtige Simulation mit der richtigen Pose auswählen kann. In mindestens einem Ausführungsbeispiel werden die Translation und die Rotation getrennt abgetastet, um eine Abtastung über die initialen Objektposen durchzuführen. In mindestens einem Ausführungsbeispiel wird die Translation aus einer isotropen Normalverteilung abgetastet, während die Rotation abgetastet werden kann, indem isotrope Tangentialvektoren mit Mittelwert Null in so(3) gezeichnet und dann auf die mittlere Rotation angewendet werden.In at least one embodiment, the first problem of uncertainty of the initial pose estimate is addressed by perturbing the initial pose estimates across the various simulations by sampling from a distribution centered around the vision-based estimated pose $p_0^{\left(i\right)}~N\left({\mathrm{<figure-callout\; id="0"\; label="p"\; filenames="DE102020127508B4\_0032.png"\; state="\{\{state\}\}">p</figure-callout>}}_0,{\sum }_p\right)$

is centered and the number of simulations K is increased. In various embodiments, if K is arbitrarily large, there can be a high probability that the true starting pose is sufficiently represented in the set of simulations, and a well-designed cost function can select the right simulation with the right pose. In at least one embodiment, translation and rotation are sampled separately to scan across the initial object poses. In at least one embodiment, translation is sampled from an isotropic normal distribution, while rotation may be sampled by plotting zero-mean isotropic tangent vectors in so(3) and then applying them to the mean rotation.

In at least one embodiment, a second sim to real physics mismatch problem is addressed by using derivative-free, sample-based optimization algorithms to tune θ during pose tracking. In various embodiments, the average cost of all simulations during this window, together with the simulation state and parameters, can be passed to a particular optimizer after every Ttimesteps. In at least one embodiment, the optimizer determines the next set of simulations with their own updated parameters. In at least one embodiment, the simulations of the next set are extracted from the simulations of the current set, adding some additional perturbations (deviations) to the simulation parameters and object pose. In at least one embodiment, such an approach can maintain the diversity of the simulations and prevent them from getting stuck in sub-optimal simulation parameters or states due to noisy observations.

Although it may be desirable to have θ ^(i*) converge to the true θ ^(gt) , this may not be necessary to achieve a good pose estimate. Furthermore, due to the differences between the simulated and real dynamics, the optimal θ for reducing C to the corresponding values in the real world may show slight deviations from the calculated theoretical predictions.

The parameters of the K-simulations can be optimized by three derivative-free, sample-based optimizers:

In one embodiment, a weighted resampling ("WRS") optimizer is used. In various exemplary embodiments, WRS can form a probability mass function (PMF) over the existing simulation states s ^(1:K) and Sample K times with substitution from this distribution to form the next set of simulations. To form the PMF, WRS can apply Softmax to the simulation cost: $$P\left(i\right)=\frac{exp-\lambda \left(C_i-mi{n}_j{C}_j\right)}{{\displaystyle {\sum }_{i=1}^{K}exp-\lambda \left(C_i-mi{n}_j{C}_j\right)}}$$

Here λ can be a temperature hyperparameter that determines the sharpness of the distribution. In at least one embodiment, after resampling, an exploration of simulations is performed by perturbing the simulation parameters θ and the object pose.

$$

wobei Σ_θ vordefiniert sein kann. Der tiefgestellte Index τ kann den Aktualisierungsschritt des Optimierers bezeichnen (nach τ Aktualisierungsschritten ist die Simulation insgesamt τT Zeitschritte lang gelaufen).In at least one embodiment, the simulation parameters are perturbed by sampling from an isotropic normal distribution around the previous parameters: ${\theta }_{\tau +1}^{\left(i\right)}~N\left({\theta }_{\tau }^{\left(i\right)},{\sum }_{\theta }\right),$

$$

where Σ _θ can be predefined. The subscript τ can denote the update step of the optimizer (after τ update steps the simulation has run for a total of τT time steps).

In one embodiment, a Relative Entropy Policy Search ("REPS") optimizer is used. In at least one embodiment, a sample-based variant of REPS is used that computes weights for each simulation and samples from a distribution formed by the softmax of those weights. While WRS can use a fixed λ parameter to shape the distribution, REPS can look for an adaptive temperature parameter η that best improves the performance of the overall distribution as a function of ∈, imposing a constraint on the KL divergence between the old and the updated sample distribution can be.

$$P\left(i\right)=\frac{\text{exp}\frac{R_i}{\eta *}}{{\displaystyle {\sum }_{j=1}^K\text{exp}\frac{R_i}{\eta }}}$$

To use REPS, in at least one embodiment, costs are reformulated as rewards by setting _Ri = max _j C _j + min _j C _j - C _i . The parameter η can be calculated in each step by optimizing the dual function g(η) and then η can be used to form the PMF: $$n*=\text{bad min}n\in +nlOG\frac{1}{K}{\displaystyle \sum _{i=1}^K\text{ex}}\frac{R_i}{n}$$

$$P\left(i\right)=\frac{\text{ex}\frac{R_i}{n*}}{{\displaystyle {\sum }_{j=1}^K\text{ex}\frac{R_i}{n}}}$$

After resampling, a simulation can be perturbed in the same way as in WRS.

In various embodiments, a population based optimization algorithm ("PBO") may be used. In at least one embodiment, the PBO algorithm ranks all simulations by their average cost and finds the best K _best simulations with the lowest cost. In at least one embodiment, the algorithm exploits, replacing the remaining K - K _best simulations with copies of the K _best simulations sampled with replacement, and explores, by examining the K _best simulations in the same manner as at WRS disturbs (engl. perturb). In at least one embodiment, PBO effectively uses shaped costs that depend only on the relative location of the simulation costs and not on their magnitudes, potentially making the optimizer more robust to noisy costs.

• • T, as die Zeitschritte repräsentieren kann, die ein Algorithmus bei jeder Aktualisierung wartet.
• • K, das die Anzahl der gleichzeitigen Simulationen repräsentieren kann.
• • θ₀, das die initiale Normalverteilung der Simulationsparameter repräsentieren kann.
• • Σ_p, das die diagonale Kovarianzmatrix für die Normalverteilung über die initiale Posenstörung repräsentieren kann.
• • Σ_θ und Σ_v, die die diagonalen Kovarianzen der Normalverteilungen der für die Exploration verwendeten Störungen repräsentieren können.

In at least one embodiment, the optimizers described above employ a distribution-shaping hyperparameter that is used to balance exploration and exploitation. In at least one embodiment, various embodiments may use combinations of additional hyperparameters such as the following:

• • T, as can represent the time steps an algorithm waits at each update.
• • K, which can represent the number of simultaneous simulations.
• • θ ₀ , which can represent the initial normal distribution of the simulation parameters.
• • Σ _p , which can represent the diagonal covariance matrix for the normal distribution over the initial pose perturbation.
• • Σ _θ and Σ _v , which can represent the diagonal covariances of the normal distributions of the faults used for exploration.

A larger K can generally be better than a smaller K, with the caveat that the resulting simulation can be slower and impractical in the application. Σ _p can be so large that the actual initial pose is well represented in the initial pose distribution. However, with a larger Σ _p and the convariance of θ ₀ , K can be increased to ensure that the sampling density is high enough to capture a broader distribution.

In mindestens einem Ausführungsbeispiel erhöht die Erweiterung von Σ_θ oder Σ_v die Geschwindigkeit, mit der sich der Satz der Simulationsparameter „bewegt“, und der Optimierer kann mehr erforschen als ausnutzen. In mindestens einem Ausführungsbeispiel verbessert eine Erhöhung von T die Optimierung für θ, da der Optimierer mehr Stichproben zur Evaluierung jeder Simulation haben kann. Eine zu langsame Aktualisierung der Simulationsparameter kann jedoch zu einer Drift bei der Posenabschätzung führen, wenn sich die Simulation mit den geringsten Kosten genügend von der realen Welt unterscheidet, was in einigen Beispielen zu divergentem Verhalten führen kann. In einigen Beispielen kann ein abweichendes Verhalten auftreten, wenn eine Kraftpertubation oder einige Simulationsparameter zu einer nicht wiederherstellbaren Konfiguration führen, bei der das Objekt aus der Hand fällt oder das Objekt in eine solche Pose gebracht wird, dass kleine Kraftpertubationen es nicht mehr in die richtige Pose zurückbringen können. In einigen Beispielen kann dies akzeptabel sein, wenn einige wenige Abtastungen divergent werden. Ihre Kosten können hoch sein, so dass sie in einigen Ausführungsbeispielen verworfen und durch solche ersetzt werden können, die während der Optimierungsaktualisierung nicht divergent sind.In at least one embodiment, there are two additional tradeoffs with these hyperparameters. In at least one embodiment, one tradeoff is the tradeoff between exploration and exploitation in the context of optimization for θ, and the other tradeoff is the tradeoff between optimization of θ and of $p_t^{\left(i*\right)}.$

In at least one embodiment, expanding _Σθ or _Σv increases the speed at which the set of simulation parameters "moves" and allows the optimizer to explore more than it exploits. In at least one embodiment, increasing T improves the optimization for θ because the optimizer can have more samples to evaluate each simulation. However, updating the simulation parameters too slowly can lead to a drift in the pose estimation if the lowest-cost simulation differs enough from the real world, which can lead to divergent behavior in some examples. In some examples, deviant behavior may occur when force perturbation or some simulation parameters result in an unrecoverable configuration where the object falls out of hand or the object is posed in such a way that small force perturbations no longer put it in the correct pose can bring back. In some examples, this may be acceptable when a few samples become divergent. Their cost can be high, so in some embodiments they can be discarded and replaced with ones that are not divergent during the optimization update.

In various embodiments, a physics simulator can be used as a forward model instead of a constant model. In addition to tracking the object pose, the proposed algorithm can also identify the context of the forward model by adjusting the simulation parameters θ, which cannot be influenced by the forward or observation model. The optimizers can be based on discrete sampling, since poses of hand-held objects from a given distribution cannot easily be sampled by contacts due to complicated network penetration limitations.

In various embodiments, the system can be evaluated with both simulations and real-world experiments using an Allegro hand on a Kuka IIWA7 robotic arm, although in various embodiments any hand, robotic arm, robotic component, and /or variations thereof may be used. The in-hand manipulation trajectories can first be collected using a hand-tracking teleoperation system, and the pose estimation errors can be evaluated by running the proposed algorithms offline on the collected trajectories. 9 Figure 1 shows an example of an algorithm that can be used by the system. These trajectories can begin and end with the object being out of hand and out of occlusion. Due to the presence of ground truth object poses in simulation experiments, detailed ablation studies can be performed in the simulation to investigate the impact of various hyperparameters on the algorithm performance. In various embodiments, PoseRBPF, a newer pose estimation algorithm based on RGB D and particle filters, may be used to obtain the initial and final object pose. The initial and final object poses can be treated as ground truths and the final pose can be compared to the pose predicted by the system.

In various embodiments, the 4-finger, 16-DoF Allegro hand can be mounted on the Kuka IIWA7 7-DoF robotic arm. To get feedback from real-world contacts, SynTouch BioTac sensors, or a variation thereof, can be attached to each of the fingertips. One or more processes performed in conjunction with various sensors and the sensors' raw electrode readings may be used to determine the contact power to predict slip direction and grip stability. A trained model can be used to estimate force vectors c _t . In various examples, the cost functions may not include terms for slack. The simulations may be performed on a computer having one or more graphical processing units, one or more central processing units, and one or more storage units.

In various embodiments, three objects from the Yale Columbia Berkeley ("YCB") object dataset (Spam Can, Foam Block, and Toy Banana) may be used with models, textures, and point clouds. The objects can be chosen based on constraints of the size of the robotic hand and that the objects be light enough for the robotic hand (e.g., the Allegro hand) to perform robust, precision grips with its fingertips.

In some examples, for each object, both in simulation and in real experiments, 2 demonstrations of 2 types of trajectories can be used for manipulation: 1) pick-and-place using a finger grip and rotate the object in the hand and 2) the same but with fingertips breaking and re-establishing contact during the grip (finger movements). This can give a total of 24 trajectories that can be analyzed for both simulations and real-world experiments. In both trajectories, the object can be translated and rotated both by inertial forces and by pressure contacts with the table. Each trajectory can last about a minute. In various exemplary embodiments, the pose estimation algorithm can be executed at approximately 30 Hz, which can result in a total of approximately 2k individual images per trajectory.

In at least one embodiment, the input to the system is a point cloud of the human demonstrator's hand. In at least one embodiment, a neural network, e.g., a neural network based on the PointNet++ neural network, associates the point cloud with an estimate of the hand pose relative to the camera and the hand's joint angles. In at least one embodiment, these estimates, along with a moving hand model and the original point cloud, are then passed to DART, which performs the tracking by refining the neural network estimates. Finally, to perform kinematic realignment, an optimization problem is solved that finds the Allegro hand joint angles that result in fingertip poses that approximate those of the human hand, according to one embodiment.

In addition to the optimizers mentioned above (WRS, REPS, PBO), the following two basic functions can also be evaluated: Open Loop ("OLP") and Identity ("EYE"). OLP can track the object pose with 1 simulation. EYE can be initialized with a set of noisy initial poses and can always choose the pose of the simulation with the lowest cost, but it cannot perform resampling or optimization updates. In various exemplary embodiments, an average distance deviation (“ADD”) can be used as an evaluation criterion. ADD can calculate the average distance between corresponding points in the object point cloud that are in the ground truth pose and the predicted pose.

In various embodiments, the arm and hand in the simulation can be controlled via a joint angle PD controller, and the controller's gains can be adjusted so that the joint angle jump responses resemble those of the real robot. To speed up the simulation, the collision meshes of the robot and objects can be simplified. This can be done with various tools, e.g. B. by applying an algorithm such as the TetWild algorithm, which yields a mesh with triangles that are more equilateral than other algorithms such as e.g. B. MeshLab 1's Quadric Edge Collapse Decimation. In total, each simulation can generate at most 200 contacts during manipulation, and K=40 simulations can be performed at 30 Hz.

The simulation experiments can be performed with different amounts of initial noise of the initial pose. Three levels can be tested: Low can have a translation standard deviation of 1 mm and a rotation standard deviation of 0.01 radians. Medium can be 5 mm and 0.1 radians, and High can be 10 mm and 1 radians.

8th shows an example of a comparison of the optimizers when tracking object poses in the hand over all trajectories of the simulation. ADD 802 may increase with increasing initial pose error, and the mean ADD for the optimizer-based methods may be lower. While EYE a comparable mean ADD can be achieved with the optimizer methods, the latter can have a much smaller error variance and a smaller maximum error. The optimizers 804 can focus the distribution of the simulations on the better performing ones over time. In the case of mean noise, REPS and PBO can achieve the best ADD with a mean of 5.8 mm and 5.9 mm, respectively.

Show in one embodiment 10 and 11 Examples of ablation study results in simulations run with the hyperparameters controlling the exploration distance (how much the simulations are disturbed), the number of parallel simulations, and whether the contact and slip detection feedback is used in the cost function or not, determine.

The algorithms can be evaluated on real-world trajectories that are similar to those from the simulation. PoseRBPF can be used to register the object pose in the first and last frame of a trajectory. The initial pose estimate can be used to initialize the simulations, while the last one can be used to assess the accuracy of the contacts-based pose tracking algorithm. In contrast to the simulation experiments, in the real-world experiments the object pose can be initialized by sampling from the distribution over the object poses of PoseRBPF such that the initial pose samples correspond to the uncertainties of the vision-based pose estimation algorithm.

12 shows an example of real test results. The ADDs can be higher than those from simulation experiments. This may be due both to the fact that the real-world dynamics are more dissimilar to the simulations than the simulations with different parameters, and to the fact that the real-world observations can be noisier than those in the simulations. In various embodiments, few or no optimizers may be able to track the toy banana for the real world data. The object's long moment arm and low coefficient of friction can make the object's slip behavior difficult to model accurately. This can be a failure mode of the algorithm where if all simulations diverge (e.g. the banana rotates in the wrong direction or falls out of hand) the algorithm cannot recover in subsequent optimizer updates. The best ADD achieved with foam may be 14.1mm with PBO, and with Spam 12.2mm with REPS.

In at least one embodiment, a sample-based optimization algorithm for tracking poses of in-hand objects during manipulation through feedback through contacts and GPU-accelerated robotic simulation is developed. In at least one embodiment, parallel simulations simultaneously maintain many principles about the real world and model changes in object poses that may be induced by complex contact dynamics. In at least one embodiment, the optimization algorithm adjusts the simulation parameters during object pose tracking to further improve tracking performance. In various embodiments, the detection of contacts in the real world can be improved through the use of vision in the loop.

13 Figure 12 shows an example of a process that, as a result of execution by a computer system, determines the pose of an object being manipulated by a robotic hand equipped with tactile force sensors. In at least one embodiment, a computer system having one or more processors executes executable instructions stored in a computer-readable memory and, as a result of execution, causes the computer system to be 13 operations shown and described below.

In at least one embodiment, in step 1302, the computer system obtains tactile information from one or more touch sensors of a robotic hand used to manipulate an object. In at least one embodiment, the touch sensors are BioTac sensors equipped with a friction coating. In at least one embodiment, the touch sensor data is a two-dimensional array of force data representing forces distributed across the surface of the touch sensor.

In at least one embodiment, in step 1304, the computer system creates a set of simulations that simulate the manipulation performed by the robot and determines a set of simulated touch-sensor information 1306 for each simulation. In at least one embodiment, each simulation uses a different pose for the object. In at least one embodiment, an estimated pose for the object is obtained from one or more images of the object.

In at least one embodiment, in step 1308, the computer system determines a cost value for each simulation based at least in part on the differences between the touch sensor information measured in the real world and the touch information determined in the simulation. In one embodiment, the cost may be determined as described above. Based at least in part on the cost determined for each simulation, a simulation that most closely matches real-world observations 1310 is identified. In at least one embodiment, the pose of the object in the identified simulation is compared to the pose of the object in the real world definitely 1312.

In at least one embodiment, the pose may be used in performing a task that involves manipulating the object in hand. For example, the pose of a nut in the hand can be determined so that the robot can position the nut onto a threaded bolt. In another example, the pose of an object in hand can be used when placing the object on a horizontal surface to ensure that the object is upright and stable when released from the robotic hand.

14A 14 shows the inference and/or training logic 1415 used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with below 14A and or 14B provided.

In at least one embodiment, inference and/or training logic 1415 may include, without limitation, code and/or data storage 1401 for storing forward and/or output weight and/or input/output data and/or other neuron or layer configuration parameters of a neural network trained and/or used for inference, according to aspects of one or more embodiments. In at least one embodiment, training logic 1415 may include or be coupled to code and/or data storage 1401 to store graph code or other software for controlling the timing and/or order in which weight and/or other parameter information is loaded to configure logic, including integer and/or floating point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code, such as graph code, loads weight or other parametric information into processor ALUs based on a neural network architecture to which the code conforms. In at least one embodiment, the code and/or data store 1401 stores weight parameters and/or input/output data of each neural network layer used during forward propagation of input/output data and/or weight parameters during training and/or inference using trained or used by aspects of one or more embodiments. In at least one embodiment, any portion of code and/or data memory 1401 may be aggregated with other on-chip or off-chip data memory, including a processor's L1, L2, or L3 cache or system memory.

In at least one embodiment, any portion of code and/or data storage 1401 may be implemented internally or externally by one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and/or code and/or data memory 1401 may be cache memory, dynamic randomly addressable memory (“DRAM”), static randomly addressable memory ( static randomly addressable memory (“SRAM”), non-volatile memory (e.g. flash memory) or other memory. In at least one embodiment, the choice of whether the code and/or code and/or data memory 1401 is, for example, on-processor or off-processor, or consists of DRAM, SRAM, Flash, or some other type of memory, may depend on the memory available on-chip versus that off-chip memory, latency requirements of the training and/or inference functions performed, the stack size of the data used in inferring and/or training a neural network, or a combination of these factors.

In at least one embodiment, the inference and/or training logic 1415 may include, without limitation, code and/or data storage 1405 to store reverse and/or output weights and/or input/output data corresponding to neurons or layers of a neural network trained and/or used in aspects of one or more inference embodiment(s). In at least one embodiment, the code and/or data store 1405 stores weight parameters and/or input/output data of each neural network layer used during back propagation of input/output data and/or weight parameters during training and/or inference trained or used using aspects of one or more embodiments. In at least one embodiment, training logic 1415 may include or be coupled to code and/or data storage 1405 to store graph code or other software for controlling the timing and/or order in which weight and/or other parameter information is loaded to configure logic, including integer and/or floating point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code, such as graph code, loads weight or other parametric information into processor ALUs based on a neural network architecture to which the code conforms. In at least one embodiment, any portion of code and/or data memory 1405 may be aggregated with other on-chip or off-chip data memory, including a processor's L1, L2, or L3 cache memory or system memory. In at least one embodiment, any portion of code and/or data storage 1405 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and/or data storage 1405 may be cache memory, DRAM, SRAM, non-volatile memory (eg, flash memory), or other memory. In at least one embodiment, the choice of whether the code and/or data storage 1405 is, for example, on-processor or off-processor, or consists of DRAM, SRAM, Flash, or other memory type, may depend on the available memory on-chip or off-chip, latency requirements of the training and/or inference functions performed, the batch size of the data used in the inference and/or training of a neural network, or a combination of these factors.

In at least one embodiment, code and/or data storage 1401 and code and/or data storage 1405 may be separate storage structures. In at least one embodiment, code and/or data memory 1401 and code and/or data memory 1405 may be the same memory structure. In at least one embodiment, code and/or data storage 1401 and code and/or data storage 1405 may be partially the same memory structure and partially separate memory structures. In at least one embodiment, any portion of code and/or data memory 1401 and code and/or data memory 1405 may be co-located with other on-chip or off-chip data memory, including L1, L2, or L3 cache -Memory or system memory of a processor.

In at least one embodiment, the inference and/or training logic 1415 may include, without limitation, one or more arithmetic logic unit(s) ("ALU(s)") 1410, including integer and/or floating point units, to provide logical and/or perform mathematical operations based at least in part on or specified by training and/or inference code (e.g. graph code), the result of which can produce activations (e.g. output values of layers or neurons within a neural network) stored in an activation memory 1420, which are functions of input/output and/or weight parameter data stored in code and/or data memory 1401 and/or code and/or data memory 1405. In at least one embodiment, activations stored in an activation memory 1420 are generated according to linear algebraic and/or matrix-based mathematics performed by ALU(s) 1410 in response to execution of instructions or other code stored in code and/or data memory 1405 and/or data 1401 stored weight values as operands together with other values, such as bias values, gradient information, pulse values or other parameters or hyperparameters, which can be stored arbitrarily or completely in code and/or data memory 1405 or code and/or or data memory 1401 or another memory can be stored on or off-chip.

In at least one embodiment, the ALU(s) 1410 reside within one or more processors or other hardware logic device or circuitry, while in another embodiment, the ALU(s) 1410 reside external to a processor or other hardware logic device or circuitry who can use it (e.g. a co-processor). In at least one embodiment, the ALU(s) 1410 may be contained within the execution units of a processor or otherwise in a group of ALUs to which the execution units of a processor are either within the same processor or distributed among different processors of different types (e.g., central processing units, graphics processing units, units with fixed functionality, etc.). In at least one embodiment, data memory 1401, code and/or data memory 1405, and activation memory 1420 may reside on the same processor or other hardware logic device or circuitry, while in another embodiment they reside in different processors or other hardware logic devices. Logic devices or circuits or a combination of the same and various processors or other hardware logic devices or circuits. In at least one embodiment, any portion of activation memory 1420 may be aggregated with other on-chip or off-chip data memory, including a processor's L1, L2, or L3 cache memory or system memory. Additionally, the inference and/or training code may be stored with other code accessible by a processor or other hardware logic or circuitry and using retrieval, decoding, scheduling, execution, shutdown and/or other logic circuits of a processor.

In at least one embodiment, enable memory 1420 may be cache memory, DRAM, SRAM, non-volatile memory (eg, flash memory), or other memory. In at least one embodiment, activation memory 1420 may reside in whole or in part internal or external to one or more processors or other logic circuits. In at least one embodiment, the choice of whether the activation memory 1420 is, for example, on-processor or off-processor, or consists of DRAM, SRAM, Flash, or another type of memory, may depend on the available on-chip or off-chip memory, the latency requirements of the training and /or inference functions, the batch size of the data used in inference and/or training a neural network, or a combination of these factors. In at least one embodiment, the 14A The illustrated inference and/or training logic 1415 may be used in conjunction with an application specific integrated circuit ("ASIC"), such as Google's Tensorflow® Processing Unit, a Graphcore™ Inference Processing Unit (IPU), or a Nervana® (e.g., " Lake Crest") processor from Intel Corp. In at least one embodiment, the 14A The illustrated inference and/or training logic 1415 may be used in conjunction with central processing unit ("CPU") hardware, graphics processing unit ("GPU") hardware, or other hardware such as field programmable gate arrays ("FPGAs") .

14B FIG. 14 shows the inference and/or training logic 1415 according to at least one embodiment. In at least one embodiment, the inference and/or training logic 1415 may include, without limitation, hardware logic in which computational resources are dedicated or otherwise used solely in connection with weight values or other information corresponding to one or more layers of neurons within a neural network . In at least one embodiment, the 14B illustrated inference and/or training logic 1415 can be used in conjunction with an application specific integrated circuit (ASIC) such as Google's Tensorflow® Processing Unit, a Graphcore™ Inference Processing Unit (IPU) or a Nervana® (e.g. "Lake Crest ") Processor by Intel Corp. In at least one embodiment, the 14B The illustrated inference and/or training logic 1415 may be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware such as field programmable gate arrays (FPGAs). In at least one embodiment, the inference and/or training logic 1415 includes, without limitation, code and/or data storage 1401 and code and/or data storage 1405 used to store code (e.g., graph code), weight values, and/or other information, including default values, gradient information, momentum values and/or other parameter or hyperparameter information. In at least one embodiment described in 14B As shown, both code and/or data storage 1401 and code and/or data storage 1405 are associated with a dedicated computing resource, such as computing hardware 1402 and computing hardware 1406, respectively. In at least one embodiment, both computing hardware include 1402 and the computing hardware 1406 one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in the code and/or data memory 1401 or in the code and/or data memory 1405 and the result thereof is stored in activation memory 1420.

In at least one embodiment, each of the code and/or data stores 1401 and 1405 and the corresponding computational hardware 1402 and 1406, respectively, corresponds to different layers of a neural network such that the activation of the code resulting from a "storage/computation pair 1401/1402". - and/or data memory 1401 and computational hardware 1402 is provided as input to the next "memory/computation pair 1405/1406" of code and/or data memory 1405 and computational hardware 1406 to conceptualize the organization of a neural network to reflect. In at least one embodiment, each of the storage/computation pairs 1401/1402 and 1405/1406 may correspond to more than one layer of a neural network. In at least one embodiment, additional storage/computation pairs (not shown) may be subsequent to or in parallel with storage/computation pairs 1401/1402 and 1405/1406 in inference and/or training logic 1415.

15 shows the training and use of a deep neural network, according to at least one embodiment. In at least one embodiment, the untrained neural network 1506 is trained using a training data set 1502 . In at least one embodiment, the training framework (framework) 1504 is a PyTorch framework, while in other embodiments the training framework 1504 is a Tensorflow, Boost, Caffe, Microsoft Cognitive Toolkit/CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training framework. In at least one embodiment, the training framework 1504 trains an untrained neural network 1506 and enables it to be trained using the processing resources described herein to generate a trained neural network 1508 . In at least one embodiment, the weights may be chosen randomly or by pre-training using a deep belief network. In at least one embodiment, the training may be either supervised, partially supervised, or unsupervised.

In at least one embodiment, the untrained neural network 1506 is trained using supervised learning, where the training data set 1502 includes an input paired with a desired output for an input, or where the training data set 1502 includes an input with a known output and an output of the neural network 1506 is manually graded. In at least one embodiment, the untrained neural network 1506 is trained under supervision, processes inputs from the training data set 1502, and compares the resulting outputs to a set of expected or desired outputs. Errors are then propagated back through the untrained neural network 1506 in at least one embodiment. In at least one embodiment, the training framework 1504 adjusts weights that control the untrained neural network 1506 . In at least one embodiment, the training framework 1504 includes tools to monitor how well the untrained neural network 1506 is converging to a model, such as the trained neural network 1508, that is appropriate given known input data such as new data 1512 to generate correct answers as in the result 1514. In at least one embodiment, the training framework 1504 repeatedly trains the untrained neural network 1506 while adjusting the weights to refine an output of the untrained neural network 1506 using a loss function and an adjustment algorithm such as stochastic gradient descent. In at least one embodiment, the training framework 1504 trains the untrained neural network 1506 until the untrained neural network 1506 achieves a desired accuracy. In at least one embodiment, the trained neural network 1508 can then be used to implement any number of machine learning operations.

In at least one embodiment, the untrained neural network 1506 is trained using unsupervised learning, where the untrained neural network 1506 attempts to train itself using untagged data. In at least one embodiment, unsupervised learning dataset 1502 includes input data with no associated output data or ground truth data. In at least one embodiment, the untrained neural network 1506 can learn groupings within the training data set 1502 and determine how individual inputs relate to the untrained data set 1502 . In at least one embodiment, unsupervised training can produce a self-organizing map that is a type of trained neural network 1508 capable of performing operations useful in reducing the dimensionality of new data 1512 . In at least one embodiment, unsupervised training may also be used to perform anomaly detection, allowing for the identification of data points in a new data set 1512 that deviate from the new data set 1512's normal patterns.

In at least one embodiment, semi-supervised learning, a technique in which the dataset 1502 under training comprises a mixture of tagged and untagged data, may be used. In at least one embodiment, the training framework 1504 may be used to perform incremental learning, e.g., through delegated learning techniques. In at least one embodiment, incremental learning allows the trained neural network 1508 to adapt to new data 1512 without forgetting the knowledge that entered the network during initial training.

16 16 shows an example of the data center 1600 in which at least one embodiment may be employed. In at least one embodiment, the data center 1600 includes a data center infrastructure layer 1610, a framework layer 1620, a software layer 1630, and an application layer 1640.

In at least one embodiment, as in 16 As illustrated, the data center infrastructure layer 1610 may include a resource orchestrator 1612, clustered computing resources 1614, and node computing resources ("CRs") 1616(1)-1616(N), where "N" is any whole, represents a positive integer. In at least one embodiment, node CRs 1616(1)-1616(N) may include any number of central processing units ("CPUs") or other processors (including accelerators, field programmable gate arrays (FPGAs), GPUs, etc.), storage devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input/output ("NW I/O") devices, network switches, virtual machines ("VMs"), power and cooling modules, etc. In at least one embodiment, one or more node CRs from among node CRs 1616(1)-1616(N) may be a server with one or more of the above computing resources.

In at least one embodiment, clustered computing resources 1614 may include separate clusters of node C.R.s housed in one or more racks (not shown) or many racks housed in data centers in different geographic locations (also not shown). Separate groupings of node C.R.s within grouped compute resources 1614 may include grouped compute, network, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, multiple node C.R.s including CPUs or processors may be grouped within one or more cabinets to provide computing resources in support of one or more workloads. In at least one embodiment, one or more cabinets may also include any number of power modules, cooling modules, and network switches in any combination.

In at least one embodiment, resource orchestrator 1612 may configure or otherwise control one or more node C.R.s 1616(1)-1616(N) and/or clustered computing resources 1614. In at least one embodiment, resource orchestrator 1612 may include a software design infrastructure ("SDI") manager for data center 1600 . In at least one embodiment, the resource orchestrator may include hardware, software, or a combination thereof.

In at least one embodiment, as in 16 shown, the framework layer 1620 includes a task scheduler 1632, a configuration manager 1634, a resource manager 1636 and a distributed file system 1638. In at least one embodiment, the framework layer 1620 can be a framework for supporting the software 1632 of the software layer 1630 and/or one or more application(s) 1642 of the application layer 1640. In at least one embodiment, software 1632 or application(s) 1642 may each include web-based service software or applications such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 1620 may be some type of free and open source software web application framework, such as Apache Spark™ (hereinafter “Spark”), which implements the distributed file system 1638 for processing large amounts of data (e.g., “big data”). can use, but is not limited to. In at least one embodiment, the task scheduler may include a Spark driver to facilitate scheduling of workloads supported by different layers of the data center 1600. In at least one embodiment, the configuration manager 1634 may be able to configure various layers such as the software layer 1630 and the framework layer 1620 including Spark and the distributed file system 1638 to support processing large amounts of data. In at least one embodiment, resource manager 1636 may be capable of managing clustered or grouped computing resources associated with or allocated to support distributed file system 1638 and task scheduler 1632 . In at least one embodiment, clustered or clustered computing resources may include clustered computing resource 1614 at data center infrastructure layer 1610 . In at least one embodiment, the resource manager 1636 may coordinate with the resource orchestrator 1612 to manage these allocated or assigned computing resources.

In at least one embodiment, software 1632 included in the software layer 1630 may include software that is generated from at least portions of the node CRs 1616(1)-1616(N), clustered computing resources 1614, and/or the distributed file system 1638 of the Framework layer 1620 is used. One or more types of software may include, but are not limited to, Internet web site search software, e-mail virus scanning software, database software, and video streaming software.

In at least one embodiment, the application(s) 1642 contained in the application layer 1640 may include one or more types of applications hosted by at least portions of the node C.R.s 1616(1)-1616(N), clustered computing resources 1614 and/or the distributed file system 1638 of the framework layer 1620. One or more types of applications can include, but are not limited to, any number of genomics applications, cognitive computation, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc .) or other machine learning applications used in connection with one or more embodiments.

In at least one embodiment, each of configuration manager 1634, resource manager 1636, and resource orchestrator 1612 may perform any number and type of self-modifying actions based on any amount and type of data collected in any technically appropriate manner. In at least one embodiment, self-modifying actions may relieve a data center 1600 operator from potentially making poor configuration decisions and potentially avoiding underutilized and/or underperforming portions of a data center.

In at least one embodiment, data center 1600 may include tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, according to at least one embodiment, a machine learning model may be trained by computing weight parameters according to a neural network architecture using the software and computational resources described above with respect to data center 1600 . In at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1600 using weight parameters defined by one or more described herein training techniques are calculated.

In at least one embodiment, a data center may use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware to perform training and/or inference using the resources described above. Additionally, one or more of the software and/or hardware resources described above may be configured as a service to enable users to train or perform inference of information, such as image recognition, speech recognition, or other artificial intelligence services.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, the inference and/or training logic 1415 in the system 16 for inferring or predicting operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

At least one embodiment may be implemented using the techniques described above. For example, a generation network or an evaluation network as in 14-16 described.

17A 17 shows an example of an autonomous vehicle 1700, according to at least one embodiment. In at least one embodiment, autonomous vehicle 1700 (alternatively referred to herein as “vehicle 1700”) may be, without limitation, a passenger vehicle, such as an automobile, truck, bus, and/or other type of vehicle that accommodates one or more passengers. At least In one embodiment, vehicle 1700 may be an articulated lorry used to transport goods. In at least one embodiment, vehicle 1700 may be an aircraft, robotic vehicle, or other type of vehicle.

Autonomous vehicles can be described in terms of the level of automation defined by the National Highway Traffic Safety Administration ("NHTSA"), a division of the US Department of Transportation, and the Society of Automotive Engineers ("SAE") "Taxonomy and Definitions for Terms in Relation to propulsion automation systems for road motor vehicles” (e.g. Standard No. J3016-201806 published on June 15, 2018, Standard No. J3016-201609 published on September 30, 2016, and previous and future versions of this standard). In one or more embodiments, the vehicle 1700 may have functionality according to one or more of levels 1-5 of the autonomous driving levels. For example, in at least one embodiment, vehicle 1700 may have limited automation (level 3), high automation (level 4), and/or full automation (level 5), depending on the embodiment.

In at least one embodiment, vehicle 1700 may include, without limitation, components such as a chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. In at least one embodiment, vehicle 1700 may include, without limitation, a propulsion system 1750, such as an internal combustion engine, hybrid electric power plant, all-electric motor, and/or other type of propulsion system. In at least one embodiment, the propulsion system 1750 may be coupled to a powertrain of the vehicle 1700, which may include, but is not limited to, a transmission to enable propulsion of the vehicle 1700. In at least one embodiment, propulsion system 1750 may be controlled in response to receiving signals from one or more throttle/accelerators 1752 .

In at least one embodiment, a steering system 1754, which may include, among other things, a steering wheel, is used to steer a vehicle 1700 (e.g., along a desired path or route) when a propulsion system 1750 is operational (e.g., when the vehicle is in motion ). In at least one embodiment, a steering system 1754 may receive signals from steering actuator(s) 1756 . The steering wheel can be optional for full functionality of automation (level 5). In at least one embodiment, a brake sensor system 1746 may be used to actuate vehicle brakes in response to receiving signals from brake actuator(s) 1748 and/or brake sensors.

In at least one embodiment, the controller(s) 1736, which include, among other things, one or more system-on-chips ("SoCs") (not included in 17A shown) and/or graphics processing unit(s) ("GPU(s)"), may provide signals (eg, representative of commands) to one or more vehicle 1700 components and/or systems. For example, in at least one embodiment, controller(s) 1736 may send signals to actuate vehicle brakes via brake actuators 1748 , to actuate steering system 1754 via steering actuator(s) 1756 , to actuate powertrain system 1750 via throttle/accelerator 1752 . Controller(s) 1736 may include one or more onboard (eg, integrated) computing devices (eg, supercomputers) that process sensor signals and issue operational commands (eg, signals representing commands) to enable autonomous driving and/or a human driver to assist in driving the vehicle 1700 . In at least one embodiment, controller(s) 1736 may include a first controller 1736 for autonomous driving functions, a second controller 1736 for functional safety functions, a third controller 1736 for artificial intelligence (eg, computer vision) functionality, a fourth controller 1736 for infotainment functionality, a fifth controller 1736 for emergency redundancy and/or other controllers. In at least one embodiment, a single controller 1736 can handle two or more of the above functionalities, two or more controllers 1736 can handle a single functionality and/or any combination thereof.

In at least one embodiment, controller(s) 1736 provide signals to control one or more components and/or systems of vehicle 1700 in response to sensor data received from one or more sensors (eg, sensor inputs). In at least one embodiment, sensor data may be received from, for example and without limitation, Global Navigation Satellite System ("GNSS") sensor(s) 1758 (e.g., Global Positioning System sensor(s), RADAR sensor(s). ) 1760, ultrasonic sensor(s) 1762, LIDAR sensor(s) 1764, inertial measurement unit ("IMU") sensor(s) 1766 (eg, accelerometer, gyroscope(s), magnetic compass(es), magnetometer, etc.), microphone( e) 1796, stereo camera(s) 1768, Wide Angle Camera(s) 1770 (e.g. Fisheye Cameras), Infrared Camera(s) 1772, Surround Camera(s) 1774 (e.g. 360 Degree Cameras), Long Distance Camera(s) (not in 17A shown), mid-range camera(s) (not in 17A shown), speed sensor(s) 1744 (eg, for measuring speed of vehicle 1700), vibration sensor(s) 1742, steering sensor(s) 1740, brake sensor(s) (eg, as part of brake sensor system 1746), and/or other types of sensors.

In at least one embodiment, one or more controllers 1736 may receive inputs (e.g., represented by input data) from an instrument cluster 1732 of vehicle 1700 and outputs (e.g., represented by output data, display data, etc.) via a human Provide a machine interface ("HMI") display 1734, an audible alarm, a speaker, and/or via other vehicle 1700 components. In at least one embodiment, the outputs may include information such as vehicle speed, speed, time, map data (e.g., a high-resolution map (not in 17A shown), location data (e.g., the position of vehicle 1700, as on a map), direction, position of other vehicles (e.g., an occupancy grid), information about objects and the status of objects as perceived by controller(s) 1736, etc . For example, in at least one embodiment, the HMI display 1734 may display information about the presence of one or more objects (e.g., a road sign, a warning sign, a traffic light change, etc.) and/or information about maneuvers that the vehicle has performed, is performing, or is performing will perform (e.g., change lanes now, exit 34B in two miles, etc.).

In at least one embodiment, the vehicle 1700 further includes a network interface 1724 that may use wireless antenna(s) 1726 and/or modem(s) to communicate over one or more networks. For example, in at least one embodiment, the network interface 1724 may support communication via Long Term Evolution ("LTE"), Wideband Code Division Multiple Access ("WCDMA"), Universal Mobile Telecommunications System ("UMTS"), Global System for Mobile Communication ( "GSM"), IMT-CDMA Multi-Carrier ("CDMA2000"), etc. In at least one embodiment, wireless antenna(s) 1726 can also enable communication between objects in the vicinity (e.g., vehicles, mobile devices, etc.) using local area networks such as Bluetooth, Bluetooth Low Energy ("LE"), Z-Wave, ZigBee etc. and/or low-power broadband networks ("LPWANs") such as LoRaWAN, SigFox, etc.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, the inference and/or training logic 1415 in the system 17A to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the robot may be an autonomous vehicle, and the evaluation and generation of networks may be implemented using a computer system in the autonomous vehicle.

17B 17 shows an example with camera positions and fields of view for the autonomous vehicle 1700 of FIG 17A according to at least one embodiment. In at least one embodiment, the camera locations and respective fields of view are exemplary and should not be construed in a limiting sense. For example, in at least one embodiment, additional and/or alternative cameras may be included and/or the cameras may be located at different locations on the vehicle 1700 .

In at least one embodiment, camera types for cameras may include, but are not limited to, digital cameras configured for use with vehicle 1700 components and/or systems. The camera(s) may operate at Safety Integrity Level (“ASIL”) B and/or any other ASIL level. In at least one embodiment, the camera types can achieve any frame rate, such as 60 frames per second (fps), 1220 fps, 240 fps, etc., depending on the embodiment. In at least one embodiment, cameras may be deployed using rolling shutters, global shutters, another shutter type, or a combination thereof. In at least one embodiment, a color filter array may be a red-clear-clear-clear color filter array ("RCCC"), a Red-Clear-Clear-Clear-Blue Color Filter Array ("RCCB"), a Red-Blue-Green-Clear Color Filter Array ("RBGC"), a Foveon X3 Color Filter array, a Bayer sensor color filter array ("RGGB"), a monochrome sensor color filter array, and/or another type of color filter array. In at least one embodiment, cameras with clear pixels, such as cameras with an RCCC, an RCCB, and/or an RBGC color filter array, can be used to increase light sensitivity.

In at least one embodiment, one or more cameras may be used to perform advanced advanced driver assistance system (“ADAS”) functions (e.g., as part of a redundant or failsafe design). For example, in at least one embodiment, a multifunction mono camera may be installed that includes features such as lane departure warning, traffic sign assist, and intelligent headlamp control. In at least one embodiment, one or more cameras (e.g., all cameras) can record and provide image data (e.g., video) simultaneously.

In at least one embodiment, one or more cameras may be mounted in a mount, e.g. B. in a custom (three-dimensional ("3D") printed) assembly to eliminate stray light and reflections from the vehicle interior (e.g. reflections from the dashboard reflected in windshield mirrors) that could interfere with the image data collection capabilities of the camera. With respect to side mirror mounting assemblies, in at least one embodiment, side mirror assemblies may be custom 3D printed such that the camera mounting plate conforms to the shape of the side mirror. In at least one embodiment, the camera(s) may be integrated into the side view mirror. Side view cameras may also have the cameras integrated into four pillars at each corner of the cab, according to at least one embodiment.

In at least one embodiment, cameras with a field of view that includes portions of the environment in front of the vehicle 1700 (e.g., forward-facing cameras) can be used for all-around vision to identify forward-facing paths and obstacles, and to use one or more Provide controllers 1736 and/or control SoCs with information critical to creating an occupancy grid and/or determining preferred vehicle paths. In at least one embodiment, forward-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including, without limitation, emergency braking, pedestrian detection, and collision avoidance. In at least one embodiment, forward-facing cameras may also be used for ADAS functions and systems including, but not limited to, Lane Departure Warnings ("LDW"), Autonomous Cruise Control ("ACC") ") and/or other functions such as traffic sign recognition.

In at least one embodiment, a variety of cameras may be used in a forward-facing configuration, including, for example, a monocular camera platform that includes a color CMOS ("Complementary Metal Oxide Semiconductor") image sensor. In at least one embodiment, a wide-angle camera 1770 may be used to perceive objects coming into view from the periphery (eg, pedestrians, crossing traffic, or bicycles). Although only a wide-angle camera 1770 in 17B As shown, there may be any number (including zero) wide-angle cameras 1770 on the vehicle 1700 in other embodiments. In at least one embodiment, any number of remote camera(s) 1798 (eg, a remote stereo camera pair) may be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. In at least one embodiment, remote cameras 1798 can also be used for object detection and classification, and basic object tracking.

In at least one embodiment, any number of stereo camera(s) 1768 may also be included in a forward-facing configuration. In at least one embodiment, one or more of the stereo camera(s) 1768 may include an integrated controller that includes a scalable processing unit that includes programmable logic ("FPGA") and a multi-core microprocessor with an integrated controller area network ("CAN") or Ethernet interface can provide on a single chip. In at least one embodiment, such a unit can be used to generate a 3D environment map of the vehicle 1700, including a distance estimate for all pixels. In at least one embodiment, stereo camera(s) 1768 may include, among other things, one or more compact stereo vision sensor(s) that may include, among other things, two camera lenses (one each left and right) and an image processing chip that measure the distance from the vehicle 1700 to the target and the information generated data (e.g. metadata) to activate autonomous emergency braking and lane departure warning functions. In at least one embodiment, other types of stereo camera(s) 1768 may be used in addition to or as an alternative to those described herein.

In at least one exemplary embodiment, cameras with a field of view that includes parts of the surroundings to the side of the vehicle 1700 (eg, side view cameras) can be used for the all-round view, providing information that is used to create and update the occupancy grid and to generate side impact warnings. collisions are used. For example, in at least one embodiment, surround camera(s) 1774 (e.g., four surround cameras 1774 as in 17B shown) positioned on vehicle 1700. Surround camera(s) 1774 may include, without limitation, any number and combination of wide angle camera(s) 1770, fisheye camera(s), 360 degree camera(s), and/or the like. For example, four fisheye cameras may be located at the front, rear, and sides of vehicle 1700 in at least one embodiment. In at least one embodiment, the vehicle 1700 may use three surround camera(s) 1774 (eg, left, right, and rear) and use one or more other camera(s) (eg, a front-facing camera) as the fourth surround camera.

In at least one embodiment, cameras with a field of view that includes portions of the environment behind the vehicle 1700 (e.g., backup cameras) may be used for parking assist, surround view, rear impact warning, and occupancy grid creation and updating. In at least one embodiment, a variety of cameras may be used, including but not limited to cameras that are also suitable as front-facing camera(s) (e.g., remote camera(s) 1798 and/or mid-range camera(s) 1776, stereo camera(s) 1768 ), infrared camera(s) 1772 etc.) as described here.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are here in connection with 14A and or 14B specified. In at least one embodiment, the inference and/or training logic 1415 in the system 17B to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the robot can be an autonomous vehicle, and the evaluation and generation networks can be implemented using a computer system in the autonomous vehicle. In at least one embodiment, the depth camera used to capture the point cloud of an object to be grasped is a camera, sonar, radar, or lidar in an autonomous vehicle.

17C 17 is a block diagram depicting an example system architecture for the autonomous vehicle 1700 of FIG 17A shows, according to at least one embodiment. In at least one embodiment, all of the components, features, and systems of the vehicle 1700 in 17C shown connected via a bus 1702 . In at least one embodiment, bus 1702 may include, without limitation, a CAN data interface (alternatively referred to herein as "CAN bus"). In at least one embodiment, a CAN may be a network within vehicle 1700 used to support control of various features and functions of vehicle 1700, such as brake application, acceleration, braking, steering, windshield wipers, etc. In at least one embodiment, the bus 1702 can be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). In at least one embodiment, bus 1702 may be read to determine steering wheel angles, vehicle speed, engine revolutions per minute ("RPMs"), button positions, and/or other vehicle status indicators. In at least one embodiment, bus 1702 may be a CAN bus that is ASIL B compliant.

In at least one embodiment, CAN, FlexRay, and/or Ethernet may be used in addition to or as an alternative. In at least one embodiment, there may be any number of buses 1702, including, without limitation, zero or more CAN buses, zero or more FlexRay buses, zero or more Ethernet buses, and/or zero or more other types of buses using any other protocol can include. In at least one embodiment, two or more buses 1702 can be used to perform different functions and/or for redundancy. For example, a first bus 1702 may be used for collision avoidance functionality and a second bus 1702 may be used for actuation control. In at least one embodiment, each bus 1702 can communicate with any component of the vehicle 1700, and two or more buses 1702 can communicate with the same components. In at least one embodiment, any of any number of systems on chip(s) ("SoC(s)") 1704, controller(s) 1736, and/or computer in the vehicle may have access to the same input data (e.g., inputs from sensors of the vehicle 1700 ) and be connected to a common bus, a CAN bus.

In at least one embodiment, the vehicle 1700 may include one or more controllers 1736, such as those referred to herein 17A described. Controller(s) 1736 can be used for a variety of functions. In at least one embodiment, controller(s) 1736 may be coupled to any other vehicle 1700 components and systems and used for vehicle 1700 control, vehicle 1700 artificial intelligence, vehicle 1700 infotainment, and/or similar functions.

In at least one embodiment, the vehicle 1700 may include any number of SoCs 1704 . Each of the SoCs 1704 may include, without limitation, central processing units ("CPU(s)") 1706, graphics processing units ("GPU(s)") 1708, processor(s) 1710, cache(s) 1712, accelerators 1714 , memory 1716, and/or other components and features not shown. In at least one embodiment, SoC(s) 1704 may be used to control vehicle 1700 in a variety of platforms and systems. For example, in at least one embodiment, SoC(s) 1704 may be combined in a system (e.g., vehicle 1700 system) with a high definition (“HD”) map 1722 that supports map updates and/or updates over the network interface 1724 can be obtained from one or more servers (in 17C not shown).

In at least one embodiment, CPU(s) 1706 may comprise a CPU cluster or CPU complex (alternatively referred to herein as "CCPLEX"). In at least one embodiment, the CPU(s) 1706 may include multiple cores and/or level 2 ("L2") caches. For example, in at least one embodiment, the CPU(s) 1706 may include eight cores in a coherent multiprocessor configuration. In at least one embodiment, the CPU(s) 1706 may include four dual-core clusters, with each cluster having a dedicated L2 cache (e.g., a 2MB L2 cache). In at least one embodiment, CPU(s) 1706 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, such that any combination of CPU(s) 1706 clusters may be active at any one time.

In at least one embodiment, one or more of the CPU(s) 1706 may implement power management functions, including without limitation one or more of the following features: Individual hardware blocks may be automatically gated from the clock signal when idle. clock-gated") to dynamically save power; any core clock may be detached when the core is not actively executing commands due to the execution of Wait-For-Interrupt ("WFI")/Wait-For-Event ("WFE") instructions; each core can be independently power gated; each core cluster can be clocked independently if all cores are clocked or power controlled; and/or each core cluster may be independently power controlled if all cores are power controlled. In at least one embodiment, the CPU(s) 1706 may further implement an improved power state management algorithm, specifying the allowable power states and expected wake-up times, and the hardware/microcode determining the best power state to enter the core, cluster, and CCPLEX be able. In at least one embodiment, processing cores may support simplified sequences for entering power states in software, with the work being offloaded to microcode.

In at least one embodiment, GPU(s) 1708 may include an integrated GPU (alternatively referred to herein as “iGPU”). In at least one embodiment, GPU(s) 1708 may be programmable and efficient for parallel workloads. In at least one embodiment, GPU(s) 1708 may use an enhanced tensor instruction set. In at least one embodiment, GPU(s) 1708 may include one or more streaming microprocessors, each streaming microprocessor having an L1 cache (eg, an L1 cache having at least 96 KB of storage capacity) and two or more streaming microprocessors having an L2 cache (e.g. an L2 cache with a storage capacity of 512 KB). In at least one embodiment, GPU(s) 1708 include at least eight streaming microprocessors. In at least one embodiment, GPU(s) 1708 may use Compute Application Programming Interface(s) (API(s)). In at least one embodiment, GPU(s) 1708 may utilize one or more parallel computing platforms and/or programming models (e.g., CUDA from NVIDIA).

In at least one embodiment, one or more of the GPU(s) 1708 may be performance optimized to achieve the best performance in automotive and embedded use cases. For example, in one embodiment, GPU(s) 1708 could be fabricated on a Fin Field Effect Transistor ("FinFET"). In at least one embodiment, each streaming microprocessor may include a number of mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores could be divided into four processing blocks. In at least one embodiment, each processing block could have 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two NVIDIA TENSOR-CORES for mixed-precision deep learning matrix arithmetic, a level-zero ("L0") instruction cache , a warp scheduler, a shipping unit, and/or a 64K register file. In at least one embodiment, streaming microprocessors may include independent parallel integer and floating point data paths to enable efficient execution of workloads with a mix of computational and addressing computations. In at least one embodiment, streaming microprocessors may include independent thread scheduling functions to enable finer grained synchronization and collaboration between parallel threads. In at least one embodiment, streaming microprocessors may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

In at least one embodiment, one or more of the GPU(s) 1708 may include high-bandwidth memory ("HBM") and/or a 16GB HBM2 memory subsystem to provide peak memory bandwidth of about 900GB/second in some examples. In at least one embodiment, synchronous graphics random access memory ("SGRAM"), e.g., double data rate synchronous graphics random access memory type 5 ("GDDR5"), may be used in addition to or as an alternative to HBM memory.

In at least one embodiment, GPU(s) 1708 may include a unified memory technology. In at least one embodiment, Address Translation Services ("ATS") support may be used to allow GPU(s) 1708 direct access to CPU(s) 1706 page tables. In at least one embodiment, an address translation request may be sent to the CPU(s) 1706 when the memory management unit ("MMU") GPU(s) 1708 experiences a miss. In response, the CPU(s) 1706 may look up their page tables for a virtual-to-physical mapping of the address and transmit the translation back to the GPU(s) 1708, according to at least one embodiment. In at least one embodiment, unified memory technology may enable a single unified virtual address space for memory of both CPU(s) 1706 and GPU(s) 1708, making programming of GPU(s) 1708 and porting of applications to the GPU(s) 1708 is simplified.

In at least one embodiment, the GPU(s) 1708 may include any number of access counters that may track the number of times GPU(s) 1708 access the memory of other processors. In at least one embodiment, access counters may help ensure that memory pages are moved to the physical memory of the processor accessing pages most frequently, thereby improving efficiency for processor-shared memory areas.

In at least one embodiment, one or more of the SoC(s) 1704 may include any number of cache(s) 1712, including those described herein. For example, in at least one embodiment, the cache(s) 1712 could comprise a level three (“L3”) cache dedicated to both the CPU(s) 1706 and the GPU(s). (s) 1708 is available (i.e., attached to both CPU(s) 1706 and GPU(s) 1708). In at least one embodiment, cache(s) 1712 may include a write-back cache that may track states of lines, e.g., using a cache coherency protocol (e.g., MEI, MESI, MSI, etc.). In at least one embodiment, the L3 cache may be 4MB or more, depending on the embodiment, although smaller cache sizes may be used.

In at least one embodiment, one or more of the SoC(s) 1704 may include one or more accelerators 1714 (eg, hardware accelerators, software accelerators, or a combination thereof). In at least one embodiment, SoC(s) 1704 may include a hardware acceleration cluster, which may include optimized hardware accelerators and/or large on-chip memory. In at least one embodiment, large on-chip memory (eg, 4MB SRAM) may enable a hardware acceleration cluster to speed up neural networks and other computations. In at least one embodiment, a hardware acceleration cluster using GPU(s) 1708 may be used to complement the graphics processor(s) 1708 and offload some tasks (e.g., by more cycles) from the graphics processor(s) 1708 free the graphics processor(s) 1708 to perform other tasks). In at least one embodiment, the accelerator(s) 1714 could be used for targeted workloads (e.g., perception, convolutional neural networks ("CNNs"), recurrent neural networks ("RNNs"), etc.) that are stable enough to support a acceleration to be accessible. In at least one embodiment, a CNN may include region-based or regional convolutional neural networks ("RCNNs") and fast RCNNs (eg, such as are used for object detection) or other types of CNNs.

In at least one embodiment, the accelerator(s) 1714 (e.g., hardware acceleration cluster) may include a Deep Learning Accelerator(s) ("DLA"). DLA(s) may include, without limitation, one or more tensor processing units ("TPUs"), which may be configured to provide an additional ten trillion operations per second for deep learning applications and inferencing. In at least one embodiment, TPUs can be accelerators configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). DLA(s) can be further optimized for a specific set of neural network types and floating point operations, as well as for inferencing. In at least one embodiment, the design of DLA(s) can provide more performance per millimeter than a typical general purpose GPU and typically far exceeds the performance of a CPU. In at least one embodiment, TPU(s) may perform multiple functions, including a single-instance convolution function that supports, for example, INT8, INT16, and FP16 data types for both features and weights, and post-processor functions. In at least one embodiment, DLA(s) can quickly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any variety of functions including, for example and without limitation: a CNN for object identification and detection using data from camera sensors; a CNN for estimating distances using data from camera sensors; a CNN for emergency vehicle detection and identification and detection using data from microphones 1796; a CNN for facial recognition and vehicle owner identification using data from camera sensors; and/or a CNN for security-related events.

In at least one embodiment, DLA(s) may perform any function of GPU(s) 1708, and using an inference accelerator, for example, a designer may employ either DLA(s) or GPU(s) 1708 for any function. For example, in at least one embodiment, the designer may concentrate the processing of CNNs and floating point operations on DLA(s) and leave other functions to GPU(s) 1708 and/or other accelerator(s) 1714 .

In at least one embodiment, the accelerator(s) 1714 (e.g., hardware acceleration cluster) may include a programmable vision accelerator(s) ("PVA"), which may alternatively be referred to herein as a computer vision accelerator/ be able. In at least one embodiment, the PVA(s) may be designed and configured to use computer vision algorithms for Advanced Driver Assistance System ("ADAS") 1738, Autonomous Driving, Augmented Reality ("AR") ) applications and/or Virtual Reality (“VR”) applications. PVA(s) can provide a balance between performance and flexibility. In at least one embodiment, each PVA may include, for example and without limitation, any number of reduced instruction set ("RISC") cores, direct memory access ("DMA") cores, and/or any number of vector processors.

In at least one embodiment, RISC cores may interact with image sensors (eg, image sensors of any of the cameras described herein), image signal processor(s), and/or the like. In at least one embodiment, each RISC core may include any memory location. In at least one embodiment, RISC cores may use one of several protocols, depending on the embodiment. In at least one embodiment, RISC cores can support a real-time operating system (“RTOS”) run. In at least one embodiment, RISC cores may be implemented using one or more integrated circuit devices, application specific integrated circuits ("ASICs"), and/or memory devices. For example, in at least one embodiment, RISC cores could include an instruction cache and/or tightly coupled RAM.

In at least one embodiment, DMA may allow components of PVA(s) to access system memory independently of CPU(s) 1706. In at least one embodiment, DMA may support any number of functions provided to optimize PVA, including but not limited to supporting multi-dimensional addressing and/or circular addressing. In at least one embodiment, DMA may support up to six or more addressing dimensions, which may include, but are not limited to, block width, block height, block depth, horizontal block gradation, vertical block gradation, and/or depth gradation.

In at least one embodiment, vector processors may be programmable processors that may be designed to efficiently and flexibly perform programming of image processing algorithms and to provide signal processing capabilities. In at least one embodiment, PVA may include a PVA core and two vector processing subsystem partitions. In at least one embodiment, the PVA core may include a processor subsystem, DMA engine(s) (e.g., two DMA engines), and/or other peripherals. In at least one embodiment, the vector processing subsystem may operate as a primary PVA processing unit and may include a vector processing unit ("VPU"), an instruction cache, and/or vector memory (e.g., "VMEM"). In at least one embodiment, the VPU core may include a digital signal processor, such as a Single-Instruction, Multiple Data ("SIMD"), Very Long Instruction Word ("VLIW") digital signal processor. In at least one embodiment, a combination of SIMD and VLIW can improve throughput and speed.

In at least one embodiment, each of the vector processors may include an instruction cache and be coupled to a dedicated memory. As a result, in at least one embodiment, each vector processor can be configured to operate independently of other vector processors. In at least one embodiment, vector processors included in a particular PVA can be configured to use data parallelism. For example, in at least one embodiment, multiple vector processors contained within a single PVA may execute the same computer image rendering algorithm, but in different regions of an image. In at least one embodiment, vector processors included in a given PVA may simultaneously execute different computer vision algorithms on the same image, or even execute different algorithms on sequential images or portions of an image. In at least one embodiment, any number of PVAs may be included in a hardware acceleration cluster and any number of vector processors may be included in each of the PVAs, among others. In at least one embodiment, PVA(s) may include additional ECC (Error Correction Code) memory to improve overall system security.

In at least one embodiment, the accelerator(s) 1714 (e.g., hardware acceleration cluster) may include an on-chip computer vision network and static random-access memory (“SRAM”) to provide high-bandwidth, low-latency SRAM for to provide the accelerator(s) 1714. In at least one embodiment, the on-chip memory may include at least 4 MB of SRAM consisting of, for example and without limitation, eight blocks of field-configurable memory that are accessible by both PVA and DLA. In at least one embodiment, each pair of memory blocks may include an Advanced Peripheral Bus ("APB") interface, configuration circuitry, a controller, and a multiplexer. In at least one embodiment, any type of memory can be used. In at least one embodiment, PVA and DLA can access memory through a framework that allows PVA and DLA high-speed memory access. In at least one embodiment, the backbone may include an on-chip computer vision network that connects PVA and DLA to storage (e.g., using APB).

In at least one embodiment, an on-chip computer vision network may include an interface that determines that both PVA and DLA provide ready and valid signals prior to transmission of control signals/address/data. In at least one embodiment, an interface may provide separate phases and separate channels for control signal/address/data transmission and burst communications for continuous data transmission. In at least one version For example, an interface may conform to International Standardization Organization ("ISO") 26262 or International Electrotechnical Commission ("IEC") 61508 standards, although other standards and protocols may be used.

In at least one embodiment, one or more of the SoC(s) 1704 may include a hardware accelerator for real-time ray-tracing. In at least one embodiment, a hardware accelerator can be used to quickly and efficiently determine positions and extents of objects (e.g., within a real-world model), generate real-time visualization simulations, interpret RADAR signals, synthesize, and/or analyze of sound propagation, for simulating SONAR systems, for a general wave propagation simulation, for comparison with LIDAR data for the purpose of localization and/or other functions and/or for other purposes.

In at least one embodiment, the accelerator(s) 1714 (e.g., hardware accelerator cluster) have a wide range of applications for autonomous driving. In at least one embodiment, PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. In at least one embodiment, the capabilities of PVA are well suited to algorithmic domains that require predictable, low-performance, low-latency processing. In other words, the PVA performs well in semi- or densely regular computations even on small data sets that require predictable, low-latency, low-performance runtimes. In at least one embodiment, PVAs for autonomous vehicles, such as vehicle 1700, are designed to execute classical computer-aided vision algorithms because they are efficient at object recognition and use integer mathematics.

For example, according to at least one embodiment of the technology, PVA is used to perform computer stereo vision. In at least one embodiment, although not intended to be limiting, an algorithm based on semi-global matching may be used in some examples. In at least one embodiment, Level 3-5 autonomous driving applications use on-the-fly motion estimation/stereo matching (e.g., texture from motion, pedestrian detection, lane detection, etc.). In at least one embodiment, the PVA can perform a computer stereo vision function with inputs from two monocular cameras.

In at least one embodiment, PVA may be used to perform dense optical flow. For example, in at least one embodiment, PVA could process raw RADAR data (e.g., using a 4D Fast Fourier Transform) to provide processed RADAR data. In at least one embodiment, PVA is used for runtime depth processing, such as processing raw runtime data to provide processed runtime data.

In at least one embodiment, DLA may be used to operate any type of network to enhance control and driving safety, such as, without limitation, a neural network that outputs a confidence level for each object detection. In at least one embodiment, confidence may be represented or interpreted as a probability or as providing a relative "weight" of each detection compared to other detections. In at least one embodiment, the confidence allows a system to make further decisions about which discoveries should be considered true positive discoveries and not false positive discoveries. For example, in at least one embodiment, a system may set a confidence threshold and only consider discoveries that exceed the threshold to be true positive discoveries. In an embodiment using an automatic emergency braking ("AEB") system, false positive detections would cause the vehicle to automatically perform emergency braking, which is obviously undesirable. In at least one embodiment, very reliable detections can be considered as triggers for AEB. In at least one embodiment, DLA may operate a neural network to regress the confidence value. In at least one embodiment, the neural network may take as input at least a subset of parameters, such as the dimensions of the bounding box, the estimate of floor area obtained (eg, from another subsystem), the output of IMU sensor(s) 1766, the correlated with vehicle 1700 orientation, range, 3D attitude estimates of the object obtained from the neural network and/or other sensors (eg, LIDAR sensor(s) 1764 or RADAR sensor(s) 1760), and others.

In at least one embodiment, one or more of the SoC(s) 1704 may include the data storage(s) 1716 (e.g., memory). In at least one embodiment, data store(s) 1716 may be on-chip memory of SoC(s) 1704 that may store neural networks to be executed on GPU(s) 1708 and/or DLA. In at least one embodiment, data store(s) 1716 may be large enough to store multiple instances of neural networks for redundancy and security. In at least one embodiment, data store(s) 1712 may include L2 or L3 cache(s).

In at least one embodiment, one or more of the SoC(s) 1704 may include any number of processor(s) 1710 (e.g., embedded processors). The processor(s) 1710 may include a boot and power management processor, which may be a dedicated processor and subsystem that performs boot performance and management functions and associated security enforcement. In at least one embodiment, the boot and power management processor may be part of the boot sequence of the SoC(s) 1704 and provide power management services at runtime. In at least one embodiment, the boot power and management processor may provide clock and voltage programming, support for state transitions in the system with lower power consumption, management of thermal and temperature sensors of the SoC(s) 1704, and/or management of the power states of the SoC(s) 1704. In at least one embodiment, each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and SoC(s) 1704 may use ring oscillators to measure CPU(s) 1706, GPU(s) 1708, and/or accelerator(s) temperatures. n) 1714 to record. In at least one embodiment, if temperatures are determined to exceed a threshold, then the boot and power management processor may enter a temperature fault routine and place the SoC(s) 1704 in a lower performance state and/or the vehicle 1700 in a lower power state Place the vehicle in safe stop mode (e.g. place the vehicle 1700 in a safe stop mode).

In at least one embodiment, processor(s) 1710 may further include a set of embedded processors that may serve as an audio processing engine. In at least one embodiment, an audio processing engine may be an audio subsystem that enables full hardware support for multi-channel audio across multiple interfaces and a wide and flexible range of audio I/O interfaces. In at least one embodiment, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM

In at least one embodiment, processor(s) 1710 may further include an always-on processor engine that may provide the necessary hardware features to support management of low-power sensors and wake-up use cases. In at least one embodiment, the always-on processor engine may include, without limitation, a processor core, tightly coupled RAM memory, supporting peripherals (e.g., timers and interrupt controllers), various I/O control peripherals, and routing logic.

In at least one embodiment, the processor(s) 1710 may further include a security cluster engine that includes, without limitation, a dedicated processor subsystem for handling security management for automotive applications. In at least one embodiment, the security cluster engine may include, without limitation, two or more processor cores, tightly coupled RAM memory, supporting peripherals (e.g., timers, an interrupt controller, etc.), and/or routing logic. In a security mode, in at least one embodiment, two or more cores may operate in a lockstep mode and act as a single core with comparison logic to detect differences between their operations. In at least one embodiment, the processor(s) 1710 may further include a real-time camera engine, which may include, among other things, a dedicated processor subsystem for handling real-time camera management. In at least one embodiment, processor(s) 1710 may further include a high dynamic range signal processor, which may include, among other things, an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

In at least one embodiment, processor(s) 1710 may include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to produce the final to generate an image for the playback window. In at least one embodiment, the video image compositor may perform lens distortion correction on the wide-angle camera(s) 1770, the surround camera(s) 1774, and/or the sensor(s) of the surveillance camera(s) in perform in the cabin. In accordance with at least one embodiment, in-cabin surveillance camera sensors are preferably monitored by a neural network running on another instance of SoC 1704 configured to identify events in the cabin and respond accordingly. In at least one embodiment, an in-cabin system may, without limitation, perform lip reading to activate cellular service and make a phone call, dictate emails, change the vehicle's destination, activate or close the vehicle's infotainment system and settings change or provide voice-activated web surfing. In at least one embodiment, certain functions are available to the driver when the vehicle is operating in an autonomous mode and are disabled otherwise.

In at least one embodiment, the video image compositor may include improved temporal denoising for both spatial and temporal denoising. For example, in at least one embodiment where motion occurs in a video, noise reduction may weight spatial information accordingly, thereby reducing the weight of information provided by neighboring frames. In at least one embodiment where an image or portion of an image does not include motion, the temporal noise reduction performed by the video image compositor may use information from the previous image to reduce noise in the current image.

In at least one embodiment, the video image compositor may also be configured to perform stereo equalization on the input stereo lens frames. In at least one embodiment, the video image compositor may be further used for user interface composition when the operating system desktop is used and the GPU(s) 1708 is/are not required to continually render new interfaces. In at least one embodiment, the video image compositor may be used using the video image compositor to offload the graphics processor(s) 1708 when the graphics processor(s) 1708 is/are powered on and actively performing 3D rendering to improve performance and improve responsiveness.

In at least one embodiment, one or more SoC(s) 1704 may further comprise a camera Mobile Industry Processor Interface (MIPI) serial interface for receiving video and input from cameras, a high-speed interface, and/or a video input block configured for camera - and associated pixel input functions can be used. In at least one embodiment, one or more SoC(s) 1704 may further include an input/output controller(s) that may be controlled by software and used to receive I/O signals other than for a specific roles are provided.

In at least one embodiment, one or more of the SoC(s) 1704 may further include a wide range of peripheral interfaces to enable communication with peripherals, audio encoders/decoders ("codecs"), power management, and/or other devices. SoC(s) 1704 can be used to collect data from cameras (e.g., connected via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LIDAR sensor(s) 1764, RADAR sensor(s) 1760, etc. connected via Ethernet may be connected), data from bus 1702 (e.g., vehicle 1700 speed, steering wheel position, etc.), data from GNSS sensor(s) 1758 (e.g., connected via Ethernet or CAN bus), etc. In at least one embodiment, one or more SoC(s) 1704 may further include dedicated high-performance mass storage controllers, which may contain their own DMA engines, and which may be used to offload the CPU(s) 1706 from routine data management tasks.

In at least one embodiment, SoC(s) 1704 may be an end-to-end platform with a flexible architecture that includes automation levels 3-5, thereby providing a comprehensive functional safety architecture that includes computer vision and ADAS techniques for diversity and redundancy, provides a platform for a flexible, reliable drive software stack together with deep learning tools. In at least one embodiment, SoC(s) 1704 may be faster, more reliable, and even more power and space efficient than conventional systems be. For example, in at least one embodiment, the accelerator(s) 1714 in combination with CPU(s) 1706, GPU(s) 1708, and memory(s) 1716 can provide a fast, efficient platform for level 3-5 autonomous vehicles.

In at least one embodiment, computational vision algorithms may execute on CPUs that may be configured using a high level programming language, such as the C programming language, to perform a variety of processing algorithms on a variety of visual data. However, in at least one embodiment, CPUs are often unable to meet the performance requirements of many computer vision applications, such as those related to runtime and power consumption. In at least one embodiment, many CPUs are unable to execute complex real-time object detection algorithms used in in-vehicle ADAS applications and in practical level 3-5 autonomous vehicles.

The exemplary embodiments described here make it possible to execute multiple neural networks simultaneously and/or sequentially and to combine the results with one another to enable level 3-5 autonomous functionality of autonomous driving. For example, in at least one embodiment, a CNN running on DLA or discrete GPU (e.g., GPU(s) 1720) may include text and word recognition so that a supercomputer can read and understand traffic signs, including signs for which the neural network has not been specially trained. In at least one embodiment, DLA may further include a neural network capable of identifying, interpreting, and providing a semantic understanding of characters, and propagating this semantic understanding to path planning engines running on a CPU complex.

In at least one embodiment, multiple neural networks may be operated simultaneously, such as when driving at level 3, 4, or 5. For example, in at least one embodiment, a warning sign reading "Caution: Flashing Light Indicates Icing" along with an electric light from multiple neural networks Networks can be interpreted independently or together. In at least one embodiment, the sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), the text "Flashing lights indicate icing conditions" may be interpreted by a second deployed neural network that understands the vehicle's path planning software ( preferably running on CPU Complex) that icing conditions are present when flashing lights are detected. In at least one embodiment, "turn signals" may be identified through the operation of a third deployed neural network over multiple frames that informs the vehicle's path planning software of the presence (or absence) of turn signals. In at least one embodiment, all three neural networks may operate concurrently, e.g., within DLA and/or on GPU(s) 1708.

In at least one embodiment, a face recognition and vehicle owner identification CNN may use data from camera sensors to identify the presence of an authorized driver and/or vehicle owner of vehicle 1700 . In at least one embodiment, an always-on sensor processing engine may be used to unlock the vehicle when the owner approaches the driver's door and turns on the lights, and to disarm the vehicle in security mode when the owner exits the vehicle. In this manner, the SoC(s) 1704 provide security against theft and/or violent car hijacking.

In at least one embodiment, the CNN for emergency vehicle detection and identification may use the data from emergency vehicle siren detection and identification microphones 1796 . In at least one embodiment, the SoC(s) 1704 use a CNN to classify environmental and city sounds, as well as to classify visual data. In at least one embodiment, the CNN running on DLA is trained to identify the relative closing speed of the emergency vehicle (eg, using the Doppler effect). In at least one embodiment, the CNN can also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by GNSS sensor(s) 1758 . In at least one embodiment, when operating in Europe, CNN will attempt to identify European sirens and in the United States, CNN will attempt to identify only North American sirens. In at least one embodiment, once an emergency vehicle is detected, a control program using ultrasonic sensor(s) 1762 can be used to execute an emergency vehicle safety routine, brake the vehicle, to pull to one side, park the vehicle and/or keep the vehicle idling until the emergency vehicle(s) passes(s) passing.

In at least one embodiment, vehicle 1700 may include CPU(s) 1718 (e.g., discrete CPU(s) or dCPU(s)) that may be coupled to SoC(s) 1704 via a high-speed interconnect (e.g., PCIe). . For example, in at least one embodiment, the CPU(s) 1718 may comprise an X86 processor. CPU(s) 1718 may be deployed using any functionality, including arbitrating potentially inconsistent results between ADAS sensors and SoC(s) 1704 and/or monitoring status and condition of controller(s) 1736 and/or a Infotainment system on a chip (“Infotainment SoC”) 1730, for example.

In at least one embodiment, vehicle 1700 may include GPU(s) 1720 (e.g., discrete GPU(s) or dGPU(s)) coupled to SoC(s) 1704 via a high-speed interconnect (e.g., NVIDIA's NVLINK). be able. In at least one embodiment, GPU(s) 1720 may provide additional artificial intelligence functionality, e.g., by running redundant and/or different neural networks, and may be used to train and/or update neural networks based at least in part on inputs (e.g., sensor data). used by vehicle 1700 sensors.

In at least one embodiment, the vehicle 1700 may further include a network interface 1724, which may include, among other things, the wireless antenna(s) 1726 (e.g., one or more wireless antennas 1726 for various communication protocols, such as a cellular antenna, a Bluetooth antenna etc.). In at least one embodiment, network interface 1724 may be used to provide wireless connectivity over the Internet to a cloud (e.g., to server(s) and/or other network devices), to other vehicles, and/or to computing devices (e.g., passenger client devices). ) to allow. In at least one embodiment, vehicle 170 may connect directly to other vehicles and/or connect indirectly (e.g., via networks and via the Internet) to communicate with other vehicles. In at least one embodiment, direct connections may be provided using a vehicle-to-vehicle communication link. A vehicle-to-vehicle communication link may provide vehicle 1700 with information about vehicles in the vicinity of vehicle 1700 (e.g., vehicles in front, to the side, and/or behind of vehicle 1700). In at least one embodiment, the foregoing functionality may be part of a cooperative adaptive cruise control functionality of vehicle 1700 .

In at least one embodiment, network interface 1724 may include an SoC that provides modulation and demodulation functionality and enables controller(s) 1736 to communicate over wireless networks. In at least one embodiment, the network interface 1724 may include a radio frequency front end for baseband to radio frequency upconversion and radio frequency to baseband downconversion. In at least one embodiment, frequency translations can be performed in any technically suitable manner. For example, frequency conversions can be performed by known methods and/or using super-heterodyne methods. In at least one embodiment, the radio frequency front end functionality may be provided by a separate chip. In at least one embodiment, the network interface may include wireless functionality for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and/or other wireless data transmission protocols.

In at least one embodiment, the vehicle 1700 may further include data storage(s) 1728, which may include, without limitation, off-chip memory (e.g., off-SoC(s) 1704). In at least one embodiment, the data memory(s) 1728 may include, without limitation, one or more memory elements including RAM, SRAM, dynamic random-access memory ("DRAM"), video RAM (video random- access memory ("VRAM"), flash, hard drives, and/or other components and/or devices capable of storing at least one bit of data.

In at least one embodiment, vehicle 1700 may further include GNSS sensor(s) 1758 (eg, GPS and/or assisted GPS sensors) to support mapping, perception, mapping, and/or path planning functions. In at least one embodiment, any number of GNSS sensor(s) 1758 may be used, including but not limited to one limitation of a GPS using a USB port with an ethernet-to-serial bridge (e.g. RS-232).

In at least one embodiment, the vehicle 1700 may further include RADAR sensor(s) 1760 . The RADAR sensor(s) 1760 may be used by the vehicle 1700 for long range vehicle detection, including in darkness and/or poor weather conditions. In at least one embodiment, the RADAR functionality assurance levels may be ASIL B. RADAR sensor(s) 1760 may use CAN and/or bus 1702 (e.g., to transmit data generated by RADAR sensor(s) 1760) for control and access object tracking data, with access to Ethernet to in some examples access raw data. In at least one embodiment, a variety of RADAR sensor types may be used. For example and without limitation, RADAR sensor(s) 1760 may be suitable for use of front, rear, and side RADAR. In at least one embodiment, one or more RADAR sensors 1760 are pulse Doppler RADAR sensors.

In at least one embodiment, the RADAR sensor(s) 1760 may include various configurations, such as long range with a narrow field of view, short range with a wide field of view, short range side coverage, etc. In at least one embodiment, the RADAR may over large distances are used for adaptive cruise control functionality. In at least one embodiment, long range RADAR systems can provide a wide field of view realized by two or more independent scans, e.g., within a 250 m range. In at least one embodiment, the RADAR sensor(s) 1760 can help distinguish between static and moving objects and can be used by the ADAS system 1738 for emergency brake assist and forward collision warning. Sensors 1760(s) included in a long-range RADAR system may include, without limitation, monostatic multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and a high-speed CAN and FlexRay interface. In at least one six-antenna embodiment, four central antennas can produce a focused radiation pattern designed to detect the surroundings of vehicles 1700 at higher speeds with minimal interference from traffic in adjacent lanes. In at least one embodiment, the other two antennas can expand the field of view so that vehicles entering or leaving the lane of vehicle 1700 can be quickly detected.

In at least one embodiment, mid-range RADAR systems may include, as an example, a range of up to 160 m (front) or 80 m (back) and a field of view of up to 42 degrees (front) or 150 degrees (back). In at least one embodiment, short-range RADAR systems may include, without limitation, any number of RADAR sensor(s) 1760 intended for installation on either end of the rear bumper. In at least one embodiment, when a RADAR sensor system is installed at both ends of the rear bumper, it can generate two beams that continuously monitor the blind spot in the rear and adjacent areas of the vehicle. In at least one embodiment, short-range RADAR systems may be used in ADAS system 1738 for blind spot detection and/or lane change assistance.

In at least one embodiment, the vehicle 1700 may further include ultrasonic sensor(s) 1762 . Ultrasonic sensor(s) 1762, which may be attached to the front, rear, and/or side of vehicle 1700, may be used for parking assistance and/or to create and update a population grid. In at least one embodiment, a wide variety of ultrasonic sensor(s) 1762 may be used, and different ultrasonic sensor(s) 1762 may be used for different detection ranges (e.g., 2.5m, 4m). In at least one embodiment, the ultrasonic sensor(s) 1762 may operate at ASIL B functional safety levels.

In at least one embodiment, the vehicle 1700 may include LIDAR sensor(s) 1764 . The LIDAR sensor(s) 1764 may be used for object and pedestrian detection, emergency braking, collision avoidance, and/or other functions. In at least one embodiment, LIDAR sensor(s) 1764 may be ASIL B functional safety level. In at least one embodiment, vehicle 1700 may include multiple LIDAR sensors 1764 (e.g., two, four, six, etc.) that may be deployed using Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

In at least one embodiment, LIDAR sensor(s) 1764 may provide a list of objects and their distances for a 360 degree field of view. In at least one embodiment, commercially available LIDAR sensor(s) 1764 may have a specified range of approximately 100 m with an accuracy of 2 cm - 3 cm and with support for a 100 Mbps Ethernet connection, for example. In at least one embodiment, one or more non-protruding LIDAR sensors 1764 may be used. In such an embodiment, the LIDAR sensor(s) 1764 may be implemented as a small device that may be embedded in the front, rear, sides, and/or corners of the vehicle 1700 . In at least one embodiment, LIDAR sensor(s) 1764 in such an implementation can provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees, with a range of 200 m even for objects with low reflectivity. In at least one embodiment, a front-mount LIDAR sensor(s) 1764 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

In at least one embodiment, lidar technologies such as 3D flash lidar may also be used. 3D Flash LIDAR uses a laser flash as a transmission source to illuminate the area around the vehicle 1700 up to a distance of approximately 200 m. In at least one embodiment, without limitation, a flash LIDAR unit includes a receptor that records the laser pulse travel time and the reflected light on each pixel, which in turn corresponds to the range from the vehicle 1700 to objects. In at least one embodiment, flash LIDAR can produce highly accurate and distortion-free images of the environment with each laser flash. In at least one embodiment, four flash LIDAR sensors may be deployed, one on each side of the vehicle 1700. In at least one embodiment, 3D flash LIDAR systems include, without limitation, a solid-state 3D flash LIDAR camera with a fixed beam Array with no moving parts other than a fan (e.g. a non-scanning LIDAR device). In at least one embodiment, a 3D flash LIDAR device may use a 5 nanosecond Class I (eye safe) laser pulse per frame and collect reflected laser light in the form of 3D range point clouds and co-registered intensity data.

In at least one embodiment, the vehicle may further include IMU sensor(s) 1766 . In at least one embodiment, IMU sensor(s) 1766 may be located at the center of the rear axle of vehicle 1700 in at least one embodiment. In at least one embodiment, the IMU sensor(s) 1766 may include, for example and without limitation, accelerometers, magnetometers, gyroscope(s), magnetic compass(es), and/or other types of sensors. In at least one embodiment, such as in six-axis applications, the IMU sensor(s) 1766 may include, without limitation, accelerometers and gyroscopes. In at least one embodiment, such as For example, in nine-axis applications, the IMU sensor(s) 1766 may include, without limitation, accelerometers, gyroscopes, and magnetometers.

In at least one embodiment, the IMU sensor(s) 1766 may be implemented as a miniature High Performance GPS-Aided Inertial Navigation System (GPS/INS) that incorporates Micro-Electro-Mechanical Systems (MEMS) inertial sensors. Electro-Mechanical Systems), a highly sensitive GPS receiver and advanced Kalman filtering algorithms to estimate position, velocity and attitude. In at least one embodiment, the IMU sensor(s) 1766 may enable the vehicle 1700 to estimate heading without requiring input from a magnetic sensor by using GPS to IMU sensor speed changes ( en) 1766 can be directly observed and correlated. In at least one embodiment, the IMU sensor(s) 1766 and the GNSS sensor(s) 1758 may be combined into a single integrated unit.

In at least one embodiment, the vehicle 1700 may include one or more microphones 1796 mounted in and/or around the vehicle 1700 . In at least one embodiment, the microphone(s) 1796 may be used for, among other things, emergency vehicle detection and identification.

In at least one embodiment, vehicle 1700 may further include any number of camera types, including stereo camera(s) 1768, wide-angle camera(s) 1770, infrared camera(s) 1772, surround camera(s) 1774, far camera(s) 1798, mid-range camera (s) 1776 and/or other camera types. In at least one embodiment, cameras for capturing image data around the entire periphery of the vehicle 1700 around can be used. In at least one embodiment, the types of cameras used depend on the vehicle 1700 . In at least one embodiment, any combination of camera types may be used to provide the required coverage around vehicle 1700 . In at least one embodiment, the number of cameras used can vary depending on the embodiment. For example, in at least one embodiment, vehicle 1700 could include six cameras, seven cameras, ten cameras, twelve cameras, or another number of cameras. For example and without limitation, the cameras may support Gigabit Multimedia Serial Link ("GMSL") and/or Gigabit Ethernet. In at least one embodiment, each of the cameras is provided with more details regarding 17A and 17B described.

In at least one embodiment, the vehicle 1700 may further include vibration sensor(s) 1742 . The vibration sensor(s) 1742 can measure vibrations of components of the vehicle 1700, such as axle(s). For example, in at least one embodiment, changes in vibration may indicate a change in road surfaces. In at least one embodiment, when two or more vibration sensors 1742 are used, differences in vibration can be used to determine friction or slippage of the road surface (e.g., when there is a difference in vibration between a driven axle and a freely rotating axle).

In at least one embodiment, the vehicle 1700 may include an ADAS system 1738 . ADAS system 1738 may include a SoC in some examples without limitation. In at least one embodiment, ADAS system 1738 may include, without limitation, any number and combination of an autonomous/adaptive/automatic cruise control ("ACC") system, a cooperative adaptive cruise control ("CACC") system, a front collision warning ("FCW") system, and an automatic Emergency Braking (“AEB”) systems include, a Lane Departure Warning (“LDW”) system, a Lane Keeping Assist (“LKA”) system, a Blind Spot Warning (“BSW”) system, a Rear Cross Traffic Warning ("RCTW") system, a Collision Warning ("CW") system, a Lane Centering ("LC") system and/or other systems, features and/or functions.

In at least one embodiment, the ACC system may use RADAR sensor(s) 1760, LIDAR sensor(s) 1764, and/or any number of camera(s). In at least one embodiment, the ACC system may include a longitudinal ACC system and/or a lateral ACC system. In at least one embodiment, the longitudinal ACC system monitors and controls the distance to the vehicle immediately ahead of the vehicle 1700 and automatically adjusts the speed of the vehicle 1700 to maintain a safe distance from vehicles ahead. In at least one embodiment, the side ACC system maintains the distance and advises the vehicle 1700 to change lanes when necessary. In at least one embodiment, side ACC is related to other ADAS applications such as LC and CW.

In at least one embodiment, the CACC system uses information from other vehicles received via network interface 1724 and/or wireless antenna(s) 1726 from other vehicles over a wireless connection or indirectly over a network connection (e.g., over the Internet). can be received. In at least one embodiment, direct connections may be provided through a vehicle-to-vehicle ("V2V") communication link, while indirect connections may be provided through an infrastructure-to-vehicle ("I2V") communication link. In general, the V2V communication concept provides information about immediately preceding vehicles (e.g., vehicles immediately ahead and in the same lane as vehicle 1700), while the I2V communication concept provides information about traffic further away. In at least one embodiment, the CACC system may include either or both of information sources 12V and V2V. In at least one embodiment, the CACC system may be more reliable in information about vehicles ahead of vehicle 1700 and has the potential to improve traffic flow and reduce road congestion.

In at least one embodiment, the FCW system is designed to alert the driver to a hazard so that corrective action can be taken. In at least one embodiment, the FCW system uses a forward-looking camera and/or one or more RADAR sensor(s) 1760 coupled to a dedicated processor, DSP, FPGA, and/or ASIC that is electrically coupled to the feedback of the driver, such as a display, speaker and/or a vibrating component. In at least one embodiment, the FCW system may provide a warning, eg, in the form of a tone, a visual warning, a vibration, and/or a rapid brake pulse.

In at least one embodiment, the AEB system detects an imminent forward collision with another vehicle or object and may automatically apply the brakes if the driver fails to take corrective action within a specified time or distance parameter. In at least one embodiment, the AEB system may utilize one or more forward-looking cameras and/or one or more RADAR sensors 1760 coupled to a dedicated processor, DSP, FPGA, and/or ASIC. In at least one embodiment, when the AEB system detects a hazard, the AEB system typically first alerts the driver to take corrective action to avoid a collision, and if the driver fails to take corrective action, the AEB system may automatically brake operate to prevent or at least mitigate the effects of the predicted collision. In at least one embodiment, the AEB system may include techniques such as dynamic assist braking and/or imminent collision braking.

In at least one embodiment, the LDW system is equipped with visual, audible, and/or tactile alerts such as steering wheel or seat vibrations to alert the driver when the vehicle 1700 is crossing lane markings. In at least one embodiment, the LDW system is not activated when the driver indicates an intention to depart from the lane by actuating a turn signal. In at least one embodiment, the LDW system may utilize forward-looking cameras coupled to a dedicated processor, DSP, FPGA, and/or ASIC that is electrically coupled to driver feedback, such as a display, speaker, and/or or a vibration component. In at least one embodiment, the LKA system is a variant of the LDW system. The LKA system is for inputting steering or braking inputs to correct the vehicle 1700 when the vehicle 1700 begins to drift out of lane.

In at least one embodiment, the BSW system detects and alerts the driver to vehicles in an automobile's blind spot. In at least one embodiment, the BSW system may provide a visual, audible, and/or tactile alert to indicate that collapsing or lane changing is unsafe. In at least one embodiment, the BSW system may provide an additional warning when the driver uses a turn signal. In at least one embodiment, the BSW system may utilize rear-facing camera(s) and/or RADAR sensor(s) 1760 coupled to a dedicated processor, DSP, FPGA, and/or ASIC that is electrically coupled to the driver's coupled to feedback, such as a display, speaker, and/or vibration component.

In at least one embodiment, the RCTW system may provide a visual, audible, and/or tactile notification when an object is detected out of range of the backup camera while vehicle 1700 is backing up. In at least one embodiment, the RCTW system includes an AEB system to ensure vehicle brakes are applied to avoid a collision. In at least one embodiment, an RCTW system may utilize one or more rear-facing RADAR sensors 1760 coupled to a dedicated processor, DSP, FPGA, and/or ASIC that is/are electrically coupled to the driver feedback, e.g. B. a display, a speaker and / or a vibrating component.

In at least one embodiment, conventional ADAS systems can be prone to false positives, which can be annoying and distracting to the driver, but are typically not catastrophic because conventional ADAS systems warn the driver and provide the opportunity to decide whether a security condition actually exists and act accordingly. In at least one embodiment, if there are conflicting results, the vehicle 1700 makes its own decision as to whether the result from a primary or a secondary computer (eg, first controller 1736 or second controller 1736) should be considered. For example, in at least one embodiment, ADAS system 1738 may be a backup and/or secondary computer that provides perceptual information to a rationality module of a backup computer. In at least one exemplary embodiment, redundant diverse software can run on hardware components on the rationality monitor of the reserve computer in order to detect perception errors and dynamic driving tasks. In at least one embodiment, ADAS system 1738 outputs may be provided to a higher-level MCU. In at least one embodiment, if output from the primary computer and secondary com computers are in conflict, the higher-level MCU determines how to resolve the conflict to ensure safe operation.

In at least one embodiment, the primary computer may be configured to provide the high-level MCU with a confidence score indicative of the primary computer's confidence in the selected outcome. In at least one embodiment, if the confidence exceeds a threshold, the master MCU may follow the direction of the primary computer, regardless of whether the secondary computer provides a conflicting or inconsistent result. In at least one embodiment where the confidence does not meet a threshold and where the primary and secondary computers display different results (e.g., a conflict), the master MCU may mediate between the computers to determine an appropriate result.

In at least one embodiment, the high-level MCU may be configured to operate a neural network or networks that are trained and configured to, based at least in part on outputs from the primary computer and Secondary computer, determines conditions under which the secondary computer will give false alarms. In at least one embodiment, the neural network(s) in the high-level MCU can learn when the output of the secondary computer can and cannot be trusted. For example, in at least one embodiment, if the secondary computer is a RADAR-based FCW system, the neural network(s) in the higher-level MCU can learn when the FCW system identifies metallic objects that it is in reality is not about hazards, such as a drainage grate or a manhole cover that sets off an alarm. In at least one embodiment, when the secondary computer is a camera-based LDW system, a neural network in the higher-level MCU can learn to override the LDW when cyclists or pedestrians are present and lane departure is actually the safest maneuver. In at least one embodiment, the high-level MCU may include at least one DLA or GPU unit capable of operating one or more neural networks with associated memory. In at least one embodiment, the control MCU may comprise and/or be included as a component of SoC(s) 1704 .

In at least one embodiment, ADAS system 1738 may include a secondary computer that executes ADAS functionality using traditional computer-aided vision rules. In at least one embodiment, the secondary computer may operate using classic computer-aided vision (if-then) rules, and having one or more neural networks in the parent MCU may improve reliability, security, and performance . For example, in at least one embodiment, differential implementation and intentional non-identity make the overall system more fault-tolerant, particularly to faults caused by software (or software-hardware interface) functionality. For example, if in at least one embodiment there is a software error or bug in the software running on the primary computer and non-identical software code on the secondary computer produces the same overall result, then the higher-level MCU may have greater confidence that the overall result is correct and the error in does not cause a material error in the software or hardware of the primary computer.

In at least one embodiment, the output of the ADAS system 1738 may be fed into the perceptual block of the host computer and/or the dynamic driving task block of the host computer. For example, in at least one embodiment, if the ADAS system 1738 is displaying a forward crash warning due to an object immediately ahead, the perception block may use that information in identifying objects using that information. In at least one embodiment, the secondary computer may have its own neural network that is trained to reduce the risk of false positives as described herein.

In at least one embodiment, the vehicle 1700 may further include an infotainment SoC 1730 (eg, an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, in at least one embodiment, infotainment system 1730 may not be an SoC and may include, without limitation, two or more discrete components. In at least one embodiment, the infotainment SoC 1730 may include, without limitation, a combination of hardware and software configured to provide audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., television, movies, streaming etc.) Telephone (e.g. hands-free calls), network connection (e.g. LTE, WiFi, etc.) and/or information services (e.g. navigation systems, parking assistance, a radio data system, vehicle-related information such as fuel level, total distance traveled, brake fluid level, oil level, door open/closed, air filter information, etc.) for Vehicle 1700. Infotainment SoC 1730 could include, for example, radios, turntables, navigation systems, video players, USB and Bluetooth connectivity, carputers, in-car entertainment, WiFi, steering wheel audio controls, hands-free voice control, a heads-up display ("HUD '), HMI display 1734, a telematics device, a control panel (e.g., for controlling and/or interacting with various components, functions, and/or systems), and/or other components. In at least one embodiment, infotainment SoC 1730 may be further used to provide information (e.g., visual and/or audible) to the vehicle user(s), such as information from ADAS system 1738, autonomous driving information such as planned vehicle maneuvers, trajectories, information about the environment (eg, intersection information, vehicle information, road information, etc.) and/or other information.

In at least one embodiment, the infotainment SoC 1730 may include any amount and type of GPU functionality. In at least one embodiment, the infotainment SoC 1730 may communicate with other devices, systems, and/or components of the vehicle 1700 via the bus 1702 (e.g., CAN bus, Ethernet, etc.). In at least one embodiment, the infotainment SoC 1730 can be coupled to a higher-level MCU so that the infotainment system's GPU can perform some self-driving functions if the primary controller(s) 1736 (e.g., primary and/or backup -Computer of vehicle 1700) fail. In at least one embodiment, the infotainment system SoC 1730 may place the vehicle 1700 in a chaffeur to safe stop mode, as described herein.

In at least one embodiment, the vehicle 1700 may further include a center display device 1732 (e.g., a digital dashboard, an electronic instrument cluster, a digital instrument panel, etc.). Central display device 1732 may include, without limitation, a controller and/or supercomputer (e.g., a discrete controller or supercomputer). In at least one embodiment, central display device 1732 may include, without limitation, any number and combination of gauges such as speedometer, fuel level, oil pressure, tachometer, odometer, turn signals, gear shift indicator, seat belt warning light(s), parking brake warning light(s), engine trouble light(s), information about supplemental restraint systems (e.g. airbag), lighting controls, security system controls, navigation information, etc. In some examples, information may be displayed and/or shared between infotainment SoC 1730 and central display device 1732 . In at least one embodiment, center display device 1732 may be included as part of infotainment SoC 1730 or vice versa.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are here in connection with 14A and or 14B provided. In at least one embodiment, the inference and/or training logic 1415 in the system 17C to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

At least one embodiment can be constructed using the techniques described above. In at least one embodiment, the robot may be an autonomous vehicle, and the evaluation and generation networks may be implemented using a computer system in the autonomous vehicle.

17D 1776 is a diagram of a system 1776 for communication between the cloud-based server(s) and the autonomous vehicle 1700 of FIG 17A , according to at least one embodiment. In at least one embodiment, system 1776 may include, without limitation, server 1778, network(s) 1790, and any number and type of vehicles, including vehicle 1700. The server(s) 1778 may include, without limitation, a variety of GPUs 1784(A)-1784(H) (collectively referred to herein as GPUs 1784), PCIe switches 1782(A)-1782(H) (collectively referred to herein as PCIe switches 1782) and/or CPUs 1780(A)-1780(B) (collectively referred to herein as CPUs 1780). 1784 GPUs, 1780 CPUs and 1782 PCIe switches can be used with high-speed connections such as, without limitation, NVLink interfaces developed by NVIDIA 1788 and/or PCIe connections 1786 are connected to each other. In at least one embodiment, the GPUs 1784 are interconnected via an NVLink and/or NVSwitch SoC, and the GPUs 1784 and PCIe switches 1782 are interconnected via PCIe connections. While eight GPUs 1784, two CPUs 1780, and four PCIe switches 1782 are shown in at least one embodiment, this is not intended to be limiting. In at least one embodiment, each of the servers 1778 may include any number of GPUs 1784, CPUs 1780, and/or PCIe switches 1782 in any combination, without limitation. For example, in at least one embodiment, each server 1778 could include eight, sixteen, thirty-two, and/or more GPUs 1784 .

In at least one embodiment, the server(s) 1778 may receive, via the network(s) 1790 and from vehicles, image data representing images showing unexpected or changed road conditions, such as recently started roadworks. In at least one embodiment, the server(s) 1778 may, via the network(s) 1790 and to the vehicles, provide neural networks 1792, updated neural networks 1792, and/or map information 1794 including, without limitation, information about traffic and road conditions , transfer. In at least one embodiment, updates to the map information 1794 may include, but are not limited to, updates to an HD map 1722, e.g., information about roadworks, potholes, detours, flooding, and/or other obstacles. In at least one embodiment, neural networks 1792, updated neural networks 1792, and/or map information 1794 may result from new training and/or new experiences represented in data received from any number of surrounding vehicles and/or based at least in part on training performed at a data center (e.g., using server(s) 1778 and/or other servers).

In at least one embodiment, server(s) 1778 may be used to train machine learning models (e.g., neural networks) based at least in part on training data. Training data can be generated by vehicles and/or in a simulation (e.g. using the engine). In at least one embodiment, any amount of training data is tagged (e.g., if the associated neural network benefits from supervised learning) and/or undergoes other pre-processing. In at least one embodiment, any set of training data is not tagged and/or pre-processed (e.g., when the associated neural network does not require supervised learning). In at least one embodiment, once machine learning models are trained, they may be used by vehicles (e.g., transmitted to vehicles over network(s) 1790 and/or machine learning models may be used by server(s) 1778 to remotely monitor vehicles.

In at least one embodiment, server(s) 1778 may receive data from vehicles and apply data to live neural networks to obtain intelligent inference in real-time. In at least one embodiment, the server(s) 1778 may include deep learning supercomputers and/or dedicated AI computers with GPU(s) 1784, such as the DGX and DGX Station machines developed by NVIDIA. However, in at least one embodiment, server(s) 1778 may include a deep learning infrastructure using CPU-based data centers.

In at least one embodiment, the server(s) 1778 deep learning infrastructure may provide fast, real-time inference services and utilize this capability to assess and verify the health of processors, software, and/or associated hardware in the vehicle 1700 . For example, in at least one embodiment, the deep learning infrastructure may receive periodic updates from vehicle 1700, such as a sequence of images and/or objects that vehicle 1700 located in that sequence of images (e.g., through computer-aided vision and/or other classification techniques for machine learning objects). In at least one embodiment, the deep learning infrastructure can run its own neural network to identify objects and compare them to the objects identified by vehicle 1700 . If the results do not match and the deep learning infrastructure concludes that the AI in vehicle 1700 is faulty, then server(s) 1778 may send a signal to vehicle 1700 and instruct a vehicle 1700 failsafe computer to take control, notify passengers and perform safe parking manoeuvres.

In at least one embodiment, server(s) 1778 may include graphics processor(s) 1784 and one or more programmable inference accelerators (eg, NVIDIA's TensorRT 3). In at least one embodiment, the combination of GPU-powered Enable real-time responsiveness for servers and inference acceleration. In at least one embodiment, eg where performance is less critical, servers powered by CPUs, FPGAs and other processors may be used for inference purposes. In at least one embodiment, the hardware structure(s) 1415 are used to perform one or more embodiments. Details of the hardware structure(x) 1415 are here in connection with 14A and or 14B provided.

18 18 is a block diagram showing an example computer system, which may be a system with interconnected devices and components, a system-on-a-chip (SOC), or a combination thereof 1800 formed with a processor. which may comprise execution units for executing an instruction, according to at least one embodiment. In at least one embodiment, the computer system 1800 may include, without limitation, a component such as a processor 1802 for using execution units including logic to perform algorithms on process data consistent with the present disclosure, such as in an embodiment described herein. In at least one embodiment, computer system 1800 may include processors such as the PENTIUM® processor family, Xeon™, Itanium®, XScale™, and/or StrongARM™, Intel® Core™, or Intel® Nervana™ microprocessors manufactured by Intel Corporation of Santa Clara, California, although other systems (including personal computers with other microprocessors, engineering workstations, set-top boxes, and the like) may be used. In at least one embodiment, computer system 1800 may run a version of the WINDOWS operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (eg, UNIX and Linux), embedded software, and/or graphical user interfaces may also be used .

Example embodiments can be used in other devices such as handheld devices and embedded applications. Some examples of handheld devices include cell phones, Internet protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld personal computers. In at least one embodiment, embedded applications may include a microcontroller, digital signal processor ("DSP"), system on a chip, network computers ("NetPCs"), set top boxes, network hubs, wide area network ("WAN") include switches or any other system capable of executing one or more instructions, according to at least one embodiment.

In at least one embodiment, the computer system 1800 may include, among other things, a processor 1802, which may include, among other things, one or more execution units 1808 for training and/or inferring machine learning models according to the techniques described herein. In at least one embodiment, system 18 is a single processor desktop or server system, but in another embodiment system 18 may be a multiprocessor system. In at least one embodiment, the processor 1802 may include, without limitation, a complex instruction set computer (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (CISC) microprocessor . very long instruction word, VLIW), a processor implementing a combination of instruction sets, or any other processing device such as a digital signal processor. In at least one embodiment, processor 1802 may be coupled to a processor bus 1810 that may transfer data signals between processor 1802 and other components in computer system 1800.

In at least one embodiment, processor 1802 may include an internal level 1 ("L1") cache memory ("cache") 1804 without limitation. In at least one embodiment, processor 1802 may have a single internal cache or multiple levels of internal cache memory. In at least one embodiment, the cache memory may be external to processor 1802. Other embodiments may also include a combination of internal and external caches, depending on the particular implementation and need. In at least one embodiment, register file 1806 may store various types of data in various registers including, without limitation, integer registers, floating point registers, status registers, and instruction pointer registers.

In at least one embodiment, execution unit 1808, which includes, among other things, logic for performing integer and floating point operations, also resides in processor 1802. Processor 1802 may also include microcode ("ucode") read only memory ("ROM"). ) that stores microcode for specific macroinstructions. In at least one embodiment, the execution Unit 1808 include logic for handling a packed instruction set 1809. In at least one embodiment, by including the packed instruction set 1809 in the instruction set of a general purpose processor 1802, along with associated circuitry for executing instructions, operations used by many multimedia applications can be performed using packed data in a general purpose processor 1802. In one or more embodiments, many multimedia applications may run faster and more efficiently by utilizing the full width of a processor's data bus to perform operations on packed data, thereby eliminating the need to transfer smaller chunks of data across the processor's data bus to transfer to perform one or more operations one data item at a time.

In at least one embodiment, execution unit 1808 may also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits. In at least one embodiment, computer system 1800 may include memory 1820 without limitation. In at least one embodiment, memory 1820 may be implemented as a dynamic random access memory (“DRAM”) device, static random access memory (“SRAM”) device, flash memory device, or other storage device. Memory 1820 may store instruction(s) 1819 and/or data 1821 represented by data signals executable by processor 1802.

In at least one embodiment, the system logic chip may be coupled to processor bus 1810 and memory 1820. In at least one embodiment, the system logic chip may include, without limitation, a memory controller hub ("MCH") 1816 and the processor 1802 may communicate with the MCH 1816 via the processor bus 1810 . In at least one embodiment, MCH 1816 may provide a high-bandwidth memory path 1818 to memory 1820 for command and data storage and for storage of graphics commands, data, and textures. In at least one embodiment, MCH 1816 may route data signals between processor 1802, memory 1820, and other components in computer system 1800, and bridge data signals between processor bus 1810, memory 1820, and a system I/O 1822. In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, MCH 1816 may be coupled to MCH 1816 via a high-speed memory path 1818 to memory 1820 and graphics/video card 1812 via an Accelerated Graphics Port ("AGP") interconnect 1814 .

In at least one embodiment, computer system 1800 may use system I/O 1822, which is a proprietary hub interface bus, to couple MCH 1816 to I/O controller hub ("ICH") 1830. In at least one embodiment, ICH 1830 may provide direct connections to some I/O devices via a local I/O bus. In at least one embodiment, the local I/O bus may include, without limitation, a high speed I/O bus for connecting peripheral devices to the memory 1820, chipset, and processor 1802. Examples may include, without limitation, an audio controller 1829, a firmware hub ("flash BIOS") 1828, a wireless transceiver 1826, a data store 1824, a legacy I/O controller 1823 containing user input and keyboard interfaces serial expansion port 1827, such as Universal Serial Bus ("USB"), and a network controller 1834. Data storage 1824 may include a hard drive, floppy disk drive, CD-ROM device, flash memory device, or other mass storage device.

According to at least one embodiment 18 a system that includes interconnected hardware devices or "chips" while 18 in other embodiments, an example system-on-a-chip ("SoC") may show. In at least one embodiment, the 18 Devices shown may be connected using proprietary connections, standardized connections (e.g. PCIe) or a combination thereof. In at least one embodiment, one or more components of system 1800 are interconnected by Compute Express Link (CXL) connections.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, the inference and/or training logic 1415 in the system 18 for inferring or predicting operations based, at least in part, on weight parameters generated using training operations with neural networks, functions and/or architectures ronal networks or use cases of neural networks described herein can be used.

19 19 is a block diagram depicting an electronic device 1900 for use with a processor 1910, according to at least one embodiment. In at least one embodiment, electronic device 1900 may include, for example and without limitation, a notebook, tower server, rack server, blade server, laptop, desktop, tablet, mobile device, phone, embedded computer or any other suitable electronic device.

In at least one embodiment, the system 1900 may include, without limitation, a processor 1910 communicatively coupled to any suitable number or type of components, peripherals, modules, or devices. In at least one embodiment, the processor 1910 is coupled via a bus or interface, such as a 1°C bus, a System Management Bus ("SMBus"), a Low Pin Count (LPC) bus, a Serial Peripheral Interface ( "SPI"), a High Definition Audio ("HDA") bus, a Serial Advance Technology Attachment ("SATA") bus, a Universal Serial Bus ("USB") (versions 1, 2, 3), or a Universal Asynchronous receiver/transmitter ("UART") bus. 19 FIG. 10 shows a system comprising interconnected hardware devices or “chips” while FIG 19 in other embodiments, an example system on a chip ("SoC") may show. In at least one embodiment, the 19 Devices shown may be interconnected by proprietary connections, standardized connections (e.g., PCIe), or a combination thereof. In at least one embodiment, one or more components of 19 interconnected by Compute Express Link (CXL) connections.

In at least one embodiment, 19 a display 1924, a touch screen 1925, a touchpad 1930, a near field communication ("NFC") unit 1945, a sensor hub 1940, a thermal sensor 1946, an express chipset ("EC") 1935, a trusted platform module ("TPM") 1938, BIOS/firmware/flash memory ("BIOS", FW Flash") 1922, a DSP 1960, a drive "SSD or HDD") 1920 such as a solid state disk ("SSD") or hard disk drive ( "HDD"), a Wireless Local Area Network ("WLAN") unit 1950, a Bluetooth unit 1952, a Wireless Wide Area Network ("WWAN") unit 1956, a Global Positioning System (GPS) 1955, a camera (“USB 3. 0 camera”) 1954, e.g., a USB 3.0 camera, or a Low Power Double Data Rate (“LPDDR”) storage device (“LPDDR3”) 1915, e.g., e.g LPDDR3 standard These components can each be implemented in any suitable manner.

In at least one embodiment, other components may be communicatively coupled to processor 1910 via the components discussed above. In at least one embodiment, an accelerometer 1941 , an ambient light sensor ("ALS") 1942 , a compass 1943 , and a gyroscope 1944 may be communicatively coupled to the sensor hub 1940 . In at least one embodiment, a temperature sensor 1939, a fan 1937, a keyboard 1946, and a touchpad 1930 can be communicatively coupled to the EC 1935. In at least one embodiment, the speaker 1963, a headphone 1964, and a microphone ("Mic") 1965 may be communicatively coupled to an audio unit ("Audio Codec and Class D Amp") 1964, which in turn is communicatively coupled to the DSP 1960 can be. In at least one embodiment, audio unit 1964 may include, for example and without limitation, an audio encoder/decoder ("codec") and a class-D amplifier. In at least one embodiment, SIM card ("SIM") 1957 may be communicatively coupled to WWAN unit 1956 . In at least one embodiment, components such as WLAN unit 1950 and Bluetooth unit 1952, as well as WWAN unit 1956 may be implemented in a Next Generation Form Factor ("NGFF").

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, the inference and/or training logic 1415 in the system 19 to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robotic control system may be implemented using a processor as described above.

20 10 shows a computer system 2000, according to at least one embodiment. In at least one embodiment, computer system 2000 is configured to implement various methods and methods described in this disclosure.

In at least one embodiment, the computer system 2000 includes, without limitation, at least one central processing unit (“CPU”) 2002 connected to a communications bus 2010 using any suitable protocol such as PCI (“Peripheral Component Interconnect”), Peripheral Component Interconnect Express (“PCI-Express”), AGP (“Accelerated Graphics Port”), HyperTransport or any other bus or point-to-point communication protocol or any other bus or point-to-point communication protocol. In at least one embodiment, computer system 2000 includes, without limitation, main memory 2004 and control logic (e.g., implemented as hardware, software, or a combination thereof), and data is stored in main memory 2004, which may take the form of random access memory ("RAM"). In at least one embodiment, a network interface subsystem ("network interface") 2022 is provided by computer system 2000 to interface to other computing devices and networks for receiving data from and transmitting data to other systems.

In at least one embodiment, the computing system 2000 includes, in at least one embodiment, the input devices 2008, the parallel processing system 2012, and the display devices 2006, which may be a conventional cathode ray tube ("CRT"), a liquid crystal display ("LCD"), a light emitting diode ( "LED"), a plasma display or other suitable display technologies can be implemented. In at least one embodiment, user input is received from input devices 2008 such as a keyboard, mouse, touchpad, microphone, and more. In at least one embodiment, each of the foregoing modules may be arranged on a single semiconductor platform to form a processing system.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, the inference and/or training logic 1415 in the system 20 to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a computer system as described above.

21 12 shows a computer system 2100, according to at least one embodiment. In at least one embodiment, computer system 2100 includes, without limitation, computer 2110 and thumb drive 2120. In at least one embodiment, computer 2110 may include, without limitation, any number and type of processor(s) (not shown) and memory (not shown). In at least one embodiment, computer 2110 includes, without limitation, a server, a cloud instance, a laptop, and a desktop computer.

In at least one embodiment, USB key 2120 includes, without limitation, processing unit 2130, USB interface 2140, and USB interface logic 2150. In at least one embodiment, processing unit 2130 may be any instruction execution system, device, or device , capable of executing commands. In at least one embodiment, processing unit 2130 may include any number and type of processing cores (not shown), without limitation. In at least one embodiment, processing core 2130 includes an application specific integrated circuit ("ASIC") optimized to perform any number and type of operations compatible with associated with machine learning. For example, in at least one embodiment, the processing core 2130 is a tensor processing unit ("TPC") optimized for performing machine learning inference operations. In at least one embodiment, the processing core 2130 is an image processing unit ("VPU") optimized for performing inference operations related to machine vision and machine learning.

In at least one embodiment, the USB interface 2140 can be any type of USB plug or USB socket. For example, in at least one embodiment, the USB interface 2140 is a USB 3.0 Type-C socket for data and power. In at least one embodiment, USB interface 2140 is a USB 3.0 Type A connector. In at least one embodiment, USB interface logic 2150 may include any amount and type of logic that enables processing unit 2130 to communicate over the USB Port 2140 to interface with devices (e.g., computer 2110).

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, the inference and/or training logic 1415 in the system 21 to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a computer system as described above.

22A Figure 1 shows an example architecture in which a plurality of GPUs 2210-2213 are communicatively coupled to a plurality of multi-core processors 2205-2206 via high-speed links 2240-2243 (eg, buses, point-to-point links, etc.). In one embodiment, the high-speed links 2240-2243 support a communication throughput of 4 GB/s, 30 GB/s, 80 GB/s, or higher. Various connection protocols can be used including, but not limited to, PCIe 4.0 or 5.0 and NVLink 2.0.

Additionally, and according to one embodiment, two or more of the GPUs 2210-2213 are interconnected via high-speed links 2229-2230, which may be implemented with the same or different protocols/links than those used for the high-speed links 2240-2243. Similarly, two or more multi-core processors 2205-2206 can be connected via high-speed interconnect 2228, which can be symmetric multiprocessor (SMP) buses operating at 20 GB/s, 30 GB/s, 120 GB/s, or work higher. Alternatively, all communication between different in 22A system components shown take place via the same protocols/links (e.g. via a common connection structure).

In one embodiment, each multi-core processor 2205-2206 is communicatively coupled to processor memory 2201-2202 via memory interconnects 2226-2227, and each graphics processor 2210-2213 is communicatively coupled to GPU memory 2220-2223 via GPU memory interconnects 2250-2253. Memory interconnects 2226-2227 and 2250-2253 may use the same or different memory access technologies. As an example, without limitation, processor memory 2201-2202 and GPU memory 2220-2223 may be volatile memory such as dynamic random access memory (DRAM) (including stacked DRAMs), graphics DDR-SDRAM (GDDR) (e.g., GDDR5, GDDR6), or High bandwidth memory (HBM) and/or non-volatile memory such as 3D XPoint or Nano-Ram. In one embodiment, a portion of the processor memories 2201-2202 may be volatile memory and another portion may be non-volatile memory (e.g., when using a two-level memory hierarchy (2LM)).

As described herein, while different processors 2205-2206 and GPUs 2210-2213 can be physically coupled to a particular memory 2201-2202 and 2220-2223, respectively, a unified memory architecture can be implemented using the same virtual system address space (aka referred to as the “effective address space”) is distributed across different physical memories. For example For example, processor memories 2201-2202 can each include 64 GB of system memory address space and GPU memories 2220-2223 can each include 32 GB of system memory address space (resulting in a total of 256 GB of addressable memory in this example).

22B 12 shows additional details for a connection between a multi-core processor 2207 and a graphics accelerator module 2246, according to one embodiment. Graphics accelerator module 2246 may include one or more GPU chips integrated on a line card coupled to processor 2207 via high speed interconnect 2240 . Alternatively, the graphics accelerator module 2246 may be integrated in the same package or chip as the processor 2207.

In at least one embodiment, the processor 2207 depicted includes a plurality of cores 2260A-2260D, each with a translation lookaside buffer 2261A-2261D and one or more caches 2262A-2262D. In at least one embodiment, cores 2260A-2260D may include various other components for executing instructions and processing data that are not shown. Caches 2262A-2262D may include level 1 (L1) and level 2 (L2) caches. Additionally, one or more shared caches 2256 may be included within caches 2262A-2262D and shared between sets of cores 2260A-2260D. For example, one embodiment of processor 2207 includes 24 cores, each with its own L1 cache, twelve shared L2 caches, and twelve shared L3 caches. In this embodiment, one or more L2 and L3 caches are shared between two adjacent cores. The processor 2207 and the graphics accelerator module 2246 are connected to the system memory 2214, which contains the processor memories 2201-2202 of 22A may include.

Coherency is maintained for data and instructions stored in various caches 2262A-2262D, 2256 and in system memory 2214 via communication between the cores over a coherency bus 2264. For example, cache coherency logic/circuitry may be associated with each cache to communicate over the coherency bus 2264 in response to detected reads or writes to particular cache lines. In one implementation, a cache snooping protocol is implemented over the coherency bus 2264 to snoop on cache accesses.

In one embodiment, a proxy circuit 2225 communicatively couples the graphics accelerator module 2246 to the coherency bus 2264 so that the graphics accelerator module 2246 can participate in a cache coherency protocol as a peer of the cores 2260A-2260D. In particular, an interface 2235 is provided for connection to the proxy circuitry 2225 via the high-speed connection 2240 (e.g. PCIe bus, NVLink, etc.), and an interface 2237 connects the graphics accelerator module 2246 to the connection 2240.

In one implementation, an accelerator integration circuit 2236 is provided for a plurality of graphics processing engines 2231, 2232, N of the graphics accelerator module 2246 for cache management, memory access, context management, and interrupt management services. The engines 2231, 2232, N can each comprise a separate graphics processing unit (GPU). Alternatively, the graphics processing engines 2231, 2232, N can include different types of graphics processing engines within a GPU, such as e.g. B. graphics execution units, media processing engines (e.g. video encoders/decoders), samplers and blit engines. In at least one embodiment, graphics accelerator module 2246 may be a GPU with a plurality of graphics processing engines 2231-2232,N or graphics processing engines 2231-2232,N may be individual GPUs integrated into a common chassis, line card, or chip.

In one embodiment, the accelerator integration circuit 2236 includes a memory management unit (MMU) 2239 for performing various memory management functions such as virtual to physical memory translations (also referred to as effective to real memory translations) and memory access protocols for accessing the system memory 2214. MMU 2239 can may also include a translation lookaside buffer (TLB) (not shown) for caching translations from virtual/effective to physical/real addresses. In one embodiment, a cache 2238 stores instructions and data for efficient access by engines 2231-2232,N. In one embodiment, the data stored in cache 2238 and graphics memories 2233-2234,M is shared with core caches 2262A-2262D , 2256 and the system memory 2214 are kept coherent. As previously mentioned, this can be done via proxy circuitry 2225 for cache 2238 and memories 2233-2234, M (e.g. sending updates to cache 2238 in related to changes/accesses to cache lines in processor caches 2262A-2262D, 2256 and receiving updates from cache 2238).

A set of registers 2245 stores context data for threads executed by the graphics processing engines 2231-2232,N and a context management circuit 2248 manages thread contexts. For example, the context management circuitry 2248 may perform save and restore operations to save and restore the contexts of different threads during context switching (e.g., when a first thread is saved and a second thread saved so that a second thread is executed by the graphics processing engine can be). For example, upon a context switch, the context management circuit 2248 may store current register values in a particular area in memory (e.g., identified by an identifier). It can then restore register values when returning to a context. In one embodiment, an interrupt management circuit 2247 receives and processes interrupts received from system devices.

In one implementation, virtual/effective addresses from graphics processing engine 2231 are translated by MMU 2239 to real/physical addresses in system memory 2214 . One embodiment of accelerator integration circuit 2236 supports multiple (e.g., 4, 8, 16) graphics accelerator modules 2246 and/or other accelerator devices. The graphics accelerator module 2246 can be dedicated to a single application running on the processor 2207 or shared between multiple applications. In one embodiment, a virtualized graphics execution environment is presented in which resources of graphics processing engines 2231-2232, N are shared between multiple applications or virtual machines (VMs). In at least one embodiment, resources may be divided into "slices" that are allocated to different VMs and/or applications based on the processing requirements and priorities associated with VMs and/or applications.

In at least one embodiment, accelerator integration circuit 2236 functions as a bridge to a system graphics accelerator module 2246 and provides address translation and system memory caching services. In addition, the accelerator integration circuit 2236 may provide virtualization facilities for a host processor to manage virtualization of the graphics processing modules 2231-2232, interrupts, and memory management.

Since the hardware resources of the graphics processing engines 2231-2232,N are explicitly mapped to a real address space seen by the host processor 2207, any host processor can directly address these resources via an effective address value. A functionality of the accelerator integrated circuit 2236, in one embodiment, is the physical separation of the engines 2231-2232, N so that they appear to a system as independent entities.

In at least one embodiment, one or more graphics memories 2233-2234,M are coupled to engines 2231-2232,N, respectively. Graphics memories 2233-2234,M store commands and data to be processed by the respective graphics processing engine 2231-2232,N. Graphics memory 2233-2234,M may be volatile memory such as DRAM (including stacked DRAM), GDDR memory (e.g., GDDR5, GDDR6), or HBM, and/or non-volatile memory such as 3D XPoint or Nano-Ram .

In one embodiment, to reduce data traffic over link 2240, biasing techniques are used to ensure that the data stored in graphics memories 2233-2234,M is the data most frequently used by the graphics processing engines. Engines 2231-2232, N and preferably not used by cores 2260A-2260D (at least not often). Similarly, a default mechanism attempts to hold the data needed by the cores (and preferably not by the graphics processing engines 2231-2232,N) in the cores' caches 2262A-2262D, 2256 and system memory 2214.

22C FIG. 12 shows another exemplary embodiment in which accelerator integrated circuit 2236 is integrated in processor 2207. FIG. In this embodiment, engines 2231-2232,N communicate directly over high speed link 2240 with accelerator integrated circuit 2236 via interface 2237 and interface 2235 (which in turn may use any type of bus or interface protocol). The accelerator integrated circuit 2236 can perform the same operations as those in FIG 22B perform as described, but possibly with a height ren throughput because it is in close proximity to the coherency bus 2264 and caches 2262A-2262D, 2256. In one embodiment, various programming models are supported, including a dedicated process programming model (no graphics accelerator virtualization) and shared programming models (with virtualization), which may include programming models controlled by the accelerator integration circuit 2236, as well as programming models that controlled by the Graphics Accelerator Module 2246.

In at least one embodiment, the graphics processing engines 2231-2232,N are dedicated to a single application or process under a single operating system. In at least one embodiment, a single application can route other application requests to the graphics processing engines 2231-2232,N, thus providing virtualization within a VM/partition.

In at least one embodiment, the graphics processing engines 2231-2232,N may be shared between multiple VM/application partitions. In at least one embodiment, shared models may use a system hypervisor to virtualize the graphics processing engines 2231-2232,N to allow access by any operating system. In systems with a partition without a hypervisor, the engines 2231-2232, N for graphics processing applications are part of an operating system. In at least one embodiment, an operating system may virtualize the graphics processing engines 2231-2232,N to allow access to any process or application.

In at least one embodiment, the graphics accelerator module 2246 or an individual graphics processing engine 2231-2232, N selects a process item using a process handle. In one embodiment, the process elements are stored in system memory 2214 and are addressable using an effective address-to-real address translation technique described herein. In at least one embodiment, a process handle may be an implementation-specific value provided to a host process upon registering its context with the graphics processing engine 2231-2232, N (i.e., calling system software to add a process item to a process item linked list to add). In at least one embodiment, the lower 16 bits of a process handle may be an offset of the process item within a process item linked list.

22D 1 shows an exemplary accelerator integration slice 2290. As used herein, a “slice” includes a specific portion of the processing resources of the accelerator integration circuit 2236. The effective application address space 2282 within the system memory 2214 stores the processing elements 2283. In one embodiment, Process items 2283 stored in response to GPU calls 2281 from applications 2280 running on processor 2207. A process element 2283 contains the process state for the corresponding application 2280. A work descriptor (WD) 2284 contained in process element 2283 may be a single job requested by an application or may contain a pointer to a queue of jobs. In at least one embodiment, WD 2284 is a pointer to a job request queue in an application's address space 2282 .

Graphics accelerator module 2246 and/or individual graphics processing engines 2231-2232,N may be shared by all or a subset of processes in a system. In at least one embodiment, infrastructure for establishing process state and sending a WD 2284 to a graphics accelerator 2246 to start a job in a virtualized environment may be included.

In at least one embodiment, a dedicated process programming model is implementation specific. In this model, a single process owns the graphics accelerator module 2246 or a single graphics processing engine 2231. Because the graphics accelerator module 2246 is owned by a single process, a hypervisor initializes the accelerator IC 2236 for an owning partition and an operating system initializes the accelerator IC 2236 for an owning process when the graphics accelerator engine 2246 is allocated.

In operation, a WD fetch unit 2291 in the accelerator integration section 2290 fetches the next WD 2284, which is an indication of the graphics processor's one or more graphics processing engines acceleration module 2246 work to be done. Data from WD 2284 may be stored in registers 2245 and used by MMU 2239, interrupt management circuitry 2247, and/or context management circuitry 2248 as shown. For example, one embodiment of MMU 2239 includes segment/page run circuitry for accessing segment/page tables 2286 within operating system virtual address space 2285 . The interrupt management circuit 2247 may process interrupt events 2292 received from the graphics accelerator module 2246 . When performing graphics operations, an effective address 2293 generated by a graphics processing engine 2231-2232,N is translated by the MMU 2239 to a real address.

In one embodiment, the same set of registers 2245 is duplicated for each graphics processing engine 2231-2232, N and/or graphics accelerator module 2246 and may be initialized by a hypervisor or operating system. Each of these duplicated registers may be included in an accelerator integration section 2290. Exemplary registers that can be initialized by a hypervisor are shown in Table 1. Table 1 - Registers initialized by the hypervisor 1 Tabs for controlling sections 2 Area pointer for processes scheduled with real addresses (RA). 3 Authority mask override register 4 Offset of an interrupt vector table entry 5 Delimitation of an interrupt vector table entry 6 state register 7 ID of a logical partition 8th Pointer to a real address (RA) hypervisor accelerator usage record 9 Register for memory description

Example registers that can be initialized by an operating system are shown in Table 2. Table 2 - Registers initialized by the operating system 1 Process and thread identification 2 Pointer for save/restore context with effective address (EA) 3 Pointer to usage record of a virtual address accelerator (VA) 4 Virtual address (VA) memory segment table pointer 5 mask of authority 6 work descriptor

In one embodiment, each WD 2284 is specific to a particular Graphics Accelerator Module 2246 and/or Graphics Processing Engines 2231-2232,N. It contains all of the information that a Graphics Processing Engine 2231-2232,N needs, or can, to perform its work be a pointer to a memory location where an application has established a command queue for work to be performed.

22E Figure 12 shows additional details for an example embodiment of a common model. This embodiment includes a hypervisor real address space 2298 in which a process item list 2299 is stored. The hypervisor's physical address space 2298 is accessible through a hypervisor 2296 that virtualizes the graphics accelerator engines for the operating system 2295 .

In at least one embodiment, shared programming models enable all or a subset of processes from all or a subset of partitions in a system the use of a graphics accelerator module 2246. There are two programming models in which the graphics accelerator module 2246 is shared between multiple processes and partitions: time-shared and graphics-directed shared.

In this model, the system hypervisor 2296 contains the graphics accelerator module 2246 and makes its functionality available to all operating systems 2295 . For a graphics accelerator 2246 to support virtualization through the system hypervisor 2296, the graphics accelerator 2246 may meet the following conditions: 1) An application's job request must be autonomous (i.e. state need not be maintained between jobs), or the graphics accelerator 2246 must provide a mechanism provide for backing up and restoring the context. 2) For an application's job request, the Graphics Accelerator 2246 engine guarantees that it will be completed in a specified amount of time, including any translation errors, or the Graphics Accelerator 2246 engine provides the ability to anticipate the processing of a job. 3) The Graphics Accelerator Module 2246 must guarantee inter-process balance when operating in a directed shared programming model.

In at least one embodiment, the application 2280 is required to invoke an operating system 2295 system call with a 2246 graphics accelerator engine, a work descriptor (WD), an authority mask register (AMR) value, and a context save (CSRP) value /Restore Area Pointer). In at least one embodiment, the graphics accelerator module 2246 type describes a targeted acceleration function for a system call. In at least one embodiment, the type of graphics accelerator module 2246 may be a native value. In at least one embodiment, the WD is formatted specifically for the graphics accelerator module 2246 and may be in the form of a graphics accelerator module 2246 command, an effective address pointer to a user-defined structure, an effective address pointer to a command queue, or any other data structure describing the graphics accelerator module 2246 to be executed works are available. In one embodiment, an AMR value is an AMR state to use for a current process. In at least one embodiment, a value passed to an operating system is similar to an application setting an AMR. If the accelerator integration circuit 2236 and graphics accelerator module 2246 implementations do not support a User Authority Mask Override Register (UAMOR), an operating system can apply a current UAMOR value to an AMR value before passing an AMR in a hypervisor call. Hypervisor 2296 may optionally apply a current AMOR (Authority Mask Override Register) value before an AMR is passed in process element 2283 . In at least one embodiment, CSRP is one of the registers 2245 that contains an effective address of a region in an application's address space 2282 for the graphics accelerator module 2246 to save and restore context state. This pointer is optional if no state needs to be saved between jobs or if a job is to be brought forward. In at least one embodiment, the context save/restore area may be fixed system memory.

Upon receiving a system call, the operating system 2295 can verify that the application 2280 has been registered and authorized to use the graphics accelerator module 2246 . Operating system 2295 then calls hypervisor 2296 with the information listed in Table 3. Table 3 - Invocation Parameters from OS to Hypervisor 1 Work Descriptor (WD) 2 Authority Mask Register Value (AMR) (potentially masked) 3 Context Save/Restore Area Pointer (CSRP) for context save/restore with effective address (EA) 4 Process ID (PID) and optional Thread ID (TID) 5 Accelerator utilization record pointer (AURP) with virtual address (VA) 6 Stored segment table pointer (SSTP) pointer with virtual address 7 Logical interrupt service number (LISN)

Upon receiving a hypervisor call, the hypervisor 2296 verifies that the operating system 2295 is registered and has been granted permission to use the graphics accelerator module 2246 . Hypervisor 2296 then inserts process item 2283 into a process item linked list for a corresponding graphics accelerator engine 2246 type. A process item may include the information listed in Table 4. Table 4 - Process Item Information 1 Work Descriptor (WD) 2 Authority Mask Register Value (AMR) (potentially masked) 3 Context Save/Restore Area Pointer (CSRP) for context save/restore with effective address (EA) 4 Process ID (PID) and optional Thread ID (TID) 5 Accelerator utilization record pointer (AURP) with virtual address (VA) 6 Stored segment table pointer (SSTP) pointer with virtual address 7 Logical interrupt service number (LISN) 8th Interrupt vector table derived from hypervisor invocation parameters 9 Value of a status register (SR) 10 Logical Partition ID (LPID) 11 Real address (RA) hypervisor accelerator usage record pointer 12 Storage Descriptor Register (SDR)

In at least one embodiment, the hypervisor initializes a plurality of registers 2290 of accelerator integration sections 2290-2245.

As in 22F As shown, in at least one embodiment, a unified memory is used that is addressable via a shared virtual memory address space used to access physical processor memory 2201-2202 and GPU memory 2220-2223. In this implementation, operations performed on GPUs 2210-2213 use a shared virtual/effective memory address space to access processor memories 2201-2202 and vice versa, simplifying programmability. In one embodiment, a first portion of a virtual/effective address space is allocated to processor memory 2201, a second portion to second processor memory 2202, a third portion to GPU memory 2220, and so on. In at least one embodiment, this distributes an entire virtual/effective memory space (sometimes referred to as effective address space) across each of processor memory 2201-2202 and GPU memory 2220-2223, giving each processor or GPU access to each physical memory with a virtual memory associated with that memory address can access.

In one embodiment, policy/coherency management circuitry 2294A-2294E within one or more MMUs 2239A-2239E provides cache coherency between the caches of one or more host processors (e.g., 2205) and GPUs 2210- 2213 secure and implements default techniques specifying physical memory in which to store certain types of data. While multiple constraints of constraint/coherency management circuitry 2294A-2294E in 22F 1, the circuitry for constraint/coherency management may be implemented within an MMU by one or more host processors 2205 and/or within the accelerator integrated circuit 2236.

In one embodiment, the GPU-bound memory 2220-2223 may be allocated as part of the system memory and accessed using shared virtual memory (SVM) technology, but without incurring the performance penalties associated with full system cache coherency. In at least one embodiment, the ability to access GPU-bound memory 2220-2223 as part of system memory without the inconvenient cache coherency overhead is a beneficial operating environment for offloading to GPUs. This arrangement allows host processor 2205 software to construct operands and access calculation results without the overhead of conventional I/O DMA data copies. Such traditional copies include driver calls, interrupts, and memory mapped I/O (MMIO) accesses, all of which are inefficient compared to simple memory accesses. In at least one embodiment, the ability to access GPU-bound memory 2220-2223 without cache coherency overhead may be critical to offloaded computation execution time. For example, in cases with significant streaming write memory traffic, the cache coherence overhead can significantly reduce an effective write bandwidth seen by a GPU 2210-2213. In at least one embodiment, operand construction efficiency, result access efficiency, and GPU computation efficiency may play a role in determining the effectiveness of GPU offloading.

In at least one embodiment, the selection of the GPU policy and the host processor policy is controlled by a policy tracker data structure. For example, a bias table can be a page-granular structure (i.e., controlled with the granularity of a memory page) containing 1 or 2 bits per GPU-attached memory page. In at least one embodiment, a default table may be implemented in a stolen memory area of one or more GPU-bound memories 2220-2223 with or without a default cache in GPU 2210-2213 (e.g., to cache frequently/recently used entries of a default table). Alternatively, an entire default table can be maintained within a graphics processor.

In at least one embodiment, a default table entry associated with each access to GPU-bound memory 2220-2223 is accessed prior to the actual access to GPU memory, causing the following operations. First, local requests from the GPU 2210-2213 that find their page in the GPU policy are routed directly to a corresponding GPU memory 2220-2223. Local requests from a graphics processor that find their page in the host memory default are forwarded to processor 2205 (e.g., over a high-speed link, as described above). In one embodiment, requests from processor 2205 that find a requested page in the host processor bias complete a request like a normal memory read. Alternatively, requests directed to a GPU designated page may be forwarded to GPU 2210-2213. In at least one embodiment, a GPU may then commit a page to host processor default when it is not using a page. In at least one embodiment, a page's default state may be changed by either a software-based mechanism, a hardware-assisted software-based mechanism, or, for a limited set of cases, a purely hardware-based mechanism.

One mechanism for changing the default state uses an API call (e.g. OpenCL), which in turn calls a GPU's device driver, which in turn sends a message (or queues a command description) to a GPU to instruct it to change a default state and to perform a cache flush operation in a host for some transitions. In at least one embodiment, the cache flush operation is used for a transition from host processor 2205 default to GPU default, but not for a reverse transition.

In one embodiment, cache coherency is maintained by temporarily not caching GPU-given pages from host processor 2205 . To access these pages, processor 2205 may request access from GPU 2210, which may not grant access immediately. Therefore, in order to reduce communication between processor 2205 and GPU 2210, it is beneficial to ensure that GPU-given pages are those required by a GPU but not required by the host processor 2205, and vice versa.

The hardware structure(s) 1415 is/are used to implement one or more embodiments. Details regarding the hardware structure(s) 1415 are in connection with herein 14A and or 14B provided.

23 FIG. 1 shows exemplary integrated circuits and associated graphics processors that may be fabricated using one or more IP cores, according to various embodiments described herein. In addition to what is shown herein, other logic and circuitry may be included in at least one embodiment, including additional graphics processors/cores, peripheral interface controllers, or general purpose processor cores.

23 13 is a block diagram depicting an exemplary system on an on-chip integrated circuit 2300 that may be fabricated with one or more IP cores, according to at least one embodiment. In at least one embodiment, the integrated circuit 2300 includes one or more application processor(s) 2305 (e.g., CPUs), at least one graphics processor 2310, and may additionally include an image processor 2315 and/or a video processor 2320, each of these processors being a modular IP core can. In at least one embodiment, integrated circuit 2300 includes peripheral or bus logic including USB controller 2325, UART controller 2330, SPI/SDIO controller 2335, and I.sup.2S/I.sup.2C controller 2340. In In at least one embodiment, the integrated circuit 2300 may include a display device 2345 coupled to one or more of the following controllers: a High-Definition Multimedia Interface (HDMI) controller 2350 and a Mobile Industry Processor Interface (MIPI) display interface 2355 In at least one embodiment, the memory may be provided by a flash memory subsystem 2360, which includes flash memory and a flash memory controller. In at least one embodiment, the memory interface may be provided via a controller 2365 to access SDRAM or SRAM memory devices. In at least one embodiment, some integrated circuits additionally include an embedded security engine 2370.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, the inference and/or training logic 1415 in the integrated circuit 2300 may be used to infer or predict operations based, at least in part, on weight parameters obtained using training operations with neural networks, functions and/or neural network architectures or use cases of neural networks described herein.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robotic control system may be implemented using a processor as described above.

24A-24B 12 show exemplary integrated circuits and associated graphics processors that may be fabricated with one or more IP cores according to various embodiments. In addition to what is shown herein, other logic and circuitry may be included in at least one embodiment, including additional graphics processors/cores, peripheral interface controllers, or general purpose processor cores.

24A-24B 12 show, as block diagrams, exemplary graphics processors for use in a SoC in accordance with the exemplary embodiments described here. 24A FIG. 24 shows an example integrated circuit graphics processor 2410 on a system-on-chip that may be fabricated with one or more IP cores, according to at least one embodiment. 24B 14 shows an additional example integrated circuit graphics processor 2440 on a system-on-a-chip that may be fabricated using one or more IP cores, in accordance with at least one embodiment. In at least one embodiment, the graphics processor 2410 is from 24A a graphics processor core with low power consumption. In at least one embodiment, the graphics processor 2440 is from 24B a GPU core with higher performance. In at least one embodiment, each of the graphics processors 2410, 2440 may be a variant of the graphics processor 2310 23 be.

In at least one embodiment, graphics processor 2410 includes a vertex processor 2405 and one or more fragment processors 2415A-2415N (e.g., 2415A, 2415B, 2415C, 2415D, through 2415N-1 and 2415N). In at least one embodiment, graphics processor 2410 may execute different shader programs via separate logic such that vertex processor 2405 is optimized for executing vertex shader operations, while one or more fragment processor(s) 2415A-2415N execute fragment shader operations (eg Pixel) for fragment or pixel shader programs. In at least one embodiment, vertex processor 2405 performs a vertex processing stage of a 3D graphics pipeline and generates primitives and vertex data. In at least one embodiment, fragment processor(s) 2415A-2415N uses the primitive and vertex data generated by vertex processor 2405 to generate a frame buffer that is displayed on a display device. In at least one embodiment, is/are the fragment processor(s) 2415A-2415N optimized to run fragment shader programs as provided in an OpenGL API that can be used to perform similar operations as a pixel shader program, as provided in a direct 3D API.

In at least one embodiment, graphics processor 2410 additionally includes one or more memory management units (MMUs) 2420A-2420B, cache(s) 2425A-2425B, and circuit interconnect(s) 2430A-2430B. In at least one embodiment, one or more MMU(s) 2420A-2420B are provided for mapping virtual to physical addresses for graphics processor 2410, including for vertex processor 2405 and/or fragment processor(s). ) 2415A-2415N, which may refer to vertex or image/texture data stored in memory in addition to vertex or image/texture data stored in one or more cache(s) 2425A-2425B. In at least one embodiment, one or more MMU(s) 2420A-2420B can be synchronized with other MMUs within the system, including one or more MMUs that interface with one or more application processor(s) 2305, image processor(s) 2315, and/or video processor (en) 2320 out 23 are associated so that each processor 2305-2420 can participate in a shared or unified virtual memory system. In at least one embodiment, circuit interconnect(s) 2430A-2430B enable graphics processor 2410 to connect to other IP cores within the SoC, either through an internal bus of the SoC or through a direct connection.

In at least one embodiment, graphics processor 2440 includes one or more MMU(s) 2420A-2420B, caches 2425A-2425B, and graphics processor 2410 circuit interconnects 2430A-2430B 24A . In at least one embodiment, graphics processor 2440 includes one or more shading core(s) 2455A-2455N (e.g., 2455A, 2455B, 2455C, 2455D, 2455E, 2455F through 2455N-1, and 2455N) that provide a unified shading core architecture in which a single core or type or core can execute all kinds of programmable shading code, including shading program code to implement vertex shading, fragment shading, and/or computational shading. In at least one embodiment, a number of shading kernels may vary. In at least one embodiment, the graphics processor 2440 includes an inter-core task manager 2445 that operates as a thread dispatcher to dispatch threads of execution to one or more shading cores 2455A-2455N, and a tiling unit 2458 for Accelerate tiling operations for tile-based rendering, in which rendering operations for a scene are partitioned in image space, for example to exploit local spatial coherence within a scene or to optimize the use of internal caches.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, inference and/or training logic 1415 in integrated circuit 24A and/or 24B may be used to infer or predict operations based at least in part on weight parameters obtained using training operations with neural networks, functions, and/or or neural network architectures or neural network use cases described herein.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robotic control system may be implemented using a processor as described above.

25A-25B 12 show further example graphics processor logic consistent with the embodiments described herein. 25A FIG. 1 shows a graphics core 2500, which in at least one embodiment may include the graphics processor 2310 of FIG 23 may comprise and in at least one embodiment a unified shading kernel 2455A-2455N as in 24B can represent. 25B Figure 2530 shows a highly parallel general purpose graphics processing unit 2530, suitable for deployment on a multi-chip module, in at least one embodiment.

In at least one embodiment, the graphics core 2500 includes a shared instruction cache 2502, a texture unit 2518, and a cache/shared memory 2520 that are shared among the execution resources in the graphics core 2500. In at least one embodiment For example, graphics core 2500 may include multiple sections 2501A-2501N or partitions for each core, and a graphics processor may include multiple instances of graphics core 2500. Sections 2501A-2501N may include support logic including a local instruction cache 2504A-2504N, a thread scheduler 2506A-2506N, a thread dispatcher 2508A-2508N, and a set of registers 2510A-2510N. In at least one embodiment, sections 2501A-2501N may include a set of additional functional units (AFUs 2512A-2512N), floating point units (FPU 2514A-2514N), integer arithmetic logic units (ALUs 2516-2516N), address calculation units (ACU 2513A-2513N), double precision floating point units (DPFPU 2515A-2515N) and matrix processing units (MPU 2517A-2517N).

In at least one embodiment, FPUs 2514A-2514N can perform single-precision (32-bit) and half-precision (16-bit) floating-point operations, while DPFPUs 2515A-2515N can perform double-precision (64-bit) floating-point operations. In at least one embodiment, ALUs 2516A-2516N can perform variable-precision integer operations with 8-bit, 16-bit, and 32-bit precision and can be configured for mixed-precision operations. In at least one embodiment, MPUs 2517A-2517N may also be configured for mixed-precision matrix operations, including half-precision floating-point and 8-bit integer operations. In at least one embodiment, MPUs 2517-2517N may perform a variety of matrix operations to accelerate machine learning application frameworks, including support for accelerated general matrix to matrix multiplication , GEMM). In at least one embodiment, AFUs 2512A-2512N can perform additional logical operations not supported by floating point or integer units, including trigonometric operations (e.g., sine, cosine, etc.).

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, the inference and/or training logic 1415 in Graphics Core 2500 for inferring or predicting operations may be based, at least in part, on weight parameters obtained using training operations with neural networks, functions and/or neural network architectures, or as described herein Use cases of neural networks were calculated can be used.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robotic control system may be implemented using a processor as described above.

25B 12 shows a general purpose processing unit (GPGPU) 2530 that can be configured to perform highly parallel computational operations by an array of graphics processing units in at least one unit, in at least one embodiment. In at least one embodiment, the GPGPU 2530 can be connected directly to other instances of the GPGPU 2530 to create a multi-GPU cluster to improve training speed for deep neural networks. In at least one embodiment, GPGPU 2530 includes a host interface 2532 to enable connection to a host processor. In at least one embodiment, host interface 2532 is a PCI Express interface. In at least one embodiment, the host interface 2532 may be a proprietary communication interface or communication structure. In at least one embodiment, the GPGPU 2530 receives commands from a host processor and uses a global scheduler 2534 to schedule the threads of execution associated with those commands onto a set of compute clusters 2536A-2536H. In at least one embodiment, compute clusters 2536A-2536H share a cache 2538. In at least one embodiment, cache 2538 may serve as a higher level cache for caches within compute clusters 2536A-2536H.

In at least one embodiment, GPGPU 2530 includes memory 2544A-2544B coupled to compute clusters 2536A-2536H via a set of memory controllers 2542A-2542B. In at least one embodiment, memory 2544A-2544B may include various types of memory devices, including dynamic random access memory memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory.

In at least one embodiment, compute clusters 2536A-2536H each include a set of graphics cores, such as graphics core 2500 of FIG 25A , which can include multiple types of integer and floating-point logic units that can perform arithmetic operations with a range of precisions that are also suitable for machine learning. For example, in at least one embodiment, at least a subset of floating point units in each of the compute clusters 2536A-2536H may be configured to perform 16-bit or 32-bit floating point operations, while another subset of floating point units may be configured to perform 64-bit floating point operations can be configured.

In at least one embodiment, multiple instances of the GPGPU 2530 can be configured to operate as a cluster of compute. In at least one embodiment, the communication used by compute clusters 2536A-2536H for synchronization and data exchange varies from embodiment to embodiment. In at least one embodiment, multiple instances of GPGPU 2530 communicate via host interface 2532. In at least one embodiment, GPGPU 2530 includes an I/O hub 2539 that couples GPGPU 2530 to a GPU connection 2540 that connects directly to other instances of the GPGPU 2530. In at least one embodiment, the GPU link 2540 is coupled to a dedicated GPU-to-GPU bridge that enables communication and synchronization between multiple GPGPU 2530 instances. In at least one embodiment, the GPU connection 2540 is coupled to a high-speed connection to send and receive data to other GPGPUs or parallel processors. In at least one embodiment, multiple instances of GPGPU 2530 reside in separate computing systems and communicate through a network device accessible through host interface 2532 . In at least one embodiment, GPU connection 2540 may be configured to allow connection to a host processor in addition to or as an alternative to host interface 2532 .

In at least one embodiment, the GPGPU 2530 can be configured to train neural networks. In at least one embodiment, GPGPU 2530 may be used within an inference platform. In at least one embodiment where GPGPU 2530 is used for inferencing, GPGPU may include fewer compute clusters 2536A-2536H than when GPGPU is used to train a neural network. In at least one embodiment, the storage technology associated with memory 2544A-2544B may differ between inference and training configurations, with higher bandwidth storage technologies being used for training configurations. In at least one embodiment, the inference configuration of the GPGPU 2530 may support specific instruction inference. For example, in at least one embodiment, an inference configuration may provide support for one or more 8-bit integer dot product instructions that may be used in inference operations for deployed neural networks.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, the inference and/or training logic 1415 in Graphics Core 2500 for inferring or predicting operations may be based, at least in part, on weight parameters obtained using training operations with neural networks, functions and/or neural network architectures, or as described herein Use cases of neural networks were calculated can be used.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robotic control system may be implemented using a processor as described above.

26 16 is a block diagram depicting a computer system 2600 in accordance with at least one embodiment. In at least one embodiment, the computing system 2600 includes a processing subsystem 2601 having one or more processors 2602 and a system memory 2604 that communicate over a connection path that may include a storage hub 2605 . In at least one embodiment, the memory hub 2605 may be a separate component within a chipset component or integrated into one or more processor(s) 2602 . In at least one embodiment, storage hub 2605 is coupled to I/O subsystem 2611 via connection 2606 . In at least one embodiment, I/O subsystem 2611 includes an I/O hub 2607 that may enable computing device 2600 to receive input from one or more input devices 2608 . In at least one embodiment, I/O hub 2607 may enable a controller for a display, which may be included in one or more processors 2602, to provide outputs to one or more display devices 2610A. In at least one embodiment, one or more display devices 2610A coupled to I/O hub 2607 may include a local, internal, or embedded display device.

In at least one embodiment, processing subsystem 2601 includes one or more parallel processors 2612 coupled to memory hub 2605 via bus or other connection 2613 . In at least one embodiment, communication link 2613 may be any number of standards-based communication link technologies or protocols, such as but not limited to PCI Express, or a proprietary communication interface or communication structure. In at least one embodiment, parallel processor(s) 2612 form a computational parallel or vector processing system that may include a large number of computational cores and/or processing clusters, such as a many integrated core (MIC) processor. . In at least one embodiment, parallel processor(s) 2612 form a graphics processing subsystem that can output pixels to one of display device(s) 2610A coupled via I/O hub 2607 . In at least one embodiment, parallel processor(s) 2612 may also include a controller and display interface (not shown) to enable direct connection to display device(s) 2610B.

In at least one embodiment, a system storage device 2614 may be coupled to I/O hub 2607 to provide a storage mechanism for computer system 2600. In at least one embodiment, an I/O switch 2616 may be used to provide an interface mechanism that enables connections between the I/O hub 2607 and other components, such as a network adapter 2618 and/or a wireless network adapter 2619, that may be used in can be integrated into the platform, and various other devices that can be added via add-on device(s) 2620. In at least one embodiment, network adapter 2618 may be an Ethernet adapter or other wired network adapter. In at least one embodiment, wireless network adapter 2619 may include one or more Wi-Fi, Bluetooth, near field communication (NFC), or other network devices that include one or more wireless radios.

In at least one embodiment, computing system 2600 may include other components not explicitly shown, including USB or other port connections, optical storage drives, video capture devices, and the like that may also connect to I/O hub 2607 . In at least one embodiment, communication paths connecting various components in 26 interconnect, be implemented using any suitable protocols, such as PCI (Peripheral Component Interconnect) based protocols (e.g. PCI-Express) or other bus or point-to-point communication interfaces and/or protocol(s) such as NVLink- High-speed connections or connection protocols.

In at least one embodiment, one or more parallel processor(s) 2612 includes circuitry optimized for graphics and video processing, including, for example, video output circuitry, and constitutes a graphics processing unit (GPU). In at least one embodiment, one or more parallel processors include (en) 2612 circuitry optimized for general-purpose processing. In at least one embodiment, components of computing system 2600 may be integrated with one or more other system elements on a single integrated circuit. For example, in at least one embodiment, one or more parallel processor(s) 2612, memory hub 2605, processor(s) 2602, and I/O hub 2607 may be integrated into a system-on-a-chip (SoC) integrated circuit. In at least one embodiment, components of computing system 2600 may be integrated into a single chassis to form a system in package (SIP) configuration. In at least one embodiment, min At least some of the components of the computer system 2600 can be integrated into a multi-chip module (MCM) that can be connected to other multi-chip modules to form a modular computer system.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, the inference and/or training logic 1415 in Graphics Core 2500 for inferring or predicting operations may be based, at least in part, on weight parameters obtained using training operations with neural networks, functions and/or neural network architectures, or as described herein Use cases of neural networks were calculated can be used.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a computer system as described above.

27A 12 shows a parallel processor 2700 according to at least one embodiment. In at least one embodiment, various components of parallel processor 2700 may be implemented using one or more integrated circuit devices, such as programmable processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGA). The parallel processor 2700 illustrated in at least one embodiment is a variant of one or more in 26 shown parallel processor(s) 2612 according to an example embodiment.

In at least one embodiment, parallel processor 2700 includes a parallel processing unit 2702. In at least one embodiment, parallel processing unit 2702 includes an I/O unit 2704 that enables communication with other devices, including other instances of parallel processing unit 2702. In at least one embodiment, I/O unit 2704 may be directly connected to other devices. In at least one embodiment, I/O unit 2704 is connected to other devices via a hub or switch interface, such as storage hub 2605. In at least one embodiment, connections between the storage hub 2605 and the I/O device 2704 form a communication link 2613. In at least one embodiment, the I/O device 2704 connects to a host interface 2706 and a storage crossbar 2716 wherein host interface 2706 receives commands directed to performing processing operations and memory crossbar 2716 receives commands directed to performing memory operations.

In at least one embodiment, when the host interface 2706 receives a command buffer via the I/O unit 2704, the host interface 2706 may direct operations to a front end 2708 for execution of those commands. In at least one embodiment, the front end 2708 is coupled to a scheduler 2710 configured to dispatch commands or other work to a processing cluster array 2712 . In at least one embodiment, scheduler 2710 ensures processing cluster array 2712 is properly configured and in a valid state before dispatching tasks to processing cluster array 2712 of processing cluster array 2712 . In at least one embodiment, scheduler 2710 is implemented via firmware logic running on a microcontroller. In at least one embodiment, the scheduler 2710 implemented in a microcontroller is configurable to perform complex scheduling and work distribution operations at coarse and fine granularity, enabling fast pre-allocation and context switching of threads running on the processing array 2712. In at least one embodiment, the host software may expose workloads for scheduling on the array 2712 via one of a plurality of graphics processing doorbells. In at least one embodiment, workloads may then be automatically distributed to the processing array 2712 across the processing array 2712 by scheduler 2710 logic within a microcontroller that includes the scheduler 2710 .

In at least one embodiment, processing cluster array 2712 may include up to "N" processing clusters (e.g., cluster 2714A, cluster 2714B, up to cluster 2714N). In at least one embodiment, each cluster 2714A-2714N of processing cluster array 2712 can execute a large number of concurrent threads. In at least one embodiment, the scheduler can 2710 allocate work to the clusters 2714A-2714N of the processing cluster array 2712 using various scheduling and/or work distribution algorithms, which may vary depending on the workload involved for each program or computation type. In at least one embodiment, scheduling may be handled dynamically by scheduler 2710 or assisted in part by compiler logic during compilation of program logic configured for execution by processing cluster array 2712 . In at least one embodiment, different clusters 2714A-2714N of the processing cluster array 2712 may be assigned to process different types of programs or to perform different types of calculations.

In at least one embodiment, the processing cluster array 2712 can be configured to perform various types of parallel processing operations. In at least one embodiment, the processing cluster array 2712 is configured to perform general parallel computing operations. For example, in at least one embodiment, the processing cluster array 2712 may include logic to perform processing tasks including filtering video and/or audio data, performing modeling operations including physical operations, and performing data transformations.

In at least one embodiment, processing cluster array 2712 is configured to perform parallel graphics processing operations. In at least one embodiment, the processing cluster array 2712 may include additional logic to support the execution of such graphics processing operations, including but not limited to texture scanning logic for performing texture operations, as well as tessellation logic and other vertex processing logic. In at least one embodiment, processing cluster array 2712 may be configured to execute graphics processing-related shading programs such as vertex shading, tessellation shading, geometry shading, and pixel shading. In at least one embodiment, parallel processing unit 2702 may transfer data from system memory through I/O unit 2704 for processing. In at least one embodiment, the transferred data may be stored in on-chip memory (e.g., parallel processor memory 2722) during processing and then written back to system memory.

In at least one embodiment, when using the parallel processing unit 2702 to perform graphics processing, the scheduler 2710 can be configured to split a processing workload into approximately equally sized tasks to better distribute graphics processing operations across multiple clusters 2714A-2714N of the processing cluster array 2712 enable. In at least one embodiment, portions of the processing cluster array 2712 can be configured to perform different types of processing. For example, in at least one embodiment, a first part can be configured to perform vertex shading and topology generation, a second part can be configured to perform tessellation and geometry shading, and a third part can be configured to perform performs pixel shading or other screen space operations to produce a rendered image for display. In at least one embodiment, intermediate data generated by one or more of clusters 2714A-2714N may be stored in buffers to facilitate the transfer of intermediate data between clusters 2714A-2714N for further processing.

In at least one embodiment, processing cluster array 2712 may receive processing tasks to be performed via scheduler 2710 , which receives processing task definition commands from frontend 2708 . In at least one embodiment, the processing tasks may include indices of the data to be processed, e.g., surface (patch) data, primitive data, vertex data, and/or pixel data, as well as state parameters and instructions that define how the data is to be processed (e.g., which program to run shall be). In at least one embodiment, scheduler 2710 may be configured to fetch indices corresponding to tasks or to receive indices from front end 2708 . In at least one embodiment, the front end 2708 may be configured to ensure that the processing cluster array 2712 is configured to a valid state before any incoming command buffers (e.g., batch buffer, push buffer, etc.) are specified workload is initiated.

In at least one embodiment, each of one or more instances of parallel processing unit 2702 may be coupled to parallel processor memory 2722 . In at least one embodiment, parallel processor memory 2722 is accessible via memory crossbar 2716, which accommodates memory requests from processing cluster array 2712 and I/O unit 2704 can catch. In at least one embodiment, parallel processor memory 2722 may be accessed through a memory interface 2718, via memory crossbar 2716. In at least one embodiment, memory interface 2718 may include multiple partition units (e.g., partition unit 2720A, partition unit 2720B, via partition unit 2720N), each with a portion (eg, memory unit) of the parallel processor memory 2722 may be coupled. In at least one embodiment, a number of partition units 2720A-2720N are configured to correspond to a number of storage units such that a first partition unit 2720A has a corresponding first storage unit 2724A, a second partition unit 2720B has a corresponding storage unit 2724B, and an Nth partition unit 2720N has a corresponding Nth storage unit 2724N. In at least one embodiment, a number of partition units 2720A-2720N may not equal a number of storage devices.

In at least one embodiment, memory units 2724A-2724N may comprise various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory (SGRAM), including double data rate ( Graphics Double Data Rate, GDDR). In at least one embodiment, memory devices 2724A-2724N may also include 3D stacked memory, including but not limited to high-bandwidth memory (HBM). In at least one embodiment, render targets, such as frame buffers or texture maps, may be stored across memory units 2724A-2724N such that partition units 2720A-2720N may write portions of each render target in parallel to utilize the available bandwidth of parallel processor memory 2722 to use efficiently. In at least one embodiment, a local instance of parallel processor memory 2722 may be eliminated in favor of a unified memory design that uses system memory in conjunction with local cache memory.

In at least one embodiment, each of clusters 2714A-2714N of processing cluster array 2712 may process data that is written to one of storage units 2724A-2724N in parallel processor memory 2722. In at least one embodiment, the memory crossbar 2716 can be configured so that an output of each cluster 2714A-2714N can be transferred to any partition unit 2720A-2720N or to another cluster 2714A-2714N that can perform additional processing operations on an output. In at least one embodiment, each cluster 2714A-2714N can communicate with storage interface 2718 via storage crossbar 2716 to read from and write to various external storage devices. In at least one embodiment, memory crossbar 2716 connects to memory interface 2718 to communicate with I/O device 2704, and connects to a local instance of parallel processor memory 2722, thereby connecting processing units within different processing clusters 2714A-2714N to system memory or another memory that is not local to the parallel processing unit 2702. In at least one embodiment, memory crossbar 2716 may use virtual channels to separate traffic flows between clusters 2714A-2714N and partition units 2720A-2720N.

In at least one embodiment, multiple instances of the parallel processing unit 2702 may be provided on a single add-on card, or multiple add-on cards may be interconnected. In at least one embodiment, different instances of parallel processing unit 2702 may be configured to work together even if the different instances have different numbers of processing cores, different amounts of local parallel processor memory, and/or other configuration differences. For example, in at least one embodiment, some instances of the parallel processing unit 2702 may include higher precision floating point units compared to other instances. In at least one embodiment, systems containing one or more instances of parallel processing unit 2702 or parallel processor 2700 may be implemented in a variety of configurations and form factors, including but not limited to desktop, laptop, or handheld personal computers, servers, workstations , game consoles and/or embedded systems.

27B FIG. 27 shows a block diagram of a partition unit 2720 according to at least one embodiment. In at least one embodiment, partition unit 2720 is an example of one of partition units 2720A-2720N 27A . In at least one embodiment, the partition unit 2720 includes an L2 cache 2721, a frame buffer interface 2725, and a ROP 2726 (raster operations unit). L2 cache 2721 is a read/write cache configured to perform load and store operations received from memory crossbar 2716 and ROP 2726 the. In at least one embodiment, read errors and urgent writeback requests are issued from L2 cache 2721 to framebuffer interface 2725 for processing. In at least one embodiment, updates may also be sent via framebuffer interface 2725 to a framebuffer for processing. In at least one embodiment, frame buffer interface 2725 is coupled to one of the storage units in parallel processor memory, such as storage units 2724A-2724N of FIG 27 (e.g. in parallel processor memory 2722).

In at least one embodiment, ROP 2726 is a processing unit that performs raster operations such as stencil, z-test, blending, and the like. In at least one embodiment, ROP 2726 then outputs processed graphics data, which is stored in graphics memory. In at least one embodiment, ROP 2726 includes compression logic to compress depth or color data that is written to memory and to decompress depth or color data that is read from memory. In at least one embodiment, the compression logic may be lossless compression logic that uses one or more of a variety of compression algorithms. The type of compression performed by ROP 2726 can vary based on statistical properties of the data being compressed. For example, in at least one embodiment, delta color compression is performed on depth and color data on a per Ktil basis.

In at least one embodiment, ROP 2726 is located in each processing cluster (eg, clusters 2714A-2714N of 27 ) instead of contained in partition unit 2720. In at least one embodiment, read and write requests for pixel data are transmitted across memory crossbar 2716 instead of pixel fragment data. In at least one embodiment, processed graphics data may be displayed on a display device, eg, one or more display devices 2610 26 , be forwarded for further processing by processor(s) 2602 or for further processing by one of the processing units within parallel processor 2700 27A to get redirected.

27C 12 shows a block diagram of a processing cluster 2714 within a parallel processing unit, according to at least one embodiment. In at least one embodiment, a processing cluster is an instance of one of processing clusters 2714A-2714N 27 . In at least one embodiment, the processing cluster 2714 can be configured to execute many threads in parallel, where the term "thread" refers to an instance of a particular program executing on a particular set of input data. In at least one embodiment, single-instruction, multiple-data (SIMD) instruction issue techniques are used to support the parallel execution of a large number of threads without providing multiple independent instruction units. In at least one embodiment, single-instruction multiple-thread (SIMT) techniques are used to support the parallel execution of a large number of commonly synchronized threads using a common instruction unit configured to issue instructions to a set of processing machines within each individual processing cluster.

In at least one embodiment, the operation of processing cluster 2714 may be controlled via a pipeline manager 2732 that distributes processing tasks to SIMT parallel processors. In at least one embodiment, pipeline manager 2732 receives instructions from scheduler 2710 from 27 and manages the execution of these instructions via a graphics multiprocessor 2734 and/or a texture unit 2736. In at least one embodiment, the graphics multiprocessor 2734 is an exemplary case of a SIMT parallel processor. However, in at least one embodiment, processing cluster 2714 may include different types of SIMT parallel processors of different architectures. In at least one embodiment, one or more instances of graphics multiprocessor 2734 may be included in a processing cluster 2714. In at least one embodiment, the graphics multiprocessor 2734 can process data, and a data crossbar 2740 can be used to distribute processed data to any of several possible destinations, including other shading units. In at least one embodiment, the pipeline manager 2732 may facilitate the distribution of processed data by specifying destinations for processed data to be distributed across the data crossbar 2740.

In at least one embodiment, each graphics multiprocessor 2734 within the processing cluster 2714 may include an identical set of functional execution logic (e.g., arithmetic logic units, load memory units). In at least one embodiment, the functional Execution logic can be configured in a pipelined fashion, allowing new instructions to be issued before previous instructions complete. In at least one embodiment, the functional execution logic supports a variety of operations, including integer and floating point arithmetic, comparison operations, Boolean operations, bit shifting, and computation of various algebraic functions. In at least one embodiment, the same hardware with functional units can be used to perform different operations, and there can be any combination of functional units.

In at least one embodiment, the instructions submitted to processing cluster 2714 form a thread. In at least one embodiment, a set of threads executing across a set of parallel processing engines forms a thread group. In at least one embodiment, a thread group forms a program with different input data. In at least one embodiment, each thread within a thread group may be mapped to a different processing engine within a graphics multiprocessor 2734. In at least one embodiment, a thread group may include fewer threads than a number of processing engines within a graphics multiprocessor 2734. In at least one embodiment, when a thread group includes fewer threads than a number of processing engines, one or more of the processing engines to be idle during the cycles in which this thread group is processed. In at least one embodiment, a thread group may also include more threads than a number of processing engines within the graphics multiprocessor 2734. In at least one embodiment, if a thread group includes more threads than the number of processing engines within the graphics multiprocessor 2734, the processing may be performed over consecutive clock cycles in at least one embodiment. In at least one embodiment, multiple groups of threads may execute concurrently on a 2734 graphics multiprocessor.

In at least one embodiment, the graphics multiprocessor 2734 includes internal cache memory for performing load and store operations. In at least one embodiment, the graphics multiprocessor 2734 may forego an internal cache memory and use cache memory (eg, L1 cache 2748) within the processing cluster 2714. In at least one embodiment, each graphics multiprocessor 2734 also has access to L2 caches within partition units (eg, partition units 2720A-2720N of 27 ) that are shared by all processing clusters 2714 and can be used to transfer data between threads. In at least one embodiment, graphics multiprocessor 2734 may also access off-chip global memory, which may include one or more local parallel processor memories and/or system memory. In at least one embodiment, any memory external to parallel processing unit 2702 may be used as global memory. In at least one embodiment, processing cluster 2714 includes multiple instances of graphics multiprocessor 2734 may share common instructions and data that may be stored in L1 cache 2748 .

In at least one embodiment, each processing cluster 2714 may include an MMU 2745 (memory management unit) configured to map virtual addresses to physical addresses. In at least one embodiment, one or more instances of MMU 2745 may reside within memory interface 2718 of 27 condition. In at least one embodiment, MMU 2745 includes a set of page table entries (PTEs) used to map a virtual address to a physical address of a tile (more on tiling) and optionally a cache line index. In at least one embodiment, MMU 2745 may include address translation lookaside (TLB) buffers or caches, which may reside in a graphics multiprocessor 2734 or L1 cache or processing cluster 2714 . In at least one embodiment, the physical address is processed to distribute the surface data access location to allow for efficient interleaving of requests between partition units. In at least one embodiment, the cache line index may be used to determine whether a request for a cache line is a hit or miss.

In at least one embodiment, a processing cluster 2714 may be configured such that each graphics multiprocessor 2734 is coupled to a texture unit 2736 to perform texture mapping operations, such as determining texture sample locations, reading texture data, and filtering texture data. In at least one embodiment, texture data is read from an internal texture L1 cache (not shown) or from an L1 cache within the graphics multiprocessor 2734 and fetched from an L2 cache, local parallel processor memory, or system memory as needed. In at least one embodiment, each graphics multiprocessor provides 2734 processed Tasks to the data crossbar 2740 to make processed tasks available to another processing cluster 2714 for further processing, or to store processed tasks via the memory crossbar 2716 to an L2 cache, local parallel processor memory, or system memory. In at least one embodiment, preROP 2742 (Pre-Raster Operations Unit) is configured to receive data from graphics multiprocessor 2734 and forward data to ROP units, which as described herein may reside on partition units (e.g., partition units 2720A- 2720N from 27 ). In at least one embodiment, the PreROP 2742 unit may perform color mixing optimizations, organize pixel color data, and perform address translations.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robotic control system may be implemented using a processor as described above.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are here in connection with 14A and or 14B provided. In at least one embodiment, the inference and/or training logic 1415 in the system 17C to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

27D 12 shows a graphics multiprocessor 2734 in accordance with at least one embodiment. In at least one embodiment, the graphics multiprocessor 2734 is coupled to the pipeline manager 2732 of the processing cluster 2714 . In at least one embodiment, the graphics multiprocessor 2734 has an execution pipeline that includes an instruction cache 2752, an instruction unit 2754, an address mapper 2756, a register file 2758, one or more General Purpose Graphics Processing Unit (GPGPU) cores 2762, and one or more loaders /memory units 2766 includes, but is not limited to. GPGPU cores 2762 and load/store units 2766 are coupled to cache memory 2772 and shared memory 2770 via a store and cache interconnect 2768 .

In at least one embodiment, instruction cache 2752 receives a stream of instructions to be executed from pipeline manager 2732. In at least one embodiment, instructions are cached in instruction cache 2752 and dispatched to instruction unit 2754 for execution. In at least one embodiment, the instruction unit 2754 may dispatch instructions as thread groups (e.g., warps), with each thread of the thread group being associated with a different execution unit within the GPGPU core 2762. In at least one embodiment, an instruction can access any local, shared, or global address space by specifying an address within a unified address space. In at least one embodiment, address mapping unit 2756 may be used to translate addresses in a uniform address space into a unique memory address accessible by load/store units 2766.

In at least one embodiment, register file 2758 provides a set of registers for functional units of graphics multiprocessor 2734. In at least one embodiment, register file 2758 provides temporary storage for operands associated with data paths of functional units (e.g., GPGPU cores 2762, load/store units 2766) of graphics multiprocessor 2734. In at least one embodiment, register file 2758 is partitioned among each of the functional units such that each functional unit has a dedicated portion of register file 2758 associated with it. In at least one embodiment, register file 2758 is divided into various warps executed by graphics multiprocessor 2734.

In at least one embodiment, GPGPU cores 2762 may each include floating point units (FPUs) and/or integer arithmetic logic units (ALUs) used to execute graphics multiprocessor 2734 instructions. GPGPU cores 2762 may be architecturally similar or architecturally different. In at least one embodiment, a first portion of the GPGPU cores 2762 includes a single-precision FPU and an integer ALU, while a second portion of the GPGPU cores includes a double-precision FPU. In at least one embodiment spiel, FPUs can implement the IEEE 754-2008 standard for floating-point arithmetic or enable variable-precision floating-point arithmetic. In at least one embodiment, the graphics multiprocessor 2734 may additionally include one or more fixed function(s) or special purpose functional units to perform specific functions such as copying rectangles or pixel blending operations. In at least one embodiment, one or more GPGPU cores may also include fixed or special purpose functional logic.

In at least one embodiment, the GPGPU cores 2762 include SIMD logic capable of executing a single instruction on multiple sets of data. In at least one embodiment, GPGPU cores 2762 can physically execute SIMD4, SIMD8, and SIMD16 instructions and logically execute SIMD1, SIMD2, and SIMD32 instructions. In at least one embodiment, SIMD instructions for GPGPU cores may be generated at compile time by a shadowing compiler or may be generated automatically when executing programs written and compiled for SPMD (Single Program Multiple Data) or SIMT architectures. In at least one embodiment, multiple threads of a program configured for a SIMT execution model may be executed from a single SIMD instruction. For example, in at least one embodiment, eight SIMT threads performing the same or similar operations may execute in parallel through a single SIMD8 logic unit.

In at least one embodiment, memory and cache interconnect 2768 is an interconnection network that connects each functional unit of graphics multiprocessor 2734 to register file 2758 and to shared memory 2770 . In at least one embodiment, store and cache interconnect 2768 is a crossbar interconnect that allows load/store unit 2766 to implement load and store operations between shared memory 2770 and register file 2758. In at least one embodiment, the register file 2758 can operate at the same frequency as the GPGPU cores 2762 so that data transfer between the GPGPU cores 2762 and the register file 2758 occurs with very low latency. In at least one embodiment, shared memory 2770 may be used to enable communication between threads executing on functional units within graphics multiprocessor 2734. In at least one embodiment, cache memory 2772 may be used as a data cache to cache texture data between functional units and texture unit 2736, for example. In at least one embodiment, shared memory 2770 may also be used as a program managed cache. For at least one embodiment, threads executing on GPGPU cores 2762 may programmatically store data in shared memory in addition to automatically cached data stored in cache memory 2772 .

In at least one embodiment, a parallel processor or GPGPU as described herein is communicatively coupled to host/processor cores to perform graphics operations, machine learning, pattern analysis operations, and various general purpose GPU (GPGPU) functions accelerate. In at least one embodiment, the GPU may be communicatively coupled to the host processor/cores via a bus or other connection (e.g., a high-speed connection such as PCIe or NVLink). In at least one embodiment, the GPU may be integrated in the same package or chip as the cores and communicatively coupled to the cores via an internal processor bus/interconnect (i.e., internal to the package or chip). In at least one embodiment, regardless of how the GPU is connected, the processor cores may assign work to the GPU in the form of sequences of commands/instructions contained in a work descriptor. In at least one embodiment, the GPU then uses dedicated circuitry/logic to efficiently process these commands/instructions.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are here in connection with 14A and or 14B provided. In at least one embodiment, the inference and/or training logic 1415 in the system 17C to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robotic control system may be implemented using a processor as described above.

28 12 shows a multi-GPU computing system 2800 in accordance with at least one embodiment. In at least one embodiment, multi-GPU computing system 2800 may include a processor 2802 coupled via a host interface switch 2804 to multiple general purpose graphics processing units (GPGPUs) 2806A-D. In at least one embodiment, host interface switch 2804 is a PCI Express switch device that couples processor 2802 to a PCI Express bus over which processor 2802 can communicate with GPGPUs 2806A-D. The GPGPUs 2806A-D can communicate with each other via a set of high-speed point-to-point links between the GPUs 2816. In at least one embodiment, GPU-to-GPU connections 2816 to each of the GPGPUs 2806A-D make a dedicated GPU connection. In at least one embodiment, P2P GPU links 2816 allow direct communication between each GPGPU 2806A-D without requiring communication over the host interface bus 2804 to which the processor 2802 is attached. In at least one embodiment where GPU-to-GPU traffic is routed to P2P GPU connections 2816, host interface bus 2804 remains available for accessing system memory or communicating with other instances of the multi-GPU Computing system 2800, for example via one or more network devices. While in at least one embodiment the GPGPUs 2806A-D connect to the processor 2802 through the host interface switch 2804, in at least one embodiment the processor 2802 includes direct support for P2P GPU links 2816 and can interface directly with the GPGPUs 2806A -D to be connected.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are here in connection with 14A and or 14B provided. In at least one embodiment, the inference and/or training logic 1415 in the system 17C to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a GPU as described above.

29 10 is a block diagram of a graphics processor 2900, according to at least one embodiment. In at least one embodiment, graphics processor 2900 includes ring interconnect 2902, pipeline front end 2904, engine 2937, and graphics cores 2980A-2980N. In at least one embodiment, ring interconnect 2902 couples graphics processor 2900 to other entities, including other graphics processors or one or more general processor cores. In at least one embodiment, graphics processor 2900 is one of many processors integrated into a multi-core processing system.

In at least one embodiment, graphics processor 2900 receives batches of commands over ring interconnect 2902. In at least one embodiment, incoming commands are interpreted by command streamer 2903 in pipeline front end 2904. In at least one embodiment, graphics processor 2900 includes scalable execution logic for performing 3D geometry processing and media processing using graphics core(s) 2980A-2980N. In at least one embodiment for 3D geometry processing instructions, instruction streamer 2903 provides instructions to geometry pipeline 2936. In at least one embodiment, for at least some media processing instructions, instruction streamer 2903 provides instructions to a video front end 2934 running media engine 2937 is coupled. In at least one embodiment, the media engine 2937 includes a video quality engine (VQE) 2930 for video and image post-processing and a multi-format encoding/decoding engine (MFX) 2933 to provide hardware-accelerated encoding and decoding of media data. In at least one embodiment, create geometry pipelines 2936 and media engine 2937 each have execution threads for thread execution resources provided by at least one graphics core 2980A.

In at least one embodiment, graphics processor 2900 includes scalable thread execution resources with modular cores 2980A-2980N (sometimes referred to as core sections) each having multiple sub-cores 2950A-550N, 2960A-2960N (sometimes referred to as core subsections). In at least one embodiment, graphics processor 2900 may have any number of graphics cores 2980A through 2980N. In at least one embodiment, the graphics processor 2900 includes a graphics core 2980A having at least a first sub-core 2950A and a second sub-core 2960A. In at least one embodiment, the graphics processor 2900 is a low-power processor with a single sub-core (e.g., 2950A). In at least one embodiment, graphics processor 2900 includes multiple graphics cores 2980A-2980N, each including a set of first sub-cores 2950A-2950N and a set of second sub-cores 2960A-2960N. In at least one embodiment, each sub-core in first sub-cores 2950A-2950N includes at least a first set of execution units 2952A-2952N and media/texture scanners 2954A-2954N. In at least one embodiment, each sub-core in the second sub-cores 2960A-2960N includes at least a second set of execution units 2962A-2962N and samplers 2964A-2964N. In at least one embodiment, each sub-core 2950A-2950N, 2960A-2960N shares a set of shared resources 2970A-2970N. In at least one embodiment, shared resources include shared cache memory and pixel operation logic.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are here in connection with 14A and or 14B provided. In at least one embodiment, the inference and/or training logic 1415 in the system 17C to infer or predict operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and/or neural network architectures, or neural network use cases described herein.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a GPU as described above.

30 FIG. 3 is a block diagram depicting a microarchitecture for a processor 3000, which may include logic circuitry for executing instructions, according to at least one embodiment. In at least one embodiment, processor 3000 may execute instructions, including x86 instructions, ARM instructions, specialized instructions for application specific integrated circuits (ASICs), etc. In at least one embodiment, processor 3010 may include registers for storing packed data, such as 64- Bit-wide MMXTM registers in microprocessors operating on MMX technology from Intel Corporation of Santa Clara, California. In at least one embodiment, MMX registers, which are available in both integer and floating point form, can operate with packed data items involving single instructions, multiple data ("SIMD"), and streaming SIMD extensions ("SSE"). In at least one embodiment, 128-bit wide XMM registers related to SSE2, SSE3, SSE4, AVX, or beyond (commonly referred to as “SSEx”) technology may contain such packed data operands. In at least one embodiment, processor 3010 may execute instructions to accelerate machine learning or deep learning algorithms, training, or inference.

In at least one embodiment, the processor 3000 includes an R in-order front end (“front end”) 3001 to fetch instructions to be executed and to prepare instructions to be used later in the processor pipeline. In at least one embodiment, the front end 3001 may include multiple entities. In at least one embodiment, an instruction prefetcher 3026 fetches instructions from memory and provides instructions to an instruction decoder 3028, which in turn decodes or interprets instructions. For example, in at least one embodiment, instruction decoder 3028 decodes a received instruction into one or more operations, referred to as "micro-instructions" or "micro-ops" (also referred to as "micro-ops" or "uops"), that may be executed by that machine . In at least one embodiment, instruction decoder 3028 decomposes an instruction into an opcode and appropriate data and control fields defined by microarchitecture ver may be used to perform operations in accordance with at least one embodiment. In at least one embodiment, a trace cache 3030 may assemble decoded uops into program-ordered sequences or traces in a uop queue 3034 for execution. In at least one embodiment, when the trace cache 3030 encounters a complex instruction, a microcode ROM 3032 provides the Uops necessary to complete the operation.

In at least one embodiment, some instructions may be converted into a single micro-op, while others require multiple micro-ops to perform a full operation. In at least one embodiment, instruction decoder 3028 may access microcode ROM 3032 to execute an instruction when more than four micro-operations are required to execute an instruction. In at least one embodiment, an instruction may be decoded into a small number of micro-ops for processing in instruction decoder 3028 . In at least one embodiment, an instruction may be stored in microcode ROM 3032 if a number of micro-ops are required to perform the operation. In at least one embodiment, trace cache 3030 refers to an Entry Point Programmable Logic Array ("PLA") for determining a correct microinstruction pointer for reading microcode sequences to complete one or more instructions from the Microcode ROM 3032, according to at least one embodiment. In at least one embodiment, after the microcode ROM 3032 finishes sequencing micro-ops for an instruction, the engine front end 3001 may resume fetching micro-ops from the sequence cache 3030 .

In at least one embodiment, out-of-order execution engine 3003 may prepare instructions for execution. In at least one embodiment, the out-of-order execution logic has a series of buffers to smooth and reorder instruction flow to optimize performance when the pipeline is shut down and execution is scheduled. The out-of-order execution engine 3003 includes, without limitation, an allocator/register renamer 3040, a memory uop queue 3042, an integer/floating point uop queue 3044, a memory scheduler 3046, a fast scheduler 3002, a slow/generic floating point scheduler ("slow/general FP scheduler") 3004 and a simple floating point scheduler ("simple FP scheduler") 3006. In at least one embodiment, the fast scheduler 3002, the slow/general floating point Scheduler 3004 and simple floating point scheduler 3006 are also referred to herein collectively as "uop scheduler 3002, 3004, 3006". The Allocator/Register Renamer 3040 allocates machine buffers and resources each uop needs to execute. In at least one embodiment, allocator/register renamer 3040 renames logical registers to entries in a register file. In at least one embodiment, the allocator/register renamer 3040 also allocates an entry for each uop in one of two uop queues, the memory uop queue 3042 for memory operations and the integer/floating point op queue 3044 for non-memory operations, before the memory scheduler 3046 and the uop schedulers 3002, 3004, 3006. In at least one embodiment, the uop schedulers 3002, 3004, 3006 determine the uops required to complete their operation based on the readiness of their dependent input register operand sources and the availability of execution resources , when a uop is ready to run. In at least one embodiment, the fast scheduler 3002 of at least one embodiment can schedule on every half of the main clock cycle, while the slow/general floating point scheduler 3004 and the simple floating point scheduler 3006 can schedule once per main processor clock cycle. In at least one embodiment, hub port uop schedulers 3002, 3004, and 3006 may arbitrate to schedule uops for execution.

In at least one embodiment, execution block b 11 includes, without limitation, an integer register file/bypass network 3008, a floating point register file/bypass network ("FP" register file/bypass network) 3010, address generation units ( "AGUs") 3012 and 3014, fast arithmetic logic units (ALUs) ("fast ALUs") 3016 and 3018, a slow arithmetic logic unit ("slow ALU") 3020, a floating point ALU ("FP") 3022 and a floating point move unit ("FP move") 3024. In at least one embodiment, integer register file/bypass network 3008 and floating point register file/bypass network 3010 are also referred to herein as "register files 3008, 3010". In at least one embodiment, AGUS 3012 and 3014, fast ALUs 3016 and 3018, slow ALU 3020, floating point ALU 3022, and floating point mover 3024 are also referred to herein as "execution units 3012, 3014, 3016, 3018 , 3020, 3022 and 3024”. In at least one embodiment, the execution block b 11 may be without a Ein constraint may include any number (including zero) and type of register files, bypass networks, address generation units, and execution units in any combination.

In at least one embodiment, register files 3008, 3010 may be located between uop schedulers 3002, 3004, 3006 and execution units 3012, 3014, 3016, 3018, 3020, 3022, and 3024. In at least one embodiment, register file/bypass network 3008 performs integer operations. In at least one embodiment, floating point register file/bypass network 3010 performs floating point operations. In at least one embodiment, each of the register files 3008, 3010 may include, without limitation, a bypass network that may bypass or forward to new dependent uops just completed results that have not yet been written to the register file. In at least one embodiment, register files 3008, 3010 can communicate with each other. In at least one embodiment, register file/bypass network 3008 may include, without limitation, two separate register files, one register file for low-order 32 bits of data and a second register file for high-order 32 bits of data. In at least one embodiment, the floating point register file/bypass network 3010 may include, without limitation, 128-bit wide entries since floating point instructions typically have operands that are 64 to 128 bits wide.

In at least one embodiment, execution units 3012, 3014, 3016, 3018, 3020, 3022, 3024 may execute instructions. In at least one embodiment, register files 3008, 3010 store integer and floating point data operand values that microinstructions are required to execute. In at least one embodiment, processor 3000 may include any number and combination of execution units 3012, 3014, 3016, 3018, 3020, 3022, 3024, without limitation. In at least one embodiment, floating point ALU 3022 and floating point mover 3024 may perform floating point, MMX, SIMD, AVX, and SSE or other operations, including special machine learning instructions. In at least one embodiment, floating point ALU 3022 may include, without limitation, a 64-bit x 64-bit floating point divider for performing division, square root, and remainder micro-operations. In at least one embodiment, instructions containing a floating point value may be processed using floating point hardware. In at least one embodiment, ALU operations may be committed to fast ALUs 3016,3018. In at least one embodiment, the fast ALUs 3016, 3018 can perform fast operations with an effective latency of half a clock cycle. In at least one embodiment, the most complex integer operations go to the slow ALU 3020, since the slow ALU 3020 may include, without limitation, integer execution hardware for long latency operations such as a multiplier, shifts, flag logic, and branch processing. In at least one embodiment, memory load/store operations may be performed by AGUS 3012, 3014. In at least one embodiment, fast ALU 3016, fast ALU 3018, and slow ALU 3020 can perform integer operations on 64-bit data operands. In at least one embodiment, fast ALU 3016, fast ALU 3018, and slow ALU 3020 can be implemented to support a variety of data bit sizes, including sixteen, thirty-two, 128, 256, etc. In at least one embodiment, floating point ALU 3022 and the Floating point mover 3024 can be implemented to support a range of operands with different bit widths. In at least one embodiment, floating point ALU 3022 and floating point mover 3024 can operate on 128-bit wide packed data operands in conjunction with SIMD and multimedia instructions.

In at least one embodiment, the uop schedulers 3002, 3004, 3006 schedule dependent operations before the parent load has finished executing. Since uops may be speculatively scheduled and executed in processor 3000, in at least one embodiment, processor 3000 may also include logic for handling misses in memory. In at least one embodiment, when there is a miss loading data in the data cache, there may be dependent operations in the pipeline during execution that exited the scheduler with temporarily bad data. In at least one embodiment, a retry mechanism tracks and re-executes instructions that use incorrect data. In at least one embodiment, dependent operations may need to be re-executed and independent operations allowed to complete. In at least one embodiment, the scheduler and retry mechanism of at least one embodiment of a processor may also be configured to capture instruction sequences for text string comparison operations.

In at least one embodiment, the term "register" may refer to storage locations in the processor that may be used as part of instructions to identify operands. In at least one embodiment, registers may be those that can be used from outside the processor (from a programmer's point of view). In at least one embodiment, registers may not be limited to any particular circuit type. Rather, in at least one embodiment, a register may store data, provide data, and perform the functions described herein. In at least one embodiment, the registers described herein may be implemented by circuitry within a processor using any number of different techniques, such as dedicated physical registers, dynamically allocated physical registers using register renaming, combinations of dedicated and dynamically allocated physical registers, etc. In In at least one embodiment, integer registers store 32-bit integer data. The register file of at least one embodiment also includes eight packed data multimedia SIMD registers.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, some or all of the inference and/or training logic 1415 may be included in EXE block 3011 and other memory or registers, shown or not shown. For example, in at least one embodiment, the training and/or inference techniques described herein may use one or more of the ALUs illustrated in EXE block 3011 . Additionally, weight parameters may be stored in on-chip or off-chip memory and/or in registers (shown or not) that configure ALUs of EXE block 3011 to implement one or more machine learning algorithms, neural network architectures, use cases or training techniques described here.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a GPU as described above.

31 12 shows a deep learning application processor 3100 according to at least one embodiment. In at least one embodiment, deep learning applications processor 3100 uses instructions that, when executed by deep learning applications processor 3100, cause deep learning applications processor 3100 to perform some or all of the processes and techniques described in this disclosure. In at least one embodiment, deep learning application processor 3100 is an application specific integrated circuit (ASIC). In at least one embodiment, applications processor 3100 performs matrix multiplication operations that are either "hardwired" into hardware or performed as a result of the execution of one or more instructions, or both. In at least one embodiment, deep learning applications processor 3100 includes, without limitation, processing clusters 3110(1)-3110(12), inter-chip interconnects ("ICLs") 3120(1)-3120(12), inter-chip controllers (“ICCs”) 3130(1)-3130(2), Second Generation High Bandwidth Memory (“HBM2”) 3140(1)-3140(4), Memory Controller (“Mem Ctrlrs”) 3142(1)-3142 (4), a high-bandwidth memory ("HBM PHY") physical layer 3144(1)-3144(4), a management controller central processing unit (CPU) ) 3150, a Serial Peripheral Interface, Inter-Integrated Circuit, and General Purpose Input/Output (SPI, I2C, GPIO) block 3160, a Peripheral Component Connection Express Control, and Direct Memory Access block ( "PCIe Controller and DMA") 3170 and a sixteen-lane Peripheral Component Connection Express port ("PCI Express x 16") 3180.

In at least one embodiment, processing clusters 3110 may perform deep learning operations, including inference or prediction operations based on weight parameters calculated by one or more training techniques, including those described herein. In at least one embodiment, each processing cluster 3110 may include any number and type of processors, without limitation. In at least one embodiment, deep learning applications processor 3100 may include any number and type of processing clusters 3100 . In at least one embodiment, the inter-chip connections 3120 are bi-directional. In at least one embodiment, inter-chip interconnects 3120 and inter-chip controllers 3130 enable multiple deep learning application processors 3100 to exchange information, including enablement information, resulting from the execution of one or more machine learning algorithms running in one or more neural networks are included. In In at least one embodiment, deep learning application processor 3100 may include any number (including zero) and type of ICLs 3120 and ICCs 3130.

In at least one embodiment, the HBM2s 3140 are provided with a total of 31 gigabytes (GB) of memory. HBM2 3140(i) is associated with both Controller 3142(i) and HBM PHY 3144(i). In at least one embodiment, any number of HBM2s 3140 can provide any type and amount of high-bandwidth memory and can be associated with any number (including zero) and type of memory controllers 3142 and HBM PHYs 3144 . In at least one embodiment, SPI, I2C, GPIO 3160, PCIe controller and DMA 3170 and/or PCIe 3180 may be replaced with any number and type of blocks that enable any number and type of communication standards in any technically appropriate manner.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, the deep learning application processor is used to train a machine learning model, such as a neural network, to predict or infer information provided to the deep learning application processor 3100 . In at least one embodiment, the deep learning application processor 3100 is used to derive or predict information trained by another processor or system or by the deep learning application processor 3100 based on a trained machine learning model (e.g., a neural network). . In at least one embodiment, processor 3100 may be used to perform one or more of the neural network use cases described herein.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a GPU as described above.

32 3200 is a block diagram of a neuromorphic processor 3200, according to at least one embodiment. In at least one embodiment, neuromorphic processor 3200 may receive one or more inputs from sources external to neuromorphic processor 3200. In at least one embodiment, these inputs may be transmitted to one or more neurons 3202 within neuromorphic processor 3200. In at least one embodiment, neurons 3202 and their components may be implemented using circuitry or logic that includes one or more arithmetic logic units (ALUs). In at least one embodiment, neuromorphic processor 3200 may include, without limitation, thousands or millions of instances of neurons 3202, but any suitable number of neurons 3202 may be used. In at least one embodiment, each instance of neuron 3202 may include a neuron input 3204 and a neuron output 3206. In at least one embodiment, neurons 3202 may generate outputs that may be transmitted to inputs of other instances of neurons 3202. For example, in at least one embodiment, neuron inputs 3204 and neuron outputs 3206 may be connected to each other via synapses 3208.

In at least one embodiment, neurons 3202 and synapses 3208 may be interconnected such that neuromorphic processor 3200 operates to process or analyze information received from neuromorphic processor 3200. In at least one embodiment, neurons 3202 may emit an output pulse (or "fire" or "spike") when input received via neuron input 3204 exceeds a threshold. In at least one embodiment, neurons 3202 may sum or integrate the signals received at neuron inputs 3204. For example, in at least one embodiment, neurons 3202 may be implemented as leaky-integrate-and-fire neurons, where when a sum (referred to as "membrane potential") exceeds a threshold, neuron 3202 outputs (or "fires"). using a transfer function such as a sigmoid or threshold function. In at least one embodiment, a leaky integrate and fire neuron may sum the signals received at neuron inputs 3204 into a membrane potential and also apply a decay factor (or "leak") to reduce a membrane potential. In at least one embodiment, a leaky integrate-and-fire neuron may fire when multiple input signals are received at neuron inputs 3204 fast enough to cross a threshold (ie, before a membrane potential decays too low to fire). In at least one embodiment The neurons 3202 can be implemented by circuitry or logic that receives inputs, integrates inputs into a membrane potential, and decays a membrane potential. In at least one embodiment, the inputs may be averaged, or any other suitable transfer function may be used. Additionally, in at least one embodiment, without limitation, neurons 3202 may include comparator circuitry or logic that generates an output spike at neuron output 3206 when the result of applying a transfer function to neuron input 3204 exceeds a threshold. In at least one embodiment, once neuron 3202 has fired, it may ignore previously received input information, for example, by resetting a membrane potential to 0 or some other suitable default value. In at least one embodiment, once the membrane potential has been reset to 0, neuron 3202 may resume normal operation after an appropriate period of time (or refractory period).

In at least one embodiment, neurons 3202 may be connected to each other via synapses 3208. In at least one embodiment, synapses 3208 may function to transmit signals from an output of a first neuron 3202 to an input of a second neuron 3202. In at least one embodiment, neurons 3202 may transmit information across more than one synapse 3208 instance. In at least one embodiment, one or more instances of neuron output 3206 may be connected to an instance of neuron input 3204 in the same neuron 3202 via an instance of synapse 3208 . In at least one embodiment, an instance of neuron 3202 that produces an output to be transmitted across an instance of synapse 3208 may be referred to as a "presynaptic neuron" with respect to that instance of synapse 3208. In at least one embodiment, a neuron 3202 that receives input transmitted across a synapse 3208 may be referred to as a "postsynaptic neuron" with respect to that synapse 3208. Because an instance of neuron 3202 can receive input from one or more instances of synapse 3208, and can also send outputs across one or more instances of synapse 3208, a single instance of neuron 3202 can therefore be both a “presynaptic neuron” and a “postsynaptic Neuron” in relation to various instances of synapse 3208 in at least one embodiment.

In at least one embodiment, neurons 3202 may be organized into one or more layers. Each instance of a neuron 3202 can have a neuron output 3206 that can fan out to one or more neuron inputs 3204 via one or more synapses 3208 . In at least one embodiment, neuron outputs 3206 of neurons 3202 in a first layer 3210 may be connected to neuron inputs 3204 of neurons 3202 in a second layer 3212. In at least one embodiment, layer 3210 may be referred to as a "feed-forward layer." In at least one embodiment, each instance of neuron 3202 in an instance of first layer 3210 may fan out to each instance of neuron 3202 in second layer 3212 . In at least one embodiment, the first layer 3210 may be referred to as a fully connected feed-forward layer. In at least one embodiment, each instance of a neuron 3202 in a second layer 3212 instance may fan out to fewer than all of the neuron 3202 instances in a third layer 3214 . In at least one embodiment, the second layer 3212 may be referred to as a "sparsely connected feed-forward layer." In at least one embodiment, neurons 3202 in second layer 3212 may fan out to neurons 3202 in multiple other layers, including neurons 3202 in the (same) second layer 3212. In at least one embodiment, second layer 3212 may be referred to as a "recurring layer." “ be designated. Neuromorphic processor 3200 may include, without limitation, any suitable combination of recurring layers and feed-forward layers, including without limitation both sparsely connected feed-forward layers and fully connected feed-forward layers.

In at least one embodiment, neuromorphic processor 3200 may include, without limitation, a reconfigurable connection architecture or dedicated hard-wired connections for connecting synapse 3208 to neurons 3202. In at least one embodiment, neuromorphic processor 3200 may include, without limitation, circuitry or logic that allows synapses to be mapped to different neurons 3202 as needed based on neural network topology and neuron fan-in/out. For example, in at least one embodiment, synapses 3208 may be connected to neurons 3202 using an interconnect fabric, such as network-on-chip, or with dedicated connections. In at least one Embodiment synapse connections and their components can be implemented by circuitry or logic.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a GPU as described above.

33 12 is a block diagram of a processing system, according to at least one embodiment. In at least one embodiment, the system 3300 includes one or more processors 3302 and one or more graphics processors 3308 and may be a uniprocessor desktop system, a multiprocessor workstation system, or a server system having a large number of processors 3302 or processor cores 3307 . In at least one embodiment, system 3300 is a processing platform integrated into a system-on-a-chip (SoC) integrated circuit for use in mobile, portable, or embedded devices.

In at least one embodiment, system 3300 may comprise a gaming console, including a gaming and media console, a mobile gaming console, a handheld gaming console, or an online gaming console, or may be integrated with a server-based gaming platform. In at least one embodiment, the system 3300 is a cell phone, a smart phone, a tablet computing device, or a mobile internet device. In at least one embodiment, the processing system 3300 may also include, be coupled to, or integrated with a wearable device, such as a wearable smart watch device, a smart eyewear device, an augmented reality device, or a virtual one reality device. In at least one embodiment, processing system 3300 is a television or set-top box device having one or more processors 3302 and a graphical interface generated by one or more graphics processors 3308 .

In at least one embodiment, one or more processors 3302 each include one or more processor cores 3307 for processing instructions that, when executed, perform system and application software operations. In at least one embodiment, each of one or more processor cores 3307 is configured to process a particular set of instructions 3309 . In at least one embodiment, instruction set 3309 may include complex instruction set computing (CISC), reduced instruction set computing (RISC), or very long instruction word (VLIW) computing ) enable. In at least one embodiment, processor cores 3307 may each process a different instruction set 3309, which may include instructions to facilitate emulation of other sets of instructions. In at least one embodiment, processor core 3307 may also include other processing devices, such as a digital signal processor (DSP).

In at least one embodiment, processor 3302 includes cache memory 3304. In at least one embodiment, processor 3302 may have a single internal cache or multiple internal cache levels. In at least one embodiment, the cache memory is shared between different components of the 3302 processor. In at least one embodiment, the processor 3302 also uses an external cache (e.g., a Level 3 (L3) cache or Last Level Cache (LLC)) (not shown) that is shared between the processor cores 3307 using known cache coherency techniques be able. In at least one embodiment, register file 3306 is additionally included in processor 3302, which may include various types of registers for storing various types of data (e.g., integer registers, floating point registers, status registers, and an instruction pointer register). In at least one embodiment, register file 3306 may include general purpose registers or other registers.

In at least one embodiment, processor(s) 3302 is/are coupled to interface bus(s) 3310 to transfer communication signals such as address, data, or control signals between processor 3302 and other components in system 3300. In at least one embodiment, interface bus 3310 may be a processor bus, such as a version of a Direct Media Interface (DMI) bus, in one embodiment. In at least one embodiment, interface 3310 is not limited to a DMI bus and may include one or more peripheral component interconnect buses (eg, PCI, PCI Express), memory buses, or other types of interface buses. In at least one embodiment, processor(s) 3302 includes an integrated memory controller 3316 and a platform control hub 3330. In at least In one embodiment, memory controller 3316 facilitates communication between a memory device and other components of system 3300, while platform control hub (PCH) 3330 provides connections to I/O devices via a local I/O bus.

In at least one embodiment, storage device 3320 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase change memory device, or another suitably performing memory device that serves as process memory. In at least one embodiment, device 3320 may operate as system memory for system 3300 to store data 3322 and instructions 3321 for use when one or more processors 3302 execute an application or process. In at least one embodiment, memory controller 3316 is also coupled to an optional external graphics processor 3312 that can communicate with one or more graphics processors 3308 in processors 3302 to perform graphics and media operations. In at least one embodiment, a display device 3311 may be coupled to the processor(s) 3302. In at least one embodiment, display device 3311 may include one or more internal display devices, such as a mobile electronic device or laptop, or an external display device connected via a display interface (e.g., DisplayPort, etc.). In at least one embodiment, the display device 3311 may comprise a head mounted display (HMD), e.g., a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

In at least one embodiment, platform control hub 3330 enables peripheral devices to be connected to memory controller 3320 and processor 3302 via a high-speed I/O bus. In at least one embodiment, the I/O peripherals include, but are not limited to, an audio controller 3346, a network controller 3334, a firmware interface 3328, a wireless transceiver (i.e., transmitter/receiver) 3326, touch sensors 3325, a data storage device 3324 (e.g., hard drive, flash memory, etc.). In at least one embodiment, data storage device 3324 may be connected via a storage interface (e.g., SATA) or via a peripheral bus, such as a peripheral component interconnect bus (e.g., PCI, PCI Express). In at least one embodiment, touch sensors 3325 may include touch screen sensors, pressure sensors, or fingerprint sensors. In at least one embodiment, the wireless transceiver 3326 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, or Long Term Evolution (LTE) transceiver. In at least one embodiment, firmware interface 3328 enables communication with system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, controller 3334 may enable network connection to a wired network. In at least one embodiment, a high performance network controller (not shown) is coupled to interface bus 3310. In at least one embodiment, controller 3346 is a multi-channel high definition audio controller. In at least one embodiment, the system 3300 includes an optional legacy I/O controller 3340 for coupling legacy devices (e.g., Personal System 2 (PS/2)) to the system. In at least one embodiment, the platform controller hub 3330 can also be connected to one or more Universal Serial Bus (USB) controllers 3342 to connect input devices such as a keyboard and mouse 3343 combo, a camera 3344, or other USB input devices.

In at least one embodiment, an instance of memory controller 3316 and platform control hub 3330 may be integrated into a discrete external graphics processor, such as external graphics processor 3312. In at least one embodiment, platform control hub 3330 and/or memory controller 3316 may be external to processor(s) 3302 . For example, in at least one embodiment, system 3300 may include an external memory controller 3316 and a platform control hub 3330, which may be configured as a memory controller hub and peripheral controller hub within a system chipset that is in communication with processor(s) 3302.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, some or all of the inference and/or training logic 1415 may be integrated into the graphics processor 3300 . For example, in at least one embodiment, the training and/or inference techniques described herein may use one or more ALUs included in 3D pipeline 3312 . In addition, in at least one embodiment the inferencing and/or training operations described here with a different logic than that in 14A or 15B logic shown are performed. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and/or registers (shown or not) that configure ALUs of graphics processor 3300 to implement one or more machine learning algorithms described herein , neural network architectures, use cases or training techniques.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a processor as described above.

34 10 is a block diagram of a processor 3400 having one or more processor cores 3402A-3402N, an integrated memory controller 3414, and an integrated graphics processor 3408, according to at least one embodiment. In at least one embodiment, processor 3400 may include additional cores, up to and including additional core 3402N, represented by dashed boxes. In at least one embodiment, each of the processor cores 3402A-3402N includes one or more internal cache units 3404A-3404N. In at least one embodiment, each processor core also has access to one or more shared cache units 3406.

In at least one embodiment, internal cache units 3404A-3404N and shared memory 3406 represent a cache memory hierarchy within processor 3400. In at least one embodiment, cache memory units 3404A-3404N may include at least one level of instruction and data cache within each processor core and include one or more levels of shared mid-level cache, such as B. Level-2 (L2), Level-3 (L3), Level-4 (L4) or other cache levels, where the highest cache level in front of the external memory is classified as LLC. In at least one embodiment, the cache coherency logic maintains coherency between different cache units 3406 and 3404A-3404N.

In at least one embodiment, processor 3400 may also include a set of one or more bus control units 3416 and a system service core 3410 . In at least one embodiment, one or more bus control units 3416 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system service core 3410 is provided for the functionality of managing various processor components. In at least one embodiment, the system service core 3410 includes one or more integrated storage controllers 3414 for managing access to various external storage devices (not shown).

In at least one embodiment, one or more of the processor cores 3402A-3402N include support for simultaneous multithreading. In at least one embodiment, the system service core 3410 includes components for coordinating and operating the cores 3402A-3402N during multi-threaded processing. In at least one embodiment, system service core 3410 may additionally include a power control unit (PCU) that includes logic and components for controlling one or more power states of processor cores 3402A-3402N and graphics processor 3408.

In at least one embodiment, processor 3400 additionally includes graphics processor 3408 for performing graphics processing operations. In at least one embodiment, graphics processor 3408 is coupled to shared cache units 3406 and system service core 3410, which includes one or more integrated memory controllers 3414. In at least one embodiment, the system service core 3410 also includes a display controller 3411 for driving the output of the graphics processor to one or more coupled displays. In at least one embodiment, display controller 3411 may also be a separate module that is coupled to graphics processor 3408 via at least one connection, or may be integrated with graphics processor 3408.

In at least one embodiment, a ring-based connection unit 3412 is used to couple internal processor 3400 components. In at least one embodiment, an alternative connection unit can be used, such as e.g. B. a point-to-point connection, a Switch connection or other techniques. In at least one embodiment, graphics processor 3408 is coupled to ring interconnect 3412 via an I/O interconnect 3413 .

In at least one embodiment, I/O connection 3413 represents at least one of several types of I/O connections, including an in-package I/O connection that enables communication between various processor components and a high-performance embedded memory module 3418, such as B. an eDRAM module allows. In at least one embodiment, each of processor cores 3402A-3402N and graphics processor 3408 uses embedded memory modules 3418 as a shared last-level cache.

In at least one embodiment, processor cores 3402A-3402N are homogeneous cores that execute a common instruction set architecture. In at least one embodiment, processor cores 3402A-3402N are instruction set architecture (ISA) heterogeneous, with one or more processor cores 3402A-3402N executing a common instruction set, while one or more other cores of processor cores 3402A-34-02N execute a subset of a common Execute instruction set or another instruction set. In at least one embodiment, processor cores 3402A-3402N are heterogeneous in terms of microarchitecture, with one or more relatively higher power consumption cores coupled with one or more lower power consumption performance cores. In at least one embodiment, the processor 3400 may be implemented on one or more chips or as a SoC integrated circuit.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, some or all of the inference and/or training logic 1415 may be integrated into the graphics processor 3410 . For example, in at least one embodiment, the techniques described herein for training and/or inferring may use one or more of the ALUs included in the 3D pipeline 3312, the graphics core(s) 3415A, the shared functional logic 3416, the graphics core(s). (en) 3415B, the shared function logic 3420 or other logic in 34 are included. In addition, in at least one embodiment, the inferencing and/or training operations described herein may be performed using a method other than that described in 14A or 14B shown logic are carried out. In at least one embodiment, weighting parameters may be stored in on-chip or off-chip memory and/or registers (shown or not shown) that configure ALUs of graphics processor 3410 to implement one or more machine learning algorithms, architectures, use cases or to carry out training techniques of neural networks.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a processor as described above.

35 Figure 12 is a block diagram of a 3500 graphics processor, which may be a discrete graphics processing unit or a graphics processor integrated into multiple processing cores. In at least one embodiment, graphics processor 3500 communicates with registers of graphics processor 3500 and with instructions input to memory via a memory-mapped I/O interface. In at least one embodiment, graphics processor 3500 includes a memory interface 3514 for memory access. In at least one embodiment, memory interface 3514 is an interface to local memory, to one or more internal caches, to one or more shared external caches, and/or to system memory.

In at least one embodiment, graphics processor 3500 also includes a display controller 3502 for controlling the output of display data to a display device 3520. In at least one embodiment, display controller 3502 includes hardware for one or more overlay layers for device 3520 and a composite of multiple layers of video or user interface elements. In at least one embodiment, display device 3520 may be an internal or external display device. In at least one embodiment, display device 3520 is a head-mounted display device, such as a virtual reality (VR) display device or an augmented reality (AR) display device. In at least one embodiment, the graphics processor comprises 3500 a video codec engine 3506 for encoding, decoding, or transcoding media to, from, or between one or more media encoding formats, including but not limited to MPEG (Moving Picture Experts Group) formats such as MPEG-2, AVC (Advanced Video Coding) such as H.264 /MPEG-4 AVC, as well as the Society of Motion Picture & Television Engineers (SMPTE) 421M/VC-1 and the Joint Photographic Experts Group (JPEG) formats such as JPEG and Motion JPEG (MJPEG) formats.

In at least one embodiment, graphics processor 3500 includes a block image transfer (BLIT) engine 3504 to perform two-dimensional (2D) rasterization operations including, for example, bit boundary block transfers. However, in at least one embodiment, the 2D graphics operations are performed using one or more graphics processing engine (GPE) 3510 components. In at least one embodiment, GPE 3510 is a computational engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

In at least one embodiment, the GPE 3510 includes a 3D pipeline 3512 for performing 3D operations, such as rendering three-dimensional images and scenes with processing functions that operate on 3D primitive shapes (e.g., rectangle, triangle, etc.). The 3D pipeline 3512 includes programmable and fixed functional elements that perform various tasks and/or spawn threads of execution on a 3D/media subsystem 3515 . While 3D pipeline 3512 may be used to perform media operations in at least one embodiment, GPE 3510 also includes a media pipeline 3516 that may be used to perform media operations such as B. video post-processing and image enhancement is used.

In at least one embodiment, media engine 3516 includes fixed functional units or programmable logic units for performing one or more specialized media operations, such as video decoding acceleration, video de-interlacing, and video encoding acceleration instead of or for video codec engine 3506. In at least one embodiment Additionally, the media pipeline 3516 includes a thread creation unit for creating threads for execution on the 3D/media subsystem 3515. In at least one embodiment, the threads created perform computations for media operations on one or more graphics execution units running in the 3D /Media subsystem 3515 are included.

In at least one embodiment, 3D/media subsystem 3515 includes logic for executing threads spawned by 3D pipeline 3512 and media pipeline 3516 . In at least one embodiment, 3D pipeline 3512 and media pipeline 3516 send thread execution requests to 3D/media subsystem 3515, which includes thread dispatch logic for arbitrating and dispatching various requests to available thread execution resources. In at least one embodiment, the execution resources include an array of graphics execution units for processing 3D and media threads. In at least one embodiment, 3D/Media subsystem 3515 includes one or more internal caches for thread instructions and data. In at least one embodiment, subsystem 3515 also includes shared memory, including registers and addressable memory, to share data between threads and to store output data.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, some or all of the inference and/or training logic 1415 may be integrated into the graphics processor 3500. For example, in at least one embodiment, the training and/or inference techniques described herein may use one or more ALUs included in the 3D pipeline 3512 . Furthermore, in at least one embodiment, the inferencing and/or training operations described herein may use logic other than that described in 14A or 15B illustrated logic are carried out. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and/or registers (shown or not) that configure ALUs of graphics processor 3500 to implement one or more machine learning algorithms described herein, Network architectures, use cases, or neural network training techniques.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the evaluation and generation networks of a robot control system can be implemented using a processor as described above.

36 16 is a block diagram of a graphics processing engine 3610 of a graphics processor, according to at least one embodiment. In at least one embodiment, the graphics processing engine (GPE) 3610 is a version of the 35 GPE 3510 shown. In at least one embodiment, media pipeline 3516 is optional and may not be included in GPE 3610 explicitly. In at least one embodiment, a separate media and/or image processor is coupled to GPE 3610.

In at least one embodiment, the GPE 3610 is coupled to or includes an instruction streamer 3603 that provides an instruction stream for the 3D pipeline 3512 and/or the media pipelines 3516 . In at least one embodiment, instruction streamer 3603 is coupled to memory, which may be system memory or one or more internal caches and shared caches. In at least one embodiment, instruction streamer 3603 receives instructions from memory and sends instructions to 3D pipeline 3512 and/or media pipeline 3516. In at least one embodiment, the instructions are instructions, primitives, or micro-operations that are retrieved from a circular buffer in which instructions for the 3D pipeline 3512 and the media pipeline 3516 are stored. In at least one embodiment, a ring buffer may additionally include batch instruction buffers storing batches of multiple instructions. In at least one embodiment, instructions for 3D pipeline 3512 may also include references to data stored in memory, such as, but not limited to, vertex and geometry data for 3D pipeline 3512 and/or image data and storage objects for media pipeline 3516. In In at least one embodiment, the 3D pipeline 3512 and media pipeline 3516 process commands and data by executing operations or sending one or more threads of execution to a graphics core array 3614 . In at least one embodiment, graphics core array 3614 includes one or more blocks of graphics cores (e.g., graphics core(s) 3615A, graphics core(s) 3615B), where each block includes one or more graphics cores. In at least one embodiment, each graphics core includes a set of graphics execution resources that includes general and graphics-specific execution logic for performing graphics and computational operations, as well as texture processing with fixed functionality and/or machine learning and artificial intelligence acceleration logic, including inference and /or training logic 1415 in 14A and 14B .

In at least one embodiment, 3D pipeline 3512 includes logic with fixed functionality and programmable logic to implement one or more shading programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders, or other shading programs. by processing commands and sending threads of execution to graphics core array 3614. In at least one embodiment, graphics core array 3614 provides a unified block of execution resources for use in processing shader programs. In at least one embodiment, the general execution logic (e.g., execution units) within the graphics core(s) 3615A-3715B of the graphics core array 3614 includes support for various 3D API shader languages and can execute multiple simultaneous threads of execution associated with multiple shaders are.

In at least one embodiment, the graphics core array 3614 also includes execution logic to perform media functions such as video and/or image processing. In at least one embodiment, in addition to the graphics processing operations, the execution units include general purpose logic that is programmable to perform parallel general purpose computing operations.

In at least one embodiment, output data generated by threads executing on the graphics core array 3614 may output data to memory in a unified return buffer (URB) 3618 . The URB 3618 can store data for multiple threads. In at least one embodiment, URB 3618 may be used to send data between different threads executing on graphics core array 3614. In at least one embodiment, URB 3618 may be used in addition to synchronization between threads on graphics core array 3614 and fixed functional logic within shared functional logic 3620.

In at least one embodiment, graphics core array 3614 is scalable such that graphics core array 3614 includes a variable number of graphics cores, each having a variable number of execution units based on a target and performance level of GPE 3610. At least In one embodiment, the execution resources are dynamically scalable so that the execution resources can be enabled or disabled as needed.

In at least one embodiment, graphics core array 3614 is coupled to shared functional logic 3620 that includes multiple resources shared by the graphics cores in graphics core array 3614 . In at least one embodiment, shared functions performed by shared function logic 3620 are embodied in hardware logic units that provide specific additional functions to graphics core array 3614 . In at least one embodiment, shared functional logic 3620 includes, but is not limited to, sampler 3621, math 3622, and inter-thread communication (ITC) 3623 logic. In at least one embodiment, one or more caches 3625 are included in or coupled to shared functional logic 3620 .

In at least one embodiment, a shared function is used when the demand for a specialized function is insufficient for graphics core array 3614 inclusion. In at least one embodiment, a single instantiation of a specialized function is used in shared function logic 3620 and shared by other execution resources within graphics core array 3614 . In at least one embodiment, specific shared functions within shared function logic 3620 that are widely used by graphics core array 3614 may be included in shared function logic 3616 within graphics core array 3614 . In at least one embodiment, shared functional logic 3616 within graphics core array 3614 may include some or all of the logic within shared functional logic 3620 . In at least one embodiment, all logic elements within shared functional logic 3620 may be duplicated in shared functional logic 3616 of graphics core array 3614 . In at least one embodiment, shared functional logic 3620 is eliminated in favor of shared functional logic 3616 within graphics core array 3614 .

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, some or all of the inference and/or training logic 1415 may be integrated into the graphics processor 3610 . For example, in at least one of the example embodiments, training and/or inference techniques described herein, one or more of the ALUs included in the 3D pipeline 3612, the graphics core(s) 3615A, the common functional logic 3616, the graphics core(s). (en) 3615B, the common functional logic 3620 or other logic in 36 are included. Furthermore, in at least one embodiment, the inferencing and/or training operations described herein may use logic other than that described in 14A or 15B logic shown are performed. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and/or registers (shown or not) that configure ALUs of graphics processor 3610 to run one or more machine learning algorithms described herein , network architectures, use cases or training techniques of neural networks.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a processor as described above.

37 10 is a block diagram of the hardware logic of a graphics processor core 3700, according to at least one embodiment described herein. In at least one embodiment, graphics processor core 3700 is included in a graphics core array. In at least one embodiment, graphics processor core 3700, sometimes referred to as a core section, may be one or more graphics cores within a modular graphics processor. In at least one embodiment, graphics processor core 3700 is exemplary of a graphics core portion, and a graphics processor as described herein may include multiple graphics core portions based on target power and performance envelopes. In at least one embodiment, each graphics core 3700 may include a fixed functional block 3730 coupled to multiple sub-cores 3701A-3701F, also referred to as subsections, which comprise modular blocks of common and fixed functional logic.

In at least one embodiment, fixed functionality block 3730 includes a fixed functionality geometry pipeline 3736 that can be shared among all sub-cores in graphics processor 3700, for example in lower performance and/or lower power consumption graphics processor implementations. In at least one embodiment, the fixed functionality geometry pipeline 3736 includes a 3D fixed functions pipeline, a video front end unit, a thread creator and thread dispatcher, and a unified return buffer manager that manages unified return buffers.

In at least one embodiment, the fixed functionality block 3730 also includes a graphics SoC interface 3737, a graphics microcontroller 3738, and a media pipeline 3739. The graphics SoC interface 3737 is an interface between the graphics core 3700 and other processor cores within of a system on an integrated circuit on a chip. In at least one embodiment, graphics microcontroller 3738 is a programmable subprocessor that is configurable to manage various functions of graphics processor 3700, including thread dispatcher, scheduler, and pre-emption. In at least one embodiment, media pipeline 3739 includes logic to facilitate decoding, encoding, pre-processing, and/or post-processing of multimedia data, including image and video data. In at least one embodiment, media pipeline 3739 implements media operations via requests for computation or sampling logic within sub-cores 3701-3701F.

In at least one embodiment, the SoC interface 3737 allows the graphics core 3700 to communicate with general application processor cores (e.g., CPUs) and/or other components within a SoC, including memory hierarchy elements such as a shared last-level cache, system RAM, and /or embedded on-chip or on-package DRAM. In at least one embodiment, the SoC interface 3737 may also enable communication with devices with fixed functions within a SoC, such as camera image processing pipelines, and enable the use and/or implementation of global memory building blocks (atoms) used by the graphics core 3700 and CPUs can be shared within a SoC. In at least one embodiment, the SoC interface 3737 may also implement power management controls for the graphics core 3700 and enable an interface between a clock domain of the graphics core 3700 and other clock domains within a SoC. In at least one embodiment, SoC interface 3737 enables receiving command buffers from a command streamer and a global thread dispatcher configured to provide commands and commands to each of one or more graphics cores within a graphics processor. In at least one embodiment, commands and instructions may be sent to media pipeline 3739 when media operations are to be performed, or to a fixed function geometry pipeline (e.g., fixed function geometry pipeline 3736, fixed function geometry pipeline 3714). , when graphics processing operations are to be performed.

In at least one embodiment, the graphics microcontroller 3738 can be configured to perform various scheduling and management tasks for the graphics core 3700. In at least one embodiment, graphics microcontroller 3738 may perform graphics and/or compute load scheduling for various parallel graphics engines within execution unit (EU) arrays 3702A-3702F and 3704A-3704F within sub-cores 3701A-3701F. In at least one embodiment, host software executing on a CPU core of a SoC including the graphics core 3700 can dispatch workloads to one of a plurality of graphics processor doorbells that invoke a scheduling operation on an appropriate graphics engine. In at least one embodiment, the scheduling operations include determining the workload to be executed next, transferring a workload to an instruction streamer, anticipating existing workloads executing on an engine, monitoring a workload's progress, and notifying the host software when a workload completes. In at least one embodiment, the graphics microcontroller 3738 may also enable low-power or idle states for the graphics core 3700 by allowing the graphics core 3700 to reset registers within the graphics core 3700 during low-power state transitions independent of an operating system and/or graphics driver software save and restore on a system.

In at least one embodiment, graphics core 3700 may have more or less than the depicted sub-cores 3701A-3701F, up to N modular sub-cores. In at least one embodiment, for each set of N sub-cores, the graphics core 3700 may also have shared functional logic 3710, shared and/or cache memory 3712, a fixed geometry pipeline Include functionality 3714, as well as additional fixed function logic 3716 to speed up various graphics and computational operations. In at least one embodiment, shared functional logic 3710 may include logic units (eg, scanners, math, and/or inter-thread communication logic) that may be shared by any of the N sub-cores within graphics core 3700 . Shared and/or cache memory 3712 may be a last level cache for N sub-cores 3701A-3701F within graphics core 3700 and may also serve as shared memory accessible by multiple sub-cores. In at least one embodiment, fixed functionality geometry pipeline 3714 may be included within fixed functional block 3730 in place of fixed functionality geometry pipeline 3736 and may include the same or similar logical units.

In at least one embodiment, graphics core 3700 includes additional fixed functionality logic 3716 that may include various fixed functionality acceleration logic for use by graphics core 3700 . In at least one embodiment, the additional logic with fixed functionality 3716 includes an additional geometry pipeline for use only in positional shading. In position-only shading there are at least two geometry pipelines, while in a full geometry pipeline within the fixed functionality geometry pipeline 3716, 3736 and a cull pipeline, which is a additional geometry pipeline that may be included in additional fixed functionality logic 3716 . In at least one embodiment, the fade-out pipeline is a lightweight version of a full geometry pipeline. In at least one embodiment, a full pipeline and a hide pipeline may run different instances of an application, with each instance having a separate context. In at least one embodiment, only positional shading can suppress long fading runs for discarded triangles, allowing shading to complete earlier in some cases. For example, in at least one embodiment, the hide pipeline logic within the additional fixed function logic 3716 can perform position shading in parallel with a main application and produce critical results generally faster than a full pipeline because the hide pipeline retrieves the position attribute of vertices and shades without perform rasterization and rendering of pixels into a frame buffer. In at least one embodiment, the hiding pipeline may use generated critical results to compute visibility information for all triangles, regardless of whether those triangles are hidden. In at least one embodiment, a complete pipeline (which in this case may be referred to as a rendering pipeline) may use the visibility information to skip hidden triangles and shade only the visible triangles, which are eventually passed to a rasterization phase.

In at least one embodiment, the additional fixed function logic 3716 may also include machine learning acceleration logic, such as fixed function matrix multiplication logic, for implementations involving machine learning optimizations or inference.

In at least one embodiment, within each graphics sub-core 3701A-3701F includes a set of execution resources that can be used to execute graphics, media, and compute operations in response to requests from the graphics pipeline, media pipeline, or shader programs. In at least one embodiment, the graphics sub-cores 3701A-3701F include multiple EU arrays 3702A-3702F, 3704A-3704F, thread distribution and inter-thread communication logic (TD/IC) 3703A-3703F, a 3D (e.g., texture 3705A-3705F scanner, 3706A-3706F media scanner, 3707A-3707F shading processor, and 3708A-3708F shared local memory (SLM). The EU arrays 3702A-3702F, 3704A-3704F each include multiple execution units, which are general purpose graphics processing units capable of performing floating point and integer/fixed point logic operations in support of graphics, media, or computational operations , including graphics, media, or compute shader programs. In at least one embodiment, the 3703A-3703F TD/IC logic performs local thread dispatch and thread control operations for execution units within a sub-core and facilitates communication between threads executing on execution units of a sub-core. In at least one embodiment, the 3D scanner 3705A-3705F can read texture or other 3D graphics data into memory. In at least one embodiment, the 3D scanner may read texture data differently based on a configured scan state and texture format associated with a given texture. In at least one embodiment, media scanner 3706A-3706F may perform similar operations based on of a type and format associated with media data. In at least one embodiment, each graphics sub-core 3701A-3701F may alternately include a unified 3D and media scanner. In at least one embodiment, threads executing on execution units within each sub-core 3701A-3701F can use shared local memory 3708A-3708F within each sub-core so that threads executing within a thread group can use a shared pool of On-chip memory can run.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, some or all of the inference and/or training logic 1415 may be integrated into the graphics processor 3710. For example, in at least one embodiment, the training and/or inference techniques described herein may utilize one or more ALUs resident in 3D pipeline 3710, graphics microcontroller 3738, fixed functionality geometry pipeline 3714 and 3736, or other logic in 35 are included. In addition, in at least one embodiment, the inferencing and/or training operations described herein may be performed with a method other than that described in 14A or 15B shown logic are carried out. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and/or registers (shown or not) that configure ALUs of graphics processor 3700 to implement one or more machine learning algorithms described herein , architectures, use cases or training techniques of neural networks.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a processor as described above.

38A-38B 12 show thread execution logic 3800 comprising an array of processing elements of a graphics processor core, according to at least one embodiment. 38A Figure 12 shows at least one embodiment in which thread execution logic 3800 is used. 38B 12 shows exemplary internal details of an execution unit according to at least one embodiment.

As in 38A As shown, the thread execution logic 3800 includes, in at least one embodiment, a shader processor 3802, a thread dispatcher 3804, the instruction cache 3806, a scalable array of execution units that includes a plurality of execution units 3808A-3808N, a scanner 3810, a data cache 3812, and a Data port 3814 included. In at least one embodiment, a scalable array of execution units may be dynamically scaled by adding one or more execution units (e.g., any of execution units 3808A, 3808B, 3808C, 3808D through 3808N-1 and 3808N) based on the computational needs of a workload can be enabled or disabled. In at least one embodiment, scalable execution units are interconnected by an interconnect fabric that connects to each of the execution units. In at least one embodiment, thread execution logic 3800 includes one or more connections to memory, such as: e.g., to system memory or cache memory, via one or more of instruction cache 3806, data port 3814, scanner 3810, and execution units 3808A-3808N. In at least one embodiment, each execution unit (eg, 3808A) is a self-contained programmable general purpose processing unit capable of executing multiple simultaneous hardware threads while processing multiple data items in parallel for each thread. In at least one embodiment, the array of execution units 3808A-3808N is scalable to include any number of individual execution units.

In at least one embodiment, execution units 3808A-3808N are used primarily to execute shading programs. In at least one embodiment, the shader processor 3802 may process different shader programs and dispatch threads of execution associated with shader programs via a thread dispatcher 3804 . In at least one embodiment, thread dispatcher 3804 includes logic for arbitrating thread initiation requests from graphics and media pipelines and instantiating requested threads on one or more execution units in execution units 3808A-3808N. For example, in at least one embodiment, a geometry pipeline may dispatch vertex, tessellation, or geometry shaders to thread execution logic for processing. In at least one According to the exemplary embodiment, the thread dispatcher 3804 can also process thread creation requests from executing shadowing programs at runtime.

In at least one embodiment, execution units 3808A-3808N support an instruction set that includes native support for many standard 3D graphics shading instructions such that shading programs from graphics libraries (e.g., Direct 3D and OpenGL) run with minimal translation. In at least one embodiment, the execution units support vertex and geometry processing (e.g., vertex programs, geometry programs, vertex shaders), pixel processing (e.g., pixel shaders, fragment shaders), and general processing (e.g., computations). - and Media Shader). In at least one embodiment, each of the execution units 3808A-3808N, comprising one or more arithmetic logic units (ALUs), is capable of executing multiple instruction SIMD (single instruction multiple data) execution, and the multi-threaded Operation enables an efficient execution environment despite higher latency memory accesses. In at least one embodiment, each hardware thread within each execution unit has a dedicated high bandwidth register file and an associated independent thread state. In at least one embodiment, execution occurs multiple times per clock on pipelines capable of performing integer, single and double precision floating point operations, SIMD branches, logical operations, transcendent operations, and other miscellaneous operations. In at least one embodiment, while waiting for data from memory or one of the shared functions, dependency logic within execution units 3808A-3808N causes a waiting thread to sleep until the requested data is returned. In at least one embodiment, while a waiting thread is idle, hardware resources may be used to process other threads. For example, in at least one embodiment, during a delay associated with a vertex shader operation, an execution unit may execute operations for a pixel shader, fragment shader, or other type of shader program including another vertex shader.

In at least one embodiment, each execution unit in execution units 3808A-3808N operates on arrays of data elements. In at least one embodiment, a number of data items is the "run size" or number of channels for an instruction. In at least one embodiment, an execution channel is a logical unit of execution for data element access, masking, and flow control within instructions. In at least one embodiment, a number of channels may be independent of a number of physical arithmetic logic units (ALUs) or floating point units (FPUs) for a particular graphics processor. In at least one embodiment, execution units 3808A-3808N support integer and floating point data types.

In at least one embodiment, an instruction set of an execution unit includes SIMD instructions. In at least one embodiment, different data items may be stored as a packed data type in a register, and the execution unit processes different items based on the data size of the items. For example, in at least one embodiment, when operating with a 256-bit wide vector, 256 bits of a vector are stored in a register and an execution unit operates on a vector as four separate 64-bit packed data elements (quad-word (QW )), eight separate 32-bit packed data items (Double Word (DW) size data items), sixteen separate 16-bit packed data items (Word (W) size data items), or thirty-two separate 8-bit data items (Double Word (W) size data items). Size bytes (B)). However, in at least one embodiment, different vector widths and register sizes are possible.

In at least one embodiment, one or more execution units may be combined into a merged execution unit 3809A-3809N with thread control logic (3807A-3807N) common to all merged EUs. In at least one embodiment, multiple EUs may be merged into an EU group. In at least one embodiment, each EU in a merged EU group can be configured to run a separate SIMD hardware thread. The number of EUs in a merged EU group may vary according to different embodiments. In at least one embodiment, different SIMD widths may be implemented per EU, including but not limited to SIMD8, SIMD 16, and SIMD32. In at least one embodiment, each merged graphics execution unit 3809A-3809N includes at least two execution units. For example, the merged execution unit includes 3809A in at least one Embodiment a first EU 3808A, a second EU 3808B and the control of the thread 3807A common to the first EU 3808A and the second EU 3808B. In at least one embodiment, thread control logic 3807A controls the threads executing on merged graphics execution unit 3809A such that each EU can execute within merged execution units 3809A-3809N using a common instruction pointer register.

In at least one embodiment, thread execution logic 3800 includes one or more internal instruction caches (e.g., 3806) to cache thread instructions for execution units. In at least one embodiment, one or more data caches (e.g., 3812) are included to cache thread data during thread execution. In at least one embodiment, a sampler 3810 is included to provide texture sampling for 3D operations and media sampling for media operations. In at least one embodiment, sampler 3810 includes specialized texture or media sampling functionality to process texture or media data during the sampling process before sampled data is provided to an execution unit.

During execution, in at least one embodiment, graphics and media pipelines send thread initiation requests to thread execution logic 3800 via thread creation and distribution logic. In at least one embodiment, once a set of geometric objects has been processed and rasterized into pixel data, Pixel processor logic (e.g. pixel shading logic, fragment shading logic, etc.) is called within the shading processor 3802 to calculate other output information and cause the results to be written to output surfaces (e.g. color buffers, depth buffers, stencil buffers, etc.). In at least one embodiment, a pixel shader or fragment shader calculates values of various vertex attributes to be interpolated across a rasterized object. In at least one embodiment, the pixel processor logic within the shading processor 3802 then executes a pixel or fragment shading program provided by an API (Application Programming Interface). In at least one embodiment, shader processor 3802 dispatches threads via thread dispatcher 3804 to an execution unit (e.g., 3808A) for execution of a shader program. In at least one embodiment, shading processor 3802 uses texture sampling logic in sampler 3810 to access texture data in memory-associated texture maps. In at least one embodiment, arithmetic operations on texture data and input geometry data calculate pixel color data for each geometric fragment or exclude one or more pixels from further processing.

In at least one embodiment, data port 3814 provides a memory access mechanism for thread execution logic 3800 to output processed data to memory for further processing on a graphics processor output pipeline. In at least one embodiment, data port 3814 includes or is coupled to one or more cache memories (e.g., data cache 3812) to cache data for memory access via a data port.

As in 38B As shown, a graphics execution unit 3808, in at least one embodiment, may include an instruction fetch unit 3837, a general register file array (GRF) 3824, an architecture register file array (ARF) 3826, a thread arbiter 3822, a transmit unit 3830, a branch unit 3832, a set of SIMD floating point units (FPUs) 3834, and in at least one embodiment a set of dedicated integer SIMD ALUs 3835. In at least one embodiment, register files GRF 3824 and ARF 3826 include a set of general register files and architectural register files associated with each concurrent hardware thread that may be active in graphics execution unit 3808. In at least one embodiment, per-thread architecture state is maintained in ARF 3826 while GRF 3824 stores data used during thread execution. In at least one embodiment, the execution state of each thread, including the instruction pointers for each thread, may be held in ARF 3826 in thread-specific registers.

In at least one embodiment, the graphics execution unit 3808 has an architecture that is a combination of Simultaneous Multi-Threading (SMT) and Fine-Grain Interleaved Multi-Threading (IMT). In at least one embodiment, the architecture has a modular configuration that can be fine-tuned at design time based on a target number of simultaneous threads and a number of registers per execution unit, where execution unit resources are allocated to the logic needed to execute multiple simultaneous threads is used.

In at least one embodiment, graphics execution unit 3808 may concurrently issue multiple instructions, each of which may be different instructions. In at least one embodiment, thread arbiter 3822 of graphics execution unit thread 3808 may send instructions to one of dispatch unit 3830, branch unit 3842, or SIMD FPU(s) 3834 for execution. In at least one embodiment, each execution thread can access 128 general purpose registers in the GRF 3824, with each register capable of storing 32 bytes accessible as a SIMD 8-element vector of 32-bit data elements. In at least one embodiment, each execution unit thread has access to 4K bytes within the GRF 3824, although the execution units are not so limited and in other embodiments more or fewer register resources may be provided. In at least one embodiment, up to seven threads may be executing concurrently, although the number of threads per execution unit may also vary by embodiment. In at least one embodiment, where seven threads can access 4K bytes, the GRF 3824 can store a total of 27K bytes. In at least one embodiment, flexible addressing modes may allow registers to be addressed together to effectively build wider registers or to represent dashed rectangular block data structures.

In at least one embodiment, memory operations, scanner operations, and other higher latency system communications are dispatched via "send" commands executed by the messaging sending unit 3830 . In at least one embodiment, branch instructions are sent to a dedicated branch unit 3832 to facilitate SIMD divergence and eventual convergence.

In at least one embodiment, graphics execution unit 3808 includes one or more SIMD floating point units (FPU(s)) 3834 for performing floating point operations. In at least one embodiment, FPU(s) 3834 also support integer math. In at least one embodiment, FPU(s) 3834 may perform SIMD up to M 32-bit floating point (or integer) operations, or SIMD up to 2M 16-bit integer or 16-bit floating point operations. In at least one embodiment, at least one of the FPU(s) is equipped with extended math functions to support high throughput transcendent math functions and 64-bit double-precision floating-point operations. In at least one embodiment, there is also a set of 8-bit integer SIMD ALUs 3835 that can be specifically optimized to perform operations associated with machine learning computations.

In at least one embodiment, arrays of multiple instances of graphics core array 3808 may be instantiated into a graphics sub-core grouping (e.g., subsection). In at least one unit of execution unit 3808 in at least one embodiment, instructions may be executed through a variety of execution channels. In at least one embodiment, each thread executing on graphics execution unit 3808 executes in a different channel.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, some or all of the inference and/or training logic 1415 may be incorporated into the execution logic 3800 . Additionally, in at least one embodiment, inferencing and/or training operations described herein may use logic other than that described in 14A or 15B logic shown can be used. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and/or registers (shown or not) that configure ALUs of execution logic 3800 to implement one or more machine learning algorithms described herein , architectures, use cases or training techniques for neural networks.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a processor as described above.

39 12 shows a parallel processing unit (“PPU”) 3900 according to at least one embodiment. In at least one embodiment, PPU 3900 is configured with machine-readable code that, when executed by PPU 3900, causes PPU 3900 to execute some or all of the Execute processes and techniques described in the disclosure. In at least one embodiment, PPU 3900 is a multithreaded processor implemented on one or more integrated circuit devices and uses multithreading as a latency reduction technique designed to execute computer-readable instructions (also referred to as machine-readable instructions or just instructions) on multiple threads to process in parallel. In at least one embodiment, a thread refers to a thread of execution and is an instantiation of an instruction set configured to be executed by PPU 3900. In at least one embodiment, PPU 3900 is a graphics processing unit ("GPU") configured to implement a graphics rendering pipeline for processing three-dimensional ("3D") graphics data to generate two-dimensional ("2D") image data for the generate a display on a display device, such as a liquid crystal display ("LCD") device. In at least one embodiment, PPU 3900 is used to perform computations such as linear algebra operations and machine learning operations. 39 Figure 13 illustrates an example of a parallel processor for purposes of illustration only and is intended as a non-limiting example of processor architectures contemplated by this disclosure, where any suitable processor may be substituted and/or supplemented.

In at least one embodiment, one or more PPUs 3900 are configured to accelerate high performance computing ("HPC"), data centers, and machine learning applications. In at least one embodiment, the PPU 3900 is configured to accelerate systems and applications for deep learning, including but not limited to the following: autonomous vehicle platforms, deep learning, high accuracy speech, image, text recognition systems, intelligent video analytics, molecular simulations, Drug discovery, disease diagnosis, weather forecasting, big data analysis, astronomy, molecular dynamics simulation, financial modeling, robotics, factory automation, real-time language translation, online search optimizations and personalized user recommendations, and much more.

In at least one embodiment, the PPU 3900 includes, without limitation, an input/output ("I/O") unit 3906, a front-end unit 3910, a scheduler unit 3912, a work distribution unit 3914, a hub 3916, a crossbar ("Xbar ’) 3920, one or more general purpose processing clusters (“GPCs”) 3918, and one or more partition units (“memory partition units”) 3922. In at least one embodiment, the PPU 3900 communicates with a host processor or other PPUs 3900 via one or more High Speed GPU Connections (“GPU Connections”) 3908 connected. In at least one embodiment, PPU 3900 is connected to a host processor or other peripheral devices via interconnect 3902 . In at least one embodiment, PPU 3900 is coupled to local storage, which includes one or more storage devices (“storage”) 3904 . In at least one embodiment, memory devices 3904 include, without limitation, one or more Dynamic Random Access Memory ("DRAM") devices. In at least one embodiment, one or more DRAM devices are configured and/or configurable as high-bandwidth memory (“HBM”) memory subsystems, with multiple DRAM chips stacked in each device.

In at least one embodiment, GPU high-speed link 3908 may refer to a wired, multi-lane communication link used by systems to scale and incorporate one or more PPUs 3900 in combination with one or more central processing units ("CPUs") that Cache coherency between PPUs 3900 and CPUs as well as CPU mastering supported. In at least one embodiment, data and/or commands are transmitted through the GPU high-speed interconnect unit 3908 via the hub 3916 to/from other units of the PPU 3900, such as one or more copy engines, video encoders, video decoders, power management units, and other components. in the 39 cannot be shown explicitly.

In at least one embodiment, I/O unit 3906 is configured to receive communications (e.g., commands, data) from a host processor (not in 39 shown) transmits and receives over the 3902 system bus. In at least one embodiment, I/O unit 3906 communicates with the host processor directly over system bus 3902 or through one or more intermediary devices such as a memory bridge. In at least one embodiment, I/O unit 3906 may communicate with one or more other processors, such as one or more PPUs 3900, via system bus 3902. In at least one embodiment, I/O unit 3906 implements a Peripheral Component Interconnect Express ("PCIe") interface for communicating over a PCIe Bus. In at least one embodiment, I/O unit 3906 implements interfaces for communicating with external devices.

In at least one embodiment, I/O unit 3906 decodes packets received over system bus 3902 . In at least one embodiment, at least some packets represent instructions configured to cause PPU 3900 to perform various operations. In at least one embodiment, I/O unit 3906 transmits decoded commands to various other units of PPU 3900 as specified by commands. In at least one embodiment, commands are transmitted to the front end unit 3910 and/or the hub 3916 or other units of the PPU 3900 such as one or more copy engines, a video encoder, a video decoder, a power management unit, and so forth. (not expressly in 39 shown). In at least one embodiment, I/O unit 3906 is configured to route communications between and between different PPU 3900 logical units.

In at least one embodiment, a program executed by the host processor encodes an instruction stream in a buffer that provides PPU 3900 workloads for processing. In at least one embodiment, a workload includes instructions and data to be processed by those instructions. In at least one embodiment, the buffer is an area in memory that is accessible (e.g., read/write) by both the host processor and the PPU 3900 - a host interface unit may be configured to handle memory requests that exceed the system bus 3902 from the I/O unit 3906 via the system bus 3902 accesses the buffer in a system memory connected to the system bus 3902. In at least one embodiment, the host processor writes an instruction stream to the buffer and then sends a pointer to the beginning of the instruction stream to PPU 3900 so that front end unit 3910 receives pointers to one or more instruction streams and manages one or more instruction streams, instructions reads from command streams and forwards commands to various units of PPU 3900.

In at least one embodiment, the front-end unit 3910 is coupled to the scheduler unit 3912, which configures various GPCs 3918 to process tasks defined by one or more instruction streams. In at least one embodiment, the scheduler unit 3912 is configured to track status information about various tasks managed by the scheduler unit 3912, where status information may indicate which of the GPCs 3918 a task is associated with, whether the task is active or inactive, the priority level associated with the task is associated, etc. In at least one embodiment, scheduler unit 3912 schedules execution of a variety of tasks on one or more GPCs 3918.

In at least one embodiment, the scheduler unit 3912 is coupled to the work distribution unit 3914, which is configured to distribute tasks to GPCs 3918 for execution. In at least one embodiment, the work distribution unit 3914 keeps track of a number of scheduled tasks received from the scheduler unit 3912, and the work distribution unit 3914 maintains a pending task pool and an active task pool for each of the GPCs 3918. In at least one embodiment, the pending task pool includes a number of slots (e.g., 32 slots) containing tasks assigned for processing by a particular GPC 3918; the active task pool may include a number of slots (e.g., 4 slots) for tasks that are being actively processed by the GPCs 3918 such that when one of the GPCs 3918 completes execution of a task, that task is removed from the active task pool for GPC 3918 is preempted and one of the other tasks is selected from the pool of pending tasks and scheduled to run on GPC 3918.

In at least one embodiment, when an active task is idle on GPC 3918, e.g. while waiting for a data dependency to be resolved, the active task is evicted from GPC 3918 and returned to the pool of pending tasks while another task is removed from the pool of pending tasks is selected and scheduled to run on GPC 3918.

In at least one embodiment, the work distribution unit 3914 communicates with one or more GPCs 3918 via the XBar 3920. In at least one embodiment, the XBar 3920 is an interconnection network that couples many units of the PPU 3900 to other units of the PPU 3900 and can be configured to the work distribution unit 3914 is paired with a particular GPC 3918 . In at least one embodiment, one or more other units of PPU 3900 may also be connected to XBar 3920 via hub 3916 .

In at least one embodiment, the tasks are managed by the scheduler unit 3912 and dispatched to one of the GPCs 3918 by the work distribution unit 3914 . GPC 3918 is configured to process tasks and produce results. In at least one embodiment, the results may be used by other tasks within GPC 3918, forwarded to another GPC 3918 via XBar 3920, or stored in memory 3904. In at least one embodiment, results may be written to memory 3904 via partition units 3922 that implement a memory interface for reading and writing data to/from memory 3904 . In at least one embodiment, results can be transmitted to another PPU 3904 or CPU via the GPU high speed link 3908 . In at least one embodiment, the PPU 3900 includes, without limitation, a number U of partition units 3922 equal to the number of separate and distinct storage devices 3904 coupled to the PPU 3900 . In at least one embodiment, the partition unit 3922 is used in conjunction with 41 described in more detail herein.

In at least one embodiment, a host processor runs a driver kernel that implements an application programming interface ("API") that allows one or more applications running on the host processor to schedule operations for execution on the PPU 3900. In at least one embodiment, multiple computing applications are executed concurrently by PPU 3900, and PPU 3900 provides isolation, quality of service ("QoS"), and independent address spaces for multiple computing applications. In at least one embodiment, an application generates commands (e.g., in the form of API calls) that cause the driver kernel to generate one or more tasks for execution by the PPU 3900, and the driver kernel issues tasks to one or more data streams , which are processed by the PPU 3900. In at least one embodiment, each task includes one or more groups of related threads, which may be referred to as a warp. In at least one embodiment, a warp includes a plurality of related threads (eg, 32 threads) that may execute in parallel. In at least one embodiment, cooperating threads may refer to a plurality of threads that contain instructions to perform tasks and that exchange data over a shared memory. In at least one embodiment, threads and cooperating threads are described in accordance with at least one embodiment in connection with 41 described in more detail.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, a deep learning application processor is used to train a machine learning model, such as a neural network, to predict or infer the information provided to PPU 3900 . In at least one embodiment, the deep learning application processor 3900 is used to infer or predict information based on a trained machine learning model (e.g., a neural network) trained by another processor or system or by the PPU 3900. In at least one embodiment, PPU 3900 may be used to perform one or more of the neural network use cases described herein.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a processor as described above.

40 12 shows a general processing cluster (“GPC”) 4000 in accordance with at least one embodiment. In at least one embodiment, GPC 4000 is GPC 3918 off 39 . In at least one embodiment, each GPC 4000 includes, without limitation, a number of hardware units for processing tasks, and each GPC 4000 includes, without limitation, a pipeline manager 4002, a pre-raster operations unit ("PROP") 4004, a raster engine 4008 , a work distribution crossbar (“WDX”) 4016, a memory management unit (“MMU”) 4018, one or more data processing clusters (“DPCs”) 4006 and any suitable combination of elements.

In at least one embodiment, the operation of GPC 4000 is controlled by pipeline manager 4002 . In at least one embodiment, pipeline manager 4002 manages the configuration of one or more DPCs 4006 for processing GPC 4000 assigned tasks. in minutes In at least one embodiment, pipeline manager 4002 configures at least one of one or more DPCs 4006 to implement at least a portion of a graphics rendering pipeline. In at least one embodiment, DPC 4006 is configured to run a shading program on a streaming programmable multiprocessor ("SM") 4014 . In at least one of the embodiments, the pipeline manager 4002 is configured to forward packets received from a work distribution unit to appropriate logical units within GPC 4000, and some packets may be sent to hardware units with fixed functionality in PROP 4004 and/or the grid Engine 4008 may be forwarded, while other packets may be forwarded to DPCs 4006 for processing by a primitive engine 4012 or SM 4014. In at least one embodiment, the pipeline manager 4002 configures at least one of the DPCs 4006 to implement a neural network model and/or a computational pipeline.

In at least one embodiment, PROP unit 4004 is configured to forward the data generated by engine 4008 and DPCs 4006 to a raster operations ("ROP") unit in partition unit 3922, described above in connection with 39 is described in more detail. In at least one embodiment, the PROP unit 4004 is configured to perform color mixing optimizations, organize pixel data, perform address translations, and more. In at least one embodiment, raster engine 4008 includes, without limitation, a set of fixed functionality hardware units configured to perform various raster operations, and raster engine 4008 includes, without limitation, a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine rasterization engine, a tile coalescing engine, and any suitable combination thereof. In at least one embodiment, the setup engine receives transformed vertices and generates plane equations associated with a geometric primitive defined by vertices; Plane equations are passed to the coarse raster engine to generate coverage information (eg, an x,y coverage mask for a tile) for the primitive; the output of the coarse raster engine is sent to the culling engine, where fragments associated with the primitive that fail a z-test are hidden, and to a clipping engine, where fragments that fall outside a Visibility cone are cut off. In at least one embodiment, the fragments that survive clipping and culling are passed to the fine raster engine to generate attributes for pixel fragments based on the plane equations generated by the setup engine. In at least one embodiment, the output of engine 4008 includes fragments that are processed by any suitable entity, such as a fragment shader implemented in DPC 4006 .

Each DPC 4006 included in GPC 4000 consists, in at least one embodiment, of, without limitation, an M-Pipe Controller ("MPC") 4010; the primitive engine 4012; one or more SMs 4014 and any suitable combination thereof. In at least one embodiment, MPC 4010 controls the operation of DPC 4006 and forwards packets received from pipeline manager 4002 to appropriate entities in DPC 4006 . In at least one embodiment, packets associated with a vertex are forwarded to primitive engine 4012, which is configured to fetch vertex attributes associated with the vertex from memory; in contrast, packets associated with a shader program may be forwarded to SM 4014.

In at least one embodiment, without limitation, SM 4014 includes a programmable streaming processor configured to process tasks represented by a number of threads. In at least one embodiment, SM 4014 is multi-threaded and configured to execute the plurality of threads (e.g., 32 threads) from a particular set of threads concurrently and adopts a single-threaded (SIMD) architecture. Instruction, Multiple-Data"), where each thread in a group of threads (e.g., a warp) is configured to process a different set of data based on the same instruction set. In at least one embodiment, all threads in a group of threads execute the same instructions. In at least one embodiment, SM 4014 implements a single-instruction, multiple-thread (SIMT) architecture, where each thread in a group of threads is configured to process a different set of data based on the same instruction set, but with individual threads in a group of threads are allowed to diverge during execution. In at least one embodiment, a program counter, call stack, and execution state are maintained for each warp, allowing for concurrency between warps and serial execution within warps when threads diverge within a warp. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, allowing equal concurrency between all threads within and between warps. In at least one embodiment, execution state is maintained for each individual thread, and threads executing the same instructions can be converged and executed in parallel for better efficiency. At least one embodiment of SM 4014 is described in more detail herein.

In at least one embodiment, MMU 4018 provides an interface between GPC 4000 and the memory partition unit (eg, partition unit 3922 of 39 ) and MMU 4018 provides virtual address to physical address translation, memory protection, and arbitration of memory requests. In at least one embodiment, MMU 4018 provides one or more translation lookaside buffers ("TLBs") for translation of virtual addresses to physical addresses in memory.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, a deep learning application processor is used to train a machine learning model, such as a neural network, to predict or infer information provided to GPC 4000. In at least one embodiment, GPC 4000 is used to infer or predict information based on a trained machine learning model (eg, a neural network) trained by another processor or system or by GPC 4000 . In at least one embodiment, GPC 4000 may be used to perform one or more of the neural network use cases described herein.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a processor as described above.

41 12 shows a memory partition unit 4100 of a parallel processing unit ("PPU"), according to at least one embodiment. In at least one embodiment, the memory partition unit 4100 includes, without limitation, a raster operations unit ("ROP") 4102, an L2 cache ("L2") 4104, a memory interface 4106, and any suitable combination thereof . The memory interface 4106 is coupled to the memory. The memory interface 4106 may implement 32, 64, 128, 1024 bit data buses or the like for high speed data transfer. In at least one embodiment, the PPU U includes memory interfaces 4106, one memory interface 4106 per pair of partition units 4100, with each pair of partition units 4100 being connected to a corresponding device. For example, in at least one embodiment, the PPU may be coupled to up to Y memory devices, such as high-bandwidth memory stacks or double graphics data rate memory, version 5, synchronous dynamic random access memory ("GDDR5 SDRAM").

In at least one embodiment, memory interface 4106 implements a high-bandwidth second generation memory interface ("HBM2"), where Y equals half of U . In at least one embodiment, the HBM2 memory stacks reside in the same package as the PPU, providing significant power and area savings compared to traditional GDDR5 SDRAM systems. In at least one embodiment, each HBM2 stack includes, without limitation, four memory devices and Y equals 4, with each HBM2 stack including two 128-bit channels per device for a total of 8 channels and a data bus width of 1024 bits. In at least one embodiment, the memory supports Single-Error Correcting Double-Error Detecting ("SECDED") Error Correction Code ("ECC") to protect the data. ECC provides higher reliability for computing applications that are sensitive to data corruption.

In at least one embodiment, the PPU implements a multi-level memory hierarchy. In at least one embodiment, memory partition unit 4100 supports unified memory to provide a single unified virtual address space for central processing unit ("CPU") and PPU memory, thereby enabling data sharing across virtual memory systems. In at least one embodiment, the frequency of accesses by a PPU to memory residing on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU that is accessing pages more frequently. In at least one In the exemplary embodiment, the GPU high-speed interconnect 3908 supports address translation services that allow the PPU to directly access a CPU's page tables and allow the PPU full access to CPU memory.

In at least one embodiment, the copy engines transfer data between multiple PPUs or between PPUs and CPUs. In at least one embodiment, copy engines can generate page faults for addresses not mapped into page tables, and memory partition unit 4100 then services page faults by mapping addresses into page tables, after which the copy engine performs the transfer. In at least one embodiment, memory for multiple copy engine operations is pinned (i.e., non-swappable) between multiple processors, thereby significantly reducing available memory. In at least one embodiment with hardware page faults, addresses can be passed to the copy engines regardless of whether memory pages are resident, and the copy process is transparent.

Data from memory 3904 from 39 or other system memory are retrieved from the memory partition unit 4100 and stored in the on-chip L2 cache 4104 shared between different GPCs according to at least one embodiment. Each memory partition unit 4100 includes, without limitation, at least a portion of the L2 cache associated with a corresponding memory device, in at least one embodiment. In at least one embodiment, lower level caches are implemented in various entities within GPCs. In at least one embodiment, each of the SMs 4014 may implement a level 1 ("L1") cache, where the L1 cache is private memory associated with a particular SM 4014, and data is fetched from the L2 cache 4104 and stored in each of the L1 caches for processing in SMs 4014 functional units. In at least one embodiment, L2 cache 4104 is coupled to memory interface 4106 and XBar 3920 .

The ROP unit 4102, in at least one embodiment, performs graphical rasterization operations related to pixel colors, such as color compression, pixel blending, and more. ROP unit 4102, in at least one embodiment, performs a depth check in conjunction with rasterization engine 4008 and receives a depth for a sample location associated with a pixel fragment from rasterization engine 4008's masking engine. In at least one embodiment, the depth is tested against a corresponding depth in a depth buffer for a sample location associated with the fragment. In at least one embodiment, if the fragment passes the depth test for the sample location, the ROP unit 4102 updates the depth buffer and transmits the result of the depth test to the raster engine 4008. Note that the number of partition units 4100 depends on the number of GPCs can differ and therefore each ROP unit 4102 can be coupled to each of the GPC in at least one embodiment. In at least one embodiment, ROP unit 4102 tracks packets received from different GPCs and determines which packets a result generated by ROP unit 4102 is forwarded via XBar 3920 to.

42 12 shows a streaming multiprocessor ("SM") 4200, according to at least one embodiment. In at least one embodiment, SM 4200 is the SM of 40 . In at least one embodiment, SM 4200 includes, without limitation, instruction cache 4202; one or more scheduler units 4204; a register file 4208; one or more processing cores (“Cores”) 4210; one or more Special Function Units ("SFUs") 4212; one or more load/store units ("LSUs") 4214; an interconnection network 4216; a shared memory/level 1 ("L1") cache 4218; and any suitable combination thereof. In at least one embodiment, a work distribution unit distributes tasks for execution across general processing clusters ("GPCs") of parallel processing units ("PPUs"), each task associated with a specific data processing cluster ("DPC") within a GPC and, if the task is associated with a shader program , assigned to one of the SMs 4200. In at least one embodiment, scheduler unit 4204 receives tasks from the work distribution unit and manages instruction scheduling for one or more thread blocks associated with SM 4200. In at least one embodiment, scheduler unit 4204 schedules thread blocks for execution as parallel thread warps, with each thread block having at least one warp associated therewith. In at least one embodiment, each warp executes threads. In at least one embodiment, scheduler unit 4204 manages a variety of different thread blocks, assigns the warps to different thread blocks, and then dispatches instructions from a variety of different cooperative groups to different functional units (e.g., processing cores 4210, SFUs 4212, and LSUs 4214) during each clock cycle. .

In at least one embodiment, cooperative groups can refer to a programming model for organizing groups of communicating threads that allows developers to specify the granularity at which the threads communicate, allowing for the design of richer, more efficient parallel decompositions allows. In at least one embodiment, cooperative launch APIs support synchronization between thread blocks for executing parallel algorithms. In at least one embodiment, applications of conventional programming models are provided with a single, simple construct for synchronizing cooperating threads: a barrier across all threads of a thread block (eg, the syncthreads() function). However, in at least one embodiment, programmers can define groups of threads with a granularity smaller than that of a thread block and synchronize within the defined groups to provide greater performance, design flexibility, and software reuse in the form of collective group-wide functional interfaces enable. In at least one embodiment, cooperative groups allow programmers to explicitly define groups of threads at sub-block (ie, as small as a single thread) and multi-block granularities and perform collective operations such as synchronizing threads in a cooperative group. The programming model supports clean composition across software boundaries, allowing libraries and utility functions to be safely synchronized within their local context without making assumptions about convergence. In at least one embodiment, the cooperative set primitives enable new patterns of cooperative parallelism, including, without limitation, producer-consumer parallelism, opportunistic parallelism, and global synchronization across an entire lattice of thread blocks.

In at least one embodiment, dispatch unit 4206 is configured to dispatch instructions to one or more functional units, and scheduler unit 4204 includes, without limitation, two dispatch units 4206 that allow two different instructions from the same chain to be scheduled during each clock cycle. In at least one embodiment, each scheduler unit 4204 includes a single distribution unit 4206 or additional distribution units 4206.

In at least one embodiment, each SM 4200 includes a register file 4208 that provides a set of registers for SM 4200 functional units. In at least one embodiment, register file 4208 is partitioned among each of the functional units such that each functional unit is allocated a specific portion of register file 4208. In at least one embodiment, register file 4208 is partitioned among various warps executed by SM 4200, and register file 4208 is provided as temporary storage for operands associated with functional unit data paths. In at least one embodiment, each SM 4200 includes, without limitation, a plurality of L processing cores 4210. In at least one embodiment, SM 4200 includes, without limitation, a large number (e.g., 128 or more) of different processing cores 4210. In at least one embodiment, each processing core 4210 includes in at least one embodiment, without limitation, a fully pipelined, single-precision, double-precision, and/or mixed-precision processing unit that includes, without limitation, a floating point arithmetic logic unit and an integer arithmetic logic unit. In at least one embodiment, floating point arithmetic logic units implement the IEEE 754-2008 standard for floating point arithmetic. In at least one embodiment, the processing cores 4210 include, without limitation, 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

Tensor cores are configured to perform matrix operations, according to at least one embodiment. In at least one embodiment, processing cores 4210 include one or more tensor cores. In at least one embodiment, the tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inference. In at least one embodiment, each tensor core operates on a 4x4 matrix and performs a matrix multiplication and accumulation operation D = A X B + C, where A, B, C, and D are 4x4 matrices.

In at least one embodiment, the inputs A and B to the matrix multiplication are 16-bit floating point matrices and the accumulation matrices C and D are 16-bit floating point or 32-bit floating point matrices. In at least one embodiment, the tensor cores operate on 16-bit floating point input data with 32-bit floating point accumulation. In at least one embodiment, 16-bit floating point multiplication uses 64 operations and results in a full precision product, which is then combined using 32-bit floating point addition with other intermediate products for a 4x4x4 mat rix multiplication is accumulated. Tensor cores are used to perform much larger two-dimensional or higher-dimensional matrix operations built from these smaller elements, in at least one embodiment. In at least one embodiment, an API, such as the CUDA 9 C++ API, provides special matrix load, matrix multiply and accumulate, and matrix store operations to efficiently use tensor cores from a CUDA C++ program. In at least one embodiment, the CUDA-level warp-level interface accepts 16x16 arrays spanning all 32 threads of the warp.

In at least one embodiment, each SM 4200 includes, without limitation, M SFUs 4212 that perform specific functions (e.g., attribute scoring, reciprocal square root, and the like). In at least one embodiment, SFUs 4212 include, without limitation, a tree traversal unit configured to traverse a hierarchical tree data structure. In at least one embodiment, without limitation, SFUs 4212 include a texture unit configured to perform texture map filtering operations. In at least one embodiment, the texture units are configured to load texture maps (e.g., a 2D array of texels) from memory and sample texture maps to generate sampled texture values for use in shading programs executed by SM 4200. In at least one embodiment, the texture maps are stored in shared memory/L1 cache 4218. In at least one embodiment, the texture units implement texture operations, such as filtering operations, using mip-maps (e.g., texture maps with varying levels of detail), in accordance with at least one embodiment. In at least one embodiment, each SM 4200 includes, without limitation, two texture units.

Each SM 4200 includes, without limitation, N LSUs 4214 that implement load and store operations between shared memory/L1 cache 4218 and register file 4208, according to at least one embodiment. Each SM 4200 includes, without limitation, interconnection network 4216 connecting each of the functional units to register file 4208 and LSU 4214 to register file 4208, and shared memory/L1 cache 4218, according to at least one embodiment. In at least one embodiment, interconnection network 4216 is a crossbar that can be configured to connect each of the functional units to each of the registers in register file 4208 and the LSUs 4214 to register file 4208 and locations in shared memory/L1 cache 4218 connects.

In at least one embodiment, shared memory/L1 cache 4218 is an array of on-chip memory that enables data storage and communication between SM 4200 and the primitive engine, and between threads within SM 4200, according to at least one embodiment an embodiment. In at least one embodiment, shared memory/L1 cache 4218 is, without limitation, 128 KB of storage capacity and is located in the path from SM 4200 to the partition unit. In at least one embodiment, shared memory/L1 cache 4218 is used to cache reads and writes. In at least one embodiment, one or more of shared memory/L1 cache 4218, L2 cache, and memory is backing memory.

Combining the functionalities of data cache and shared memory in a single block of memory provides improved performance for both types of memory access, in accordance with at least one embodiment. In at least one embodiment, a capacity is or is usable as a cache by programs that do not use shared memory, e.g., if the shared memory is configured to use half the capacity, texture and load /memory operations use the remaining capacity. Integration with a shared memory/L1 cache 4218 allows the shared memory/L1 cache 4218 to act as a high-throughput pipeline for a stream of data while providing high-bandwidth, low-latency access to frequently reused data, according to FIG at least one embodiment. In at least one embodiment, when configured for general parallel computations, a simpler configuration can be used compared to graphics processing. In at least one embodiment, fixed functionality graphics processing units are bypassed, providing a much simpler programming model. In a general configuration for parallel computations, in at least one embodiment, the work distribution unit allocates and distributes blocks of threads directly to the DPCs. In at least one step in at least one embodiment, threads in a block execute the same program using a unique thread ID in the computation to ensure that each thread produces unique results, with SM 4200 for program execution and performing calculations, shared memory/L1 cache 4218 for inter-thread communication, and LSU 4214 for reading and writing global memory via shared memory/L1 -Cache 4218 and the memory partition unit are used. In at least one embodiment, when configured for general parallel computations, SM 4200 writes instructions that scheduler unit 4204 can use to start new work on DPCs.

In at least one embodiment, the PPU is in a desktop computer, a laptop computer, a tablet computer, servers, supercomputers, a smartphone (e.g., a wireless handheld device), a personal digital assistant ("PDA"), a digital camera , a vehicle, a head-mounted display, a portable electronic device, and more. In at least one embodiment, the PPU is formed on a single semiconductor substrate. In at least one embodiment, the PPU resides in a system-on-a-chip (“SoC”) along with one or more other devices such as additional PPUs, memory, a reduced instruction set CPU (“RISC”), a memory management unit (“MMU ’), a digital-to-analog converter (‘DAC’) and the like.

In at least one embodiment, the PPU may be included on a graphics card that includes one or more memory devices. The graphics card can be configured to connect to a PCIe slot on a desktop computer motherboard. In at least one embodiment, the PPU may be an integrated graphics processing unit ("iGPU") included in the motherboard chipset.

Inference and/or training logic 1415 is used to perform inference and/or training operations associated with one or more embodiments. Details of the inference and/or training logic 1415 are in connection with here 14A and or 14B provided. In at least one embodiment, a deep learning application processor is used to train a machine learning model, such as a neural network, to predict or infer information provided to SM 4200 . In at least one embodiment, SM 4200 is used to infer or predict information based on a trained machine learning model (eg, a neural network) trained by another processor or system or by SM 4200 . In at least one embodiment, SM 4200 may be used to perform one or more of the neural network use cases described herein.

At least one embodiment may be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system may be implemented using a processor as described above.

In at least one embodiment, a single semiconductor platform may refer to a single unitary semiconductor-based integrated circuit or chip. In at least one embodiment, multi-chip modules may be used with increased connectivity that simulate on-chip operation and provide significant improvements over the use of a traditional central processing unit ("CPU") and bus implementation. In at least one embodiment, different modules can also be arranged separately or in different combinations of semiconductor platforms as desired by the user.

In at least one embodiment, computer programs are stored in main memory 2004 and/or secondary storage in the form of machine-readable executable code or computer control logic algorithms. Computer programs, when executed by one or more processors, enable system 2000 to perform various functions, in accordance with at least one embodiment. Storage 2004, memory, and/or any other storage are possible examples of computer-readable media. In at least one embodiment, secondary storage may refer to any suitable storage device or system, such as a hard disk drive and/or removable storage drive, including a floppy disk drive, a magnetic tape drive, a compact disk drive, a DVD drive, a recorder, a USB flash memory (Universal Serial Bus), etc. In at least one embodiment, the architecture and/or functionality of various previous figures relating to CPU 2002, parallel processing system 2012, an integrated circuit that may provide at least a portion of the capabilities of CPU 2002, parallel processing system 2012, a chipset (e.g., a group integrated circuits designed and sold as a unit to perform related functions, etc.) and any suitable combination of integrated circuits.

In at least one embodiment, the architecture and/or functionality of various preceding figures are implemented in the context of a general purpose computing system, a printed circuit board system, an entertainment game console system, an application specific system, and more. In at least one embodiment, the computing device 2000 can take the form of a desktop computer, a laptop, a tablet computer, servers, supercomputers, a smartphone (e.g., a wireless handheld device), a personal digital assistant ("PDA"), a digital camera, vehicle, head mounted display, handheld electronic device, cell phone, television, workstation, gaming console, embedded system and/or any other type of logic.

In at least one embodiment, the parallel processing system 2012 includes, without limitation, a plurality of parallel processing units ("PPUs") 2014 and associated memories 2016. In at least one embodiment, the PPUs 2014 are interconnected 2018 and a switch 2020 or multiplexer connected to a host processor or other peripheral devices. In at least one embodiment, the parallel processing system 2012 distributes computational tasks across the PPUs 2014, which may be parallelizable, e.g., as part of the distribution of computational tasks across multiple GPU (Graphics Processing Unit) thread blocks. In at least one embodiment, memory (e.g., for read and/or write access) is shared and made accessible across some or all PPUs 2014, although such shared memory incurs performance penalties related to the use of local memory and registers resident in a PPU 2014 are resident, can entail. In at least one embodiment, the operation of the PPUs 2014 is synchronized through the use of an instruction such as syncthreads(), where all threads execute in a block (e.g., across multiple PPUs 2014) to reach a specified point in code execution before proceeding.

Other variations are within the spirit of the present disclosure. Thus, while the disclosed techniques are susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof have been shown in the drawings and described in detail above. It should be understood, however, that the disclosure is not intended to be limited to the particular form(s) disclosed, but on the contrary is intended to cover all modifications, alternative constructions, and equivalents as may come within the spirit and scope of the disclosure. as defined in the appended claims.

The use of the terms "a" and "the" and similar references in connection with the description of disclosed example embodiments (particularly in connection with the following claims) should be construed to include both singular and plural where used herein unless otherwise indicated or clearly contradicted by the context, and not as a definition of a term. The terms "comprise", "have", "include" and "contain" shall be construed as having unlimited terms (i.e. "including but not limited to") unless otherwise specified. The term "connected," when referring to physical connections and remaining unaltered, shall be construed as partially or fully included, attached, or connected to each other, even if anything in between. The mention of ranges of values in this document is for a brief method of individually referencing each individual value that falls within the range, unless otherwise specified herein, and each individual value is incorporated into the specification as if recited individually would. The use of the term "set" (e.g. "a set of items") or "subset" is intended to mean a non-empty set comprising one or more elements, unless otherwise indicated or the context indicates otherwise. Further, unless otherwise indicated or contradicted by context, the term "subset" of a corresponding set does not necessarily mean a true subset of the corresponding set, but subset and corresponding set may be the same.

Subjunctive language, such as phrases of the form "at least one of A, B and C" or "at least one of A, B and C", unless expressly stated otherwise or otherwise clearly contradicted by the context, is otherwise understood to mean as the context generally uses to represent that item, term, etc., can be either A or B or C or any non-empty subset of the set of A and B and C. For example, in an illustrative example of a three-member clause, subjunctive clauses "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following clauses: {A}, {B} , {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. a sole Thus, subjunctive language is generally not intended to imply that at least one of A, at least one of B, and at least one of C must be present for particular embodiments. Also, unless otherwise indicated or otherwise indicated by context, the term "plurality" denotes a state of plurality (e.g., "a plurality of elements" denotes multiple elements). The number of elements in a plurality is at least two, but can be more if indicated either explicitly or by context. Further, unless otherwise stated or otherwise clear from the context, the phrase "based on" means "based at least in part on" and not "based solely on."

Operation of the processes described herein may be performed in any suitable order, unless otherwise indicated herein or unless context clearly indicates the contrary. In at least one embodiment, a process such as the processes described herein (or variants and/or combinations thereof) is executed under the control of one or more computer systems configured with executable instructions and executed as code (e.g., executable instructions, one or more computer programs, or one or more applications) running collectively on one or more processors, by hardware, or combinations thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, e.g., in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, a computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagated transient electrical or electromagnetic transmission), but includes non-volatile data storage circuitry (e.g., buffers, cache, and queuing) within transceivers for transient includes signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media storing executable instructions (or other memory for storing executable instructions) that, when executed (i.e., as a result their execution) by one or more processors of a computer system cause the computer system to perform the operations described herein. Set of non-transitory computer-readable storage media, in at least one embodiment, includes multiple non-transitory computer-readable storage media, and one or more individual non-transitory storage media of multiple non-transitory computer-readable storage media lacks all code, while multiple non-transitory computer-readable storage media collectively store all code. In at least one embodiment, executable instructions are executed such that different instructions are executed by different processors - e.g., a non-transitory computer-readable storage medium stores instructions, and a main central processing unit ("CPU") executes some of the instructions, while a graphics processing unit ("GPU") executes others executes commands. In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

Accordingly, in at least one embodiment, computer systems are configured to implement one or more services that individually or collectively perform operations of the processes described herein, and such computer systems are configured with appropriate hardware and/or software that enable operations to be performed. Furthermore, a computer system that implements at least one embodiment of the present disclosure is a single device, and in another embodiment, a distributed computer system that includes multiple devices that operate differently such that the distributed computer system performs the operations described herein and a single device does not performs all operations.

The use of examples or exemplary language (e.g., "such as") is intended solely to better illustrate embodiments of the disclosure and should not be construed as a limitation on the scope of the disclosure unless otherwise noted. Nothing in the specification should be construed as identifying any non-claimed element as essential to the practice of the disclosure.

All references cited herein, including publications, patent applications and patents, are hereby incorporated by reference to the same extent as if each reference were individually and specifically stated as incorporated by reference and reproduced in its entirety.

In the specification and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It is to be understood that these terms are not intended as synonyms for each other. Rather, in individual examples, the terms "connected" or "Coupled" can be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" can also mean that two or more elements are not in direct contact with each other, but can still cooperate or interact with each other.

Unless expressly stated otherwise, it is to be understood that throughout the specification, terms such as "processing", "computing", "calculating", "determining" or the like refer to the actions and/or processes of a computer or computer system or similar electronic computing device that manipulate and/or transform data represented as physical, e.g. electronic, quantities in the registers and/or memories of the computing system into other data that are similar to physical quantities in the memories, registers or other such devices for storage , transmission or display of information.

Similarly, the term “processor” may refer to any device or part of a device that processes electronic data from registers and/or memory and converts that electronic data into other electronic data that is stored in registers and/or memory can become. As non-limiting examples, "processor" can be a CPU or a GPU. A "computing platform" may include one or more processors. As used herein, "software" processes may include, for example, software and/or hardware entities that perform work over time, such as tasks, threads, and intelligent handlers. Also, each process can refer to multiple processes to execute commands in sequence or in parallel, continuously or intermittently. The terms "system" and "method" are used interchangeably herein in that system can embody one or more methods and methods can be considered as a system.

This document may refer to the acquisition, acquisition, receipt, or input of analog or digital data into a subsystem, computer system, or computer-implemented machine. The process of obtaining, capturing, receiving, or inputting analog and digital data can be performed in a variety of ways, such as receiving data as a parameter of a function call or an application programming interface call. In some implementations, the process of obtaining, capturing, receiving, or inputting analog or digital data may be accomplished by transmitting data over a serial or parallel interface. In another implementation, the process of obtaining, gathering, receiving, or inputting analog or digital data may be performed by the transmission of data over a computer network from the providing entity to the receiving entity. It may also refer to the provision, output, transmission, broadcast, or presentation of analog or digital data. In various examples, the process of providing, outputting, transferring, sending, or representing analog or digital data may be accomplished by transferring data as an input or output parameter of a function call, as a parameter of an application programming interface, or as an interprocess communication mechanism.

Claims (25)

Computer-implemented method, comprising: obtaining (1302) touch sensor information from a robotic member (104, 108, 112, 116, 204, 504, 618-633) manipulating an object in the real world; creating (1304) a plurality of simulations of the robotic limb (104, 108, 112, 116, 204, 504, 618-633) manipulating the object, each simulation having different poses for the object; determining (1308) a plurality of costs, wherein each cost of the plurality of costs corresponds to a respective simulation of the plurality of simulations and each cost of the plurality of costs is based at least in part on differences between the touch-sensor information and simulated touch-sensor information determined by the respective simulation of the variety of simulations is generated; identifying (1310) a single simulation of the plurality of simulations based at least in part on the cost; determining (1312) a pose of the object in the real world based at least in part on a pose of the object in the identified individual simulation; and providing the pose of the object to a robotic control system (702) that controls a robot to perform a task based at least in part on the pose of the object. Computer-implemented method claim 1 , further comprising updating one or more physical parameters of the plurality of simulations to reduce a difference between a simulation and a real world observation. Computer-implemented method claim 1 or 2 , wherein the plurality of simulations are implemented using a GPU accelerated physics simulator (708). The computer-implemented method of any preceding claim, wherein the single simulation is identified by identifying a single simulation that produces simulated touch-sensor information that most closely resembles the touch-sensor information. Computer-implemented method according to one of the preceding claims, further comprising: obtaining an initial pose estimate of the object; and generating a plurality of possible poses to be used in the plurality of simulations, the plurality of possible poses being generated by modifying the initial pose estimate. Computer-implemented method claim 5 wherein the initial pose estimate of the object is determined based on an image of the object obtained before the object is grasped by the robotic member (104, 108, 112, 116, 204, 504, 618-633). A computer-implemented method as claimed in any preceding claim, wherein the single simulation is identified by identifying a single simulation with the lowest associated cost. The computer-implemented method of any preceding claim, wherein the touch sensor information comprises a two-dimensional array of force values for each finger of the robotic limb (104, 108, 112, 116, 204, 504, 618-633). System comprising: one or more processors (1802, 1910, 2002, 2014, 2130, 2205-2207, 2210-2213, 2231, 2232, 2305, 2310, 2315, 2320, 2405, 2410, 2440, 2500, 261,220, 261,220, 261,220, 261,220, 261,220, , 2806A-2806D, 2900, 3000, 3100, 3200, 3302, 3400, 3500, 3610, 3800, 3808, 3900); and a computer-readable memory (1802, 2004, 2016, 2201, 2202, 2220, 2220-2223, 2233, 2234, 2365, 2544A, 2544B, 2604, 2722, 2724, 2770, 3320, 3418, 2104.4) containing executable instructions stores which, when executed by the one or more processors, cause the system to: obtaining (1302) data describing forces on a robotic limb (104, 108, 112, 116, 204, 504, 618-633) grasping an object; creating (1304) simulations of the robotic limb gripping the object, individual simulations of the simulations using different poses for the object; determining (1308) a plurality of values, each value of the plurality of values corresponding to a respective one of the plurality of simulations and each value of the plurality of values being based at least in part on differences between the forces and simulated-tactile simulated forces generated by the respective simulation of the variety of simulations generated; identifying (1310) a single simulation of the simulations based at least in part on the value; determining (1312) a pose of the object in the real world based at least in part on a pose of the object in the identified single simulation; and providing the pose of the object to a robotic control system (702) that controls a robot to perform a task based at least in part on the pose of the object. system after claim 9 , wherein the executable instructions cause the system to further update one or more physical parameters of the simulations to reduce a difference between a state of a simulation and an observed state in the real world. system after claim 9 or 10 wherein the one or more processors comprises a graphics processing unit (2210-2213, , 2231, 2232, 2310, 2315, 2410, 2440, 2500, 2530, 2734, 2806A-2806D, 2900, 3500, 3610). system according to one of the claims 9 until 11 , where the single simulation is identified by identifying a single simulation that produces simulated data that most closely matches the data. system according to one of the claims 9 until 12 , wherein the executable instructions further cause the system to: obtain an initial pose of the object; and generating a plurality of poses to be used on objects in the simulations, the plurality of possible poses being generated by changing the initial pose. system after Claim 13 , wherein the initial object pose is determined using an image of the object obtained before the object is grasped by the robotic member (104, 108, 112, 116, 204, 504, 618-633). system according to one of the claims 9 until 14 , where: the value is a measure of a difference between the forces and the simulated forces, and the single simulation is identified by identifying a single simulation with a lowest associated value. system according to one of the claims 9 until 15 , wherein the data is touch sensor information generated by a touch force sensor on each finger of the robotic limb (104, 108, 112, 116, 204, 504, 618-633). Computer readable medium (1802, 2004, 2016, 2201, 2202, 2220, 2220-2223, 2233, 2234, 2365, 2544A, 2544B, 2604, 2722, 2724, 2770, 3320, 3418, 2104,4,4,3904,4) which when executed by one or more processors (1802, 1910, 2002, 2014, 2130, 2205-2207, 2210-2213, 2231, 2232, 2305, 2310, 2315, 2320, 2405, 2410, 2440, 2500, 26020, , 2612, 2734, 2806A-2806D, 2900, 3000, 3100, 3200, 3302, 3400, 3500, 3610, 3800, 3808, 3900) of a computer system cause the computer system to: obtaining (1302) data describing forces on a robotic limb grasping an object; performing (1304) simulations of the robotic limb (104, 108, 112, 116, 204, 504, 618-633) grasping the object, individual simulations of the simulations having different poses for the object; determining (1308) a plurality of values, each value of the plurality of values corresponding to a respective one of the plurality of simulations and each value of the plurality of values being based at least in part on differences between the forces and simulated-tactile simulated forces generated by the respective simulation of the variety of simulations generated; identifying (1310) a single simulation of the simulations based at least in part on the value; determining (1312) a pose of the object in the real world based at least in part on a pose of the object in the identified single simulation; and providing the pose of the object to a robotic control system (702) that controls a robot to perform a task based at least in part on the pose of the object. Computer readable medium (1802, 2004, 2016, 2201, 2202, 2220, 2220-2223, 2233, 2234, 2365, 2544A, 2544B, 2604, 2722, 2724, 2770, 3320, 3418, 21804) after.4 Claim 17 , wherein the instructions further cause the computer system to update one or more parameters of the simulations to reduce a difference between a state of a simulation and an observed state in the real world. Computer readable medium (1802, 2004, 2016, 2201, 2202, 2220, 2220-2223, 2233, 2234, 2365, 2544A, 2544B, 2604, 2722, 2724, 2770, 3320, 3418, 21804) after.4 Claim 17 or 18 , wherein the one or more processors comprise a multi-core graphics processing unit. Computer-readable medium (1802, 2004, 2016, 2201, 2202, 2220, 2220-2223, 2233, 2234, 2365, 2544A, 2544B, 2604, 2722, 2724, 2770, 3320, 3418, 3904) after any of claims 17 until 19 , where the simulations are performed in parallel using a plurality of processors. Computer-readable medium (1802, 2004, 2016, 2201, 2202, 2220, 2220-2223, 2233, 2234, 2365, 2544A, 2544B, 2604, 2722, 2724, 2770, 3320, 3418, 3904) after any of claims 17 until 20 , wherein the instructions further cause the computer system to: obtain an initial pose of the object; and generating a plurality of poses to be applied to objects in the simulations, the plurality of possible poses being generated by changing the initial pose. Computer readable medium (1802, 2004, 2016, 2201, 2202, 2220, 2220-2223, 2233, 2234, 2365, 2544A, 2544B, 2604, 2722, 2724, 2770, 3320, 3418, 21804) after.4 Claim 21 , wherein the initial pose of the object is determined using an image of the object obtained with a depth camera. Computer-readable medium (1802, 2004, 2016, 2201, 2202, 2220, 2220-2223, 2233, 2234, 2365, 2544A, 2544B, 2604, 2722, 2724, 2770, 3320, 3418, 3904) after any of claims 17 until 22 , where: the value is a measure of the difference between the forces and the simulated forces, and the single simulation is identified by identifying a single simulation with a lowest associated value. Computer-readable medium (1802, 2004, 2016, 2201, 2202, 2220, 2220-2223, 2233, 2234, 2365, 2544A, 2544B, 2604, 2722, 2724, 2770, 3320, 3418, 3904) after any of claims 17 until 23 , wherein the data is touch sensor information generated by a touch force sensor on the robot limb. A robot comprising: an arm comprising one or more movable elements connected via one or more servomotors; a robotic limb (104, 108, 112, 116, 204, 504, 618-633) connected to the arm, the robotic limb having one or more touch sensors; one or more processors (1802, 1910, 2002, 2014, 2130, 2205-2207, 2210-2213, 2231, 2232, 2305, 2310, 2315, 2320, 2405, 2410, 2440, 2500, 261,220, 261,220, 261,220, 261,220, 261,220, , 2806A-2806D, 2900, 3000, 3100, 3200, 3302, 3400, 3500, 3610, 3800, 3808, 3900); and the computer-readable medium (1802, 2004, 2016, 2201, 2202, 2220, 2220-2223, 2233, 2234, 2365, 2544A, 2544B, 2604, 2722, 2724, 2770, 3320, 3418, 4904) according to any one of 3904 claims 17 until 24 , which is connected to the one or more processors.

Images