DE102020127508A1 - POSITION TRACKING OBJECTS IN HAND - Google Patents

Abstract

Devices, systems, and techniques are described for estimating the pose of an object while the object is being manipulated by a robotic limb. In at least one exemplary embodiment, a sample-based optimization algorithm tracks object poses in the hand during the manipulation by means of feedback from contacts and a GPU-accelerated robot simulation is developed. In at least one exemplary embodiment, parallel simulations simultaneously model object pose changes that can be caused by complex contact dynamics. In at least one embodiment, the optimization algorithm adjusts the simulation parameters while tracking the object pose to further improve tracking performance. In various embodiments, contact sensing in the real world can be improved by using vision in the control loop.

Description

TECHNICAL PART

At least one embodiment relates to training and simulating robots to perform and facilitate tasks. For example, at least one embodiment relates to training and simulating robots using artificial intelligence according to various novel techniques described herein.

BACKGROUND

Training and simulating robots to accurately perform tasks can consume significant memory, time, or computational resources. Training a robotic control system to track the pose of an object held and manipulated by a robotic hand is challenging for vision-based systems to track the pose of an object because the object is heavily obscured while the robotic hand is holding it. Such obscuration reduces the amount of data that can be used in determining the movements a robot should perform, which creates the risk that the task at hand is performed incorrectly and / or inefficiently, potentially damaging the object or other objects in the Environment will be damaged. Such tracking is particularly complex because objects sometimes slip or otherwise move during the process, creating changes in the orientation of the object that go undetected and are therefore not accounted for. The amount of memory, time, or computational resources used to accurately train and simulate robots can be improved.

Figure list

• 1 shows examples of teleoperations for different tasks, according to at least one embodiment;
• 2 shows an example of a robot with touch sensors, according to at least one embodiment;
• 3 FIG. 11 shows an example of a system that tracks objects in real time, according to at least one embodiment; FIG.
• 4th shows an example for the estimation of a hand pose, according to at least one embodiment;
• 5 shows an example of a pose of a human hand and a robot gripper that executes a corresponding pose, according to at least one embodiment;
• 6th shows an example of poses of a human hand and corresponding poses of a robot gripper, according to at least one embodiment;
• 7th shows an example of a framework (programming frame structure) for tracking object poses in the hand, according to at least one embodiment;
• 8th shows an example for a comparison of optimizers, according to at least one embodiment;
• 9 FIG. 11 shows an example of an algorithm used by a system according to at least one embodiment; FIG.
• 10 shows a first example of results of ablation studies, according to at least one embodiment;
• 11 shows a second example for results of ablation studies, according to at least one embodiment;
• 12th shows an example of test results from the real world, according to at least one embodiment;
• 13th Figure 13 shows an example of a method that, as a result of being executed by a computer system, determines the pose of an object that is manipulated by a touch sensor equipped robotic hand;
• 14A shows an inference and / or training logic, according to at least one embodiment;
• 14B shows an inference and / or training logic, according to at least one embodiment;
• 15th shows a training and the use of a neural network, according to at least one embodiment;
• 16 shows an example of a data center system, according to at least one embodiment;
• 17A shows an example of an autonomous vehicle, according to at least one embodiment;
• 17B shows an example of camera positions and fields of view for the autonomous vehicle of FIG 17A , according to at least one embodiment;
• 17C FIG. 13 is a block diagram illustrating an exemplary system architecture for the autonomous vehicle of FIG 17A shows, according to at least one embodiment;
• 17D Figure 13 is a diagram illustrating a system for communicating between cloud-based servers and the autonomous vehicle of the 17A shows, according to at least one embodiment;
• 18th Figure 3 is a block diagram showing a computer system according to at least one embodiment;
• 19th Figure 3 is a block diagram showing a computer system according to at least one embodiment;
• 20th shows a computer system in accordance with at least one embodiment;
• 21 shows a computer system in accordance with at least one embodiment;
• 22A shows a computer system in accordance with at least one embodiment;
• 22B shows a computer system in accordance with at least one embodiment;
• 22C shows a computer system in accordance with at least one embodiment;
• 22D shows a computer system in accordance with at least one embodiment;
• 22E and 22F show a common programming model, according to at least one embodiment;
• 23 FIG. 11 shows example integrated circuits and associated graphics processors, in accordance with at least one embodiment; FIG.
• 24A and 24B Figure 12 shows example integrated circuits and associated graphics processors, in accordance with at least one embodiment;
• 25A and 25B show further exemplary graphics processor logic, in accordance with at least one embodiment;
• 26th shows a computer system in accordance with at least one embodiment;
• 27A shows a parallel processor according to at least one embodiment;
• 27B shows a partition unit, according to at least one embodiment;
• 27C shows a processing cluster, according to at least one embodiment;
• 27D shows a graphics multiprocessor according to at least one embodiment;
• 28 shows a multi-graphics processing unit (GPU), in accordance with at least one exemplary embodiment;
• 29 shows a graphics processor in accordance with at least one embodiment;
• 30th Figure 3 is a block diagram showing a processor microarchitecture for a processor, according to at least one embodiment;
• 31 shows a processor for deep leaming applications, according to at least one embodiment;
• 32 FIG. 10 is a block diagram showing an exemplary neuromorphic processor, according to at least one embodiment; FIG.
• 33 shows at least parts of a graphics processor, according to one or more exemplary embodiments;
• 34 shows at least parts of a graphics processor, according to one or more exemplary embodiments;
• 35 shows at least parts of a graphics processor, according to one or more exemplary embodiments;
• 36 Figure 3 is a block diagram of a graphics processing engine 3610 a graphics processor, in accordance with at least one embodiment;
• 37 shows a block diagram of at least parts of a graphics processor core, according to at least one embodiment;
• 38A and 38B show thread execution logic 3800 , comprising an array of processing elements of a graphics processor core, according to at least one embodiment;
• 39 shows a parallel processing unit (“PPU”) in accordance with at least one embodiment;
• 40 shows a general processing cluster (“GPC”) in accordance with at least one embodiment;
• 41 shows a memory partition unit of a parallel processing unit (“PPU”), according to at least one embodiment; and
• 42 shows a streaming multiprocessor, according to at least one embodiment.

DETAILED DESCRIPTION

The present document describes a system and method for estimating the pose of an object while the object is being manipulated by a robotic hand, claw or manipulator. When an object is held by a robot, image-based pose estimation systems may, in various examples, produce inaccurate pose estimates due to object occlusion. In at least one embodiment, a robotic hand is equipped with touch sensors (tactile sensors), and while an object is being manipulated by the robotic hand, the sensor signals generated by the touch sensors are used to better estimate the pose of the object. In some situations, dynamic effects such as sliding can occur during active manipulation, which increase the difficulty of estimating the pose. In at least one embodiment, a physical model of the object is used to improve the modeling of robot-object interactions.

In at least one embodiment, a graphics processing unit (“GPU”) accelerated physics engine with derivative-free, sample-based (scan-based, sample-based) optimizers tracks the poses of an object in the hand with feedback from contacts during manipulation. In at least one embodiment, the physics simulation is used as a forward model for robot-object interactions, and the techniques described here jointly optimize for the state and parameters of the simulations so that the simulations can more accurately estimate the real world.

In at least one exemplary embodiment, the techniques described here explicitly model the dynamics of robot-object interactions for object pose tracking and optimize for the simulation parameters during pose tracking. In various examples, these features allow the system to track the object pose under complex dynamic behaviors, such as translational and torsional slippage due to inertia and external forces, as well as breaking and re-establishing contact. In addition, by using the GPU accelerated physics engine, these techniques can often be applied in real time (30 Hz) using the GPU.

Several of the exemplary embodiments demonstrate promising applications of GPU-accelerated physics simulation for robotics. In some exemplary embodiments, for example, the speed of the physics engine allows complex, high-contact simulations and sample-based optimization methods that rely on data from many simultaneous simulations, all in real time on the same machine, which is often difficult with CPU-based simulations. Various embodiments can be used as an aid to estimating an object pose in the hand and to reduce the restriction that many researchers face with skillful (agile, manual) manipulation, i.e. that the object is usually placed in such a way that it is only intended for Part of it is visible from a camera, which limits the selection of manipulation tasks that can be investigated.

Teleoperation can equip lifeless robotic systems with highly developed logical skills, intuition, and creativity. However, teleoperation solutions for robots with a high degree of actuation (“DoA”) and multiple fingers can be complex. In at least one embodiment, a system is developed that enables complete control of a high DoA robotic system by mere observation of the bare human hand. The system can enable operators to solve a wide variety of complex manipulation tasks that go beyond simple pick-and-place operations. In various embodiments, the system can be implemented by one or more systems, as shown in FIG 13-40 described / illustrated.

Tracking the pose of an object while it is being held and manipulated by a robotic hand can be difficult for vision-based methods due to significant occlusion. The techniques described here use GPU-accelerated parallel robot simulations and derivative-free, sample-based optimizers to track the poses of objects in hand with feedback from contacts during the manipulation. In some examples, a physics simulation is used as a forward model for robot-object interactions, and the algorithm works together to optimize the state and parameters of the simulations so that they better match those in the real world. At least one exemplary embodiment runs in real time (30 Hz) on the GPU and achieves an average point cloud spacing error of 6 mm in simulation experiments and 13 mm in the real world.

In at least one embodiment, implementation of dexterous manipulation policies benefits from a robust assessment of the pose of the hand-held object. In many implementations, however, tracking the hand-held object pose is still challenging due to significant occlusion. In such implementations, work requiring poses of hand-held objects can be limited to experiments in which the object is mostly visible , or who use multiple cameras, or where the hand-object transformation is fixed or known. In some examples, the problem of visual occlusion is reduced by looking more closely at an estimate of object poses through contacts or tactile feedback, often using particulate filters and knowledge of object geometry and contact locations. In at least one embodiment, these techniques can be applied to a static gripping situation in which an object is stationary and in grasp. In at least one embodiment, these techniques are extended to tracking object poses during hand manipulation, which requires modeling of complex object-hand contact dynamics.

To enable tracking of objects in the hand during manipulation by a robot, in at least one embodiment a GPU-accelerated high-precision physics simulator is combined as a forward dynamics model with a sample-based optimization framework for tracking object poses with feedback through contacts, as in 7th shown. In at least one embodiment, a simultaneous set of simulations is initialized with the initial states of a real robot and the initial pose of the real object, which can be obtained by a vision-based pose registration algorithm that assumes that the object is not initially in obscurity. In at least one embodiment, the initial poses of the simulated objects are shifted slightly and reflect the uncertainty of the vision-based pose registration algorithm. In at least one embodiment, the GPU-accelerated physics simulator runs many simultaneous simulations in real time on a single GPU. While a certain guideline controls the real robot in at least one embodiment in such a way that it approaches the object, grasps the object in the hand and manipulates it, the system executes the same robot control commands on the simulated robots. In at least one embodiment, observations of the real robot and the simulated robots are collected, which include, for example, the size and direction of contacts on the contact sensors of the robot hand. In at least one embodiment, a sample-based optimization algorithm periodically updates the states and parameters of the simulations according to a cost function that records how well the observations of each simulation match those of the real world. In addition, in some exemplary embodiments, the algorithm updates simulation parameters such as mass and friction to further improve the dynamic models of the real world simulations. In at least one exemplary embodiment, the object pose estimated at any point in time is the pose of the robot-object system.

In at least one exemplary embodiment, a total of 24 trajectories of manipulation in the hand with three different objects in the simulation and in the real world were collected to evaluate the proposed algorithm. In at least one embodiment, a Kuka IIWA7 arm with the 4-finger Wonik Robotics Allegro hand was used as the end effector, each finger being equipped with a SynTouch BioTac touch sensor. In at least one embodiment, it is the Object manipulation trajectories around human demonstrations collected via a teleoperation system that tracks a hand. In at least one exemplary embodiment, detailed ablation studies are carried out in simulation experiments in order to examine the properties of the proposed algorithm, since object poses are available as basic truth data in the simulation. In at least one exemplary embodiment, a vision-based algorithm is used for real experiments in order to obtain the object pose in the first and last individual image of the collected trajectories in which the object is not concealed. In at least one embodiment, the pose in the first frame is used to initialize the simulations and the pose in the last frame is used to assess the accuracy of the proposed contact-based algorithm.

In various examples, a position of an object in the hand is identified using only vision, by first segmenting the robot or human hand in an image before estimating the pose. However, with greater occlusion, vision-only approaches may degrade in performance. Some embodiments use tactile feedback to aid in estimating the object pose. Tactile perception can be object properties such as B. material and pose, and also provide feedback during object manipulation.

In at least one embodiment, experiments with dynamics models and particle filter techniques show that adding noise to the applied forces rather than the underlying dynamics provides more accurate tracking results. In at least one embodiment, tactile feedback is combined with a vision-based object tracker to track trajectories of objects during planar sliding tasks, and in another embodiment, incremental smoothing and mapping (iSAM) is used to provide global visual Combine pose estimates with local contact pose measurements.

In at least one embodiment, a robotic hand grasps an object and locates the object pose without moving. Some examples use point contact locations and some examples use a full tactile map to extract local geometry information around the contacts.

In at least one embodiment, feedback from contact positions is used for pose estimation, and some implementations use a variant of Bayesian or particle filters. In some exemplary embodiments, the filtering is performed collectively via visual features, hand-joint positions, force-torque measurements, and binary contact modes. Some techniques can also be applied to estimating the pose when the object is not being held by the robotic hand using force sensors.

In at least one embodiment, tactile maps are used for pose estimation, with some examples using large, low-resolution tactile arrays to detect contacts in a grid, while other examples use high-resolution tactile sensors attached to the fingertips of the robot. In at least one embodiment, the system searches for similar local areas on an object surface to locate the object with respect to the contact position, and other systems merge GelSight data with a point cloud sensed by a depth sensor before performing a pose estimation.

In some embodiments, tracking of the object pose in hand is implemented during object manipulation, which is more challenging than when the object is static. In at least one exemplary embodiment, an algorithm is used that combines contact positions with dense articulated real-time tracking (“DART”). In at least one embodiment, the algorithm fuses contact positions with visual color features, joint positions, and force-torque measurements. In at least one exemplary embodiment, the algorithm is sensitive to the initialization of the object poses, in particular if the object appears small in the depth image. In at least one exemplary embodiment, the techniques described here do not require access to robust visual features during the manipulation, but instead use a physics simulator for modeling both the kinematics and the dynamics of the robot-object system.

In various examples, a teleoperation of robots can be used in the fields of search and rescue, space travel, medicine and applied machine learning. The motivation for a teleoperation ability can be to enable a robotic system to solve complex tasks by using a human's cognition, creativity and ability to react via a human Makes use of the machine interface ("HMI"). In one embodiment, this system is a glove-free solution to control a multi-fingered, strongly moving robot system for solving a large number of gripping and manipulation tasks. In some examples, depth cameras and various graphics processing units (“GPU”) can be used in conjunction with deep learning and optimization to create a nifty teleoperation system with a minimal footprint. In some examples, a variety of physical tasks can be performed with visual feedback alone. Therefore, this system can utilize the human ability to plan, move, and predict the consequences of physical actions through vision alone, which can be a sufficient condition for solving a wide variety of tasks.

In various exemplary embodiments, the developed system enables such skillful robot manipulation through deep observation of the bare human hand with several cameras. In some examples, the system may be a fully vision-based, glove-free teleoperation system that skillfully moves a highly articulated robotic hand arm system through direct imitation. The system can also perform a number of tasks that require particularly fine manipulation and dexterity (e.g., pulling paper money out of a wallet and picking up two dice with four fingers at the same time, as in FIG 1 shown).

1 shows an example of a teleoperation for various tasks, according to at least one embodiment. In one example, a robotic gripper engages 104 a cylinder using a grasping pose based on a pose of a human hand 102 . In another example, a robotic gripper engages 108 a cube with a grasping float based on a human hand 106 . In another example, a robotic gripper engages 112 a cup using a grasping float based on a human hand 110 . In another example, a robotic gripper engages 116 a wallet using a grasping float based on a human hand 104 .

The teleoperation structure may include a robotic system and an adjacent arena for human operators, as in FIG 2 shown. 2 shows an example of a robot with tactile sensors according to at least one embodiment. In at least one embodiment, a robot has 202 via a robotic gripper 204 , which is used to grasp objects using tactile sensors. In at least one embodiment, a set of cameras 206 , 208 , 210 and 212 used to be the working space of the robot 202 to observe. In at least one embodiment, the gripper comprises 204 a set of tactile sensors 216 , 218 , 220 and 222 that provide sensory information to a control computer system. In at least one embodiment, the tactile sensors can be covered with a friction material to enhance the robot's ability to grip an object.

In some embodiments, as in 2 shown, the robot system can be an arm from the KUKA LBR iiwa7 R800 series with a Wonik Robotics Allegro hand with four SynTouch BioTac touch sensors on the fingertips and a 3M TB641 grip strap that is applied to the inner surfaces of the phalanx and the palm , where the rubbery surfaces of both the BioTac sensors and 3M tape can improve hand friction, while the BioTacs themselves can generate 23 signals that can later be used to learn sensorimotor control using demonstrations . The human sphere of influence can be a black-clad table, which is surrounded by four calibrated and time-synchronized cameras, such as Intel RealSense RGB D cameras, which can be spatially arranged so that they have a work area of 80 cm x 55 cm x 38 cm cover. In some examples, since teleoperation is entirely based on human vision and spatial reasoning, the cameras can be right next to the robot to improve line of sight and visual proximity. It should be noted that 2 is intended as one embodiment, and in various embodiments, the system may include any robotic system using any robotic components (e.g., different types of robotic arms, hands, tactile sensors, handles, other sensors, cameras, and / or variations thereof) in any suitable environment can.

To create a natural feel teleoperation system, an imitation type paradigm can be used. The mere movement of a human hand - pose and finger configuration - can be constantly observed and measured by a visual perception module. The movement of the human hand can then be passed on to the robot system in such a way that the copied movement is self-explanatory. This approach can allow a human operator to flex and arrange their fingers, form grips, realign their palms, and translate, with the robotic system follows in a similar manner. In at least one embodiment, the system relies heavily on Dense Articulated Real Time Tracking (“DART”), which can form the backbone of tracking the pose and the joint angles of the human hand. The full system architecture and component connections are in 3 shown in one embodiment.

3 FIG. 10 shows an example of a system that tracks objects in real time, according to at least one embodiment. In at least one embodiment, the system operates using three threads, which are independent processes that run on one or more processors of a computer system. In at least one embodiment, one or more images of a hand are obtained from cameras with RGB depth (“RGB-D”) 302. The images are from a Pointnet: Level 1 304, a Pointnet: Level 2 306 and a Jointnet 308 processed to create a hand pose for the hand in the images. In at least one embodiment, a movable hand model 310 and the hand pose with DART 312 and kinematic realignment (tracking, readjusting, redetermining, adapting to a new target, kinematic retargeting) 314 in order to generate a corresponding hand pose for a robot gripper. In at least one embodiment, a control thread applies the Riemann motion guidelines 318 on the hand pose of the gripper, and the resulting information is used to control the robot 320 used.

In at least one exemplary embodiment, DART is used for continuous tracking of the pose and the joint angle of a human hand. In at least one embodiment, DART uses a moving model of the hand that is registered using an input point cloud. A model of the human hand can be obtained and converted into a single mesh model. Using Computer Aided Design (CAD) software, the fingers of the mesh model can be divided into their respective proximal, medial and distal members and re-created as separate meshes together with an associated Extensible Markup Language (XML) file describing their kinematic arrangement can be exported. Overall, the model can be of the human hand 20th Rotary joints have: four joints per finger with one abduction joint and three flexion joints.

In at least one embodiment, DART is a model-based tracker that performs non-linear optimization and initialization (e.g. from the previous frame or a first estimate). In some examples, if this initialization is not within the convergence basin, the tracker may not converge to the correct solution. In various exemplary embodiments, when tracking the model of the human hand with point cloud data, the hand model can often lock onto incorrect local minima, which leads to tracking errors every few minutes. In order to be able to reliably track the human hand over long periods of time - which is needed for teleoperation - it can therefore be desirable to have reliable specifications for the hand pose of a human hand, a clean segmentation of the hand and a studio with several cameras in order to prevent the hand model from locking onto unexpected local minima. In various exemplary embodiments, a method for generating hand pose specifications is the training of a neural network on a large data set of poses of a human hand, which are generated by camera images.

In at least one embodiment, the data collection is initiated with DART and without hand pose specifications, whereby a first network is trained to produce hand pose specifications. As a result, DART and the last trained neural network can generate increasing amounts of data. In at least one exemplary embodiment, the network is constantly updated with the latest data records in order to generate ever better specifications for DART, which can ultimately expand the area in which DART can function without failures. In some examples, the neural network for hand poses may be a PointNet-based architecture that works directly with fused point cloud data obtained by back-projecting depth images from extrinsically calibrated depth cameras into a single global frame of reference with annotations provided by DART. Since the fused point cloud in various exemplary embodiments contains both the points on the table and the human body and arm, it can be essential in various cases to first localize the hand. Points can be removed from the table by fitting a plane and forwarding the remaining points, which contain the arm and the human body, to PointNet, which can both locate the hand and provide the hand pose. PointNet may be based on estimating the hand pose using a voting-based regression scheme to the 3D positions of specified keypoints on the hand, a technique that may be associated with spatial softmax that is often used in 2D keypoint localization. In various embodiments, PointNet can be trained to predict 3D coordinates of 23 key points specified on the hand - four joint key points for each of the five fingers and three key points on the back of the hand to estimate the hand pose. The Loss function can be the Euclidean distance between the predicted and the key points of the fundamental truth. Additionally, an auxiliary segmentation loss may be included to obtain hand segmentation. For efficiency reasons, any input point cloud can be evenly subsampled up to a fixed size of 8192 x 3 before it is fed into PointNet. In at least one embodiment, a reasonable estimate of the hand pose and segmentation can be achieved, and high quality predictions may not yet be made for the 20 common key points on the fingers. In at least one embodiment, the uniform subsampling used in the input may indicate that the points on the fingers are not densely sampled, so a refinement may be required in a second stage in which the points are undersampled on the hand from the original raw point cloud Taking into account the pose and segmentation of the first stage are re-sampled. In at least one embodiment, the second level can be trained on the same loss functions, but instead can only use the points sampled on the hand to accurately predict the 23 key points. In at least one exemplary embodiment, random disturbances can be added to the hand pose for the second stage in order to ensure robustness against possible inaccuracies in the hand pose of the first stage. 4th illustrates the refinement of the second level within the system, according to at least one exemplary embodiment. In at least one embodiment, both levels of PointNet can be trained on 100K point clouds collected over a 30-45 minute stack for a total of 7-8 hours by running DART to provide annotation for key points, joint angles, and segmentation. In at least one embodiment, a third neural network can be trained that assigns the key points predicted by PointNet to the corresponding joint angles in order to provide specifications for the joint angles of the fingers. This neural network, which can be referred to as a JointNet, can be a two-tier, fully connected network that inputs size 23 x 3 and predicts a 20-dimensional vector of joint angles for fingers.

In at least one exemplary embodiment, the neural networks are trained on the basis of data that have been collected on a number of human hands, which ensures an exact adjustment of the pose for this system and enables reasonable specifications for DART. In some embodiments, hand tracking may work better for hands that are geometrically close to the human DART hand model.

wobei q_h, q_a die Winkel des Modells der menschlichen Hand bzw. der Allegro-Hand sein können, r_i ∈ R³ der Vektor, der vom Ursprung eines Koordinatensystems zu einem anderen zeigt, ausgedrückt im Ursprungskoordinatensystem sein kann (siehe 7). Darüber hinaus ist in mindestens einem Ausführungsbeispiel $d_i=\left|\left|r_i\left(q_h\right)\right|\right|\text{ und }{r}_i^{}\left(q_h\right)=\frac{r_i\left(q_h\right)}{\left|\left|r_i\left(q_h\right)\right|\right|}.$

Die Umschaltgewichtsfunktion s(d_i) kann wie folgt definiert sein: $$s\left(d_i\right)=\{\begin{array}{c}\mathrm{1,}{d}_i>\epsilon \\ \mathrm{200,}{d}_i\le \epsilon \wedge r_i\left(q_h\right)\in S_1\\ \mathrm{400,}{d}_i\le \epsilon \wedge r_i\left(q_h\right)\in S_2\end{array}$$

wobei S₁ Vektoren sein können, die von einem Primärfinger (Zeige-, Mittel-, Ringfinger) ausgehen und auf den Daumen zeigen, und S₂ m Vektoren zwischen zwei Primärfingern sein können, wenn beide Primärfinger assoziierte Vektoren ∈ S₁ haben (z.B. beide Primärfinger werden mit dem Daumen projiziert). In mindestens einem Ausführungsbeispiel ist die Distanzfunktion f(d_i) ∈ R definiert als: $$f\left(d_i\right)=\{\begin{array}{c}\mathrm{\beta }{d}_i,d_i>\epsilon \\ {\mathrm{\eta }}_1,d_i\le \epsilon \wedge r_i\left(q_h\right)\in S_1\\ {\mathrm{\eta }}_2,d_i\le \epsilon \wedge r_i\left(q_h\right)\in S_2\end{array}$$

wobei ß = 1.6 ein Skalierungsfaktor sein kann, η₁ = 1 x 10^-4 m ein Abstand zwischen einem primären Finger und dem Daumen sein kann, und η₂ = 3 x 10^-2 m kann ein Mindestabstand zwischen zwei primären Fingern sein kann, wenn beide primären Finger mit dem Daumen projiziert werden. In mindestens einem Ausführungsbeispiel stellen diese Projektionen sicher, dass die Kontakte zwischen den primären Fingern und dem Daumen nahe beieinander liegen, ohne dass es in einem Präzisionsgriff zu Kollisionen der primären Finger kommt. In mindestens einem Ausführungsbeispiel kann dies bei Ungenauigkeiten der visuellen Fingerverfolgung besonders nützlich sein. In einigen Beispielen können die Vektoren r_i nicht nur Abstand und Richtung von einem Aufgabenraum zum anderen erfassen, sondern ihr Ausdruck in lokalen Koordinaten kann weiter Informationen darüber enthalten, wie die Koordinatensysteme und damit die Fingerspitzen zueinander orientiert sind. In mindestens einem Ausführungsbeispiel können die Koordinatensysteme des Modells der menschlichen Hand daher äquivalente Koordinatensysteme auf dem Allegro-Modell mit Ähnlichkeit in Orientierung und Platzierung haben. Die in 5 gezeigten Vektoren können einen minimalen Satz bilden, der das gewünschte Verhalten beim Neuausrichten (engl. retargeting) erzeugt. In einigen Ausführungsbeispielen kann y = 2,5 x 10^-3 ein Gewicht bei der Regularisierung der Allegro-Winkel auf Null sein (äquivalent zur vollständig geöffneten Hand). In mindestens einem Ausführungsbeispiel trägt dieser Term dazu bei, die Redundanz in der Lösung zu reduzieren und sicherzustellen, dass die Hand nie in ungewöhnliche Minima gerät, von denen sie sich nur schwer wieder erholen kann (z.B. wenn sich die Finger in die Handfläche einbetten). In mindestens einem Ausführungsbeispiel werden in 6 verschiedene Zuordnungen von der menschlichen Hand 602-617 zu einer Allegro-Roboterhand 618-633 gezeigt, wie sie durch kinematisches Neuausrichten erzeugt werden.In at least one embodiment, teleoperation of a robotic hand that is kinematically different from the human hand may require a module that can map the observed human wrists to the robotic wrists, which in some embodiments may be referred to as Allegro wrists. 5 shows an example of a pose 502 a human hand and a robotic gripper 504 , who performs a corresponding pose, according to at least one embodiment. There can be many different approaches to kinematic realignment. For example, a module according to at least one exemplary embodiment can be used to match the positions from the palm to the fingertips and the medial joints and the directional dependency of the proximal phalanx and the distal phalanx. In at least one exemplary embodiment, the optimized assignment can be assigned using human depth images in such a way that a depth network can record a depth image and output joint angles. In at least one exemplary embodiment, motion retargeting is also used. For example, a deep recurrent neural network can be trained unsupervised to track movements between skeletons. In at least one embodiment, the system uses the task space metric of the fingertips, since distal areas can be of the highest priority in gripping and manipulation tasks, measured by their contact prevalence, the degree of innervation and the increased controllability for a fine hand manipulation ability. In at least one embodiment, the joint axes and locations between two hands may be different, so no metrics can be used that directly compare the joint angles between the two hands. In at least one exemplary embodiment, both the distance and the direction between the fingertips are taken into account in order to detect and optimize the positioning of the fingertips. Specifically, in at least one embodiment, the cost function for the kinematic realignment can be selected as follows: $$\mathrm{C.}\left(q_H,q_a\right)=\frac{1}{2}{\displaystyle {\sum }_{i=0}^{N}s\left(d_i\right){\left|\left|r_i\left(q_a\right)-f\left(d_i\right)r_i^{}\left(q_H\right)\right|\right|}^2+\gamma {\left|\left|q_a\right|\right|}^2}$$

where q _h , q _{a can be} the angles of the model of the human hand and the Allegro hand, respectively, r _i ∈ R ³ the vector that points from the origin of one coordinate system to another, expressed in the original coordinate system (see 7th ). In addition, in at least one embodiment $d_i=\left|\left|r_i\left(q_H\right)\right|\right|\text{and}{r}_i^{}\left(q_H\right)=\frac{r_i\left(q_H\right)}{\left|\left|r_i\left(q_H\right)\right|\right|}.$

The switching weight function s (d _i ) can be defined as follows: $$s\left(d_i\right)=\{\begin{array}{c}\mathrm{1,}{d}_i>\epsilon \\ 200d_i\le \epsilon \wedge r_i\left(q_H\right)\in {\mathrm{S.}}_1\\ \mathrm{400,}{d}_i\le \epsilon \wedge r_i\left(q_H\right)\in {\mathrm{S.}}_2\end{array}$$

where S ₁ can be vectors that start from a primary finger (index, middle, ring finger) and point to the thumb, and S ₂ m can be vectors between two primary fingers if both primary fingers have associated vectors ∈ S ₁ (e.g. both Primary fingers are projected with the thumb). In at least one embodiment, the distance function f (d _i ) ∈ R is defined as: $$f\left(d_i\right)=\{\begin{array}{c}\mathrm{\beta }{d}_i,d_i>\epsilon \\ {\mathrm{\eta }}_1,d_i\le \epsilon \wedge r_i\left(q_H\right)\in {\mathrm{S.}}_1\\ {\mathrm{<figure-callout\; id="2"\; label="\eta "\; filenames="DE102020127508A1\_0022.png,DE102020127508A1\_0035.png"\; state="\{\{state\}\}">\eta </figure-callout>}}_2,d_i\le \epsilon \wedge r_i\left(q_H\right)\in {\mathrm{S.}}_2\end{array}$$

where ß = 1.6 can be a scaling factor, η ₁ = 1 x 10 ^-4 m can be a distance between a primary finger and the thumb, and η ₂ = 3 x 10 ^-2 m can be a minimum distance between two primary fingers, when both primary fingers are projected with the thumb. In at least one embodiment, these projections ensure that the contacts between the primary fingers and the thumb are close together without the primary fingers colliding in a precision grip. In at least one exemplary embodiment, this can be particularly useful in the event of inaccuracies in visual finger tracking. In some examples, the vectors r _i can not only record the distance and direction from one task space to another, but their expression in local coordinates can also contain information about how the coordinate systems and thus the fingertips are oriented to one another. In at least one embodiment, the coordinate systems of the model of the human hand can therefore have equivalent coordinate systems on the Allegro model with similarity in orientation and placement. In the 5 The vectors shown can form a minimal set that generates the desired behavior when realigning (retargeting). In some exemplary embodiments, y = 2.5 x 10 ^{-3 can be} a weight in the regularization of the Allegro angles to zero (equivalent to the fully open hand). In at least one embodiment, this term helps to reduce the redundancy in the solution and to ensure that the hand never falls into unusual minima from which it is difficult to recover (e.g. when the fingers are embedded in the palm of the hand). In at least one embodiment, in 6th different mappings from the human hand 602-617 to an Allegro robot hand 618-633 shown how they are created by kinematic realignment.

In at least one embodiment, the above functionality is minimized in real time using the sequential least squares (SLSQP) programming algorithm. In at least one embodiment, the routine is initiated with the Allegro joint angles set to zero, and any solution thereafter can be initiated with the previous solution. In at least one exemplary embodiment, the forward kinematic calculations are carried out between the various coordinate systems of both the model of the human hand and of the Allegro hand. In at least one embodiment, a first order low pass filter is applied to the raw transmission joint angles to remove high frequency noise that occurs in tracking the human hand and to smooth discrete events, such as the projection algorithm that induces step response changes in the realigned angles.

In one embodiment, Riemann motion guidelines (“RMPs”) are real-time motion generation methods that calculate acceleration fields from potential functional gradients and corresponding Riemann metrics. RMPs can combine the generation of Cartesian trajectories with multiple priorities and collision avoidance behavior in a coherent reference system. In at least one embodiment, they are performed using the observed pose a human hand to control the Cartesian pose of the Allegro palm while avoiding arm or palm collisions with the table or operator using collision planes. In at least one embodiment, with these objectives in mind, the RMPs generated arm-joint target trajectories that are sent at 200 Hz to the torque-impedance controller of the arm. In at least one embodiment, the kinematically adjusted Allegro angles are sent to the arm's torque impedance controller at 30 Hz. In at least one exemplary embodiment, a teleoperation instance is initialized by registering the studio cameras via an initial, static robot pose and the initial observation of the human hand with the robot base coordinate system. In at least one exemplary embodiment, the axes of the human hand model and the axes of the robot end effector are approximately aligned in such a way that the direction of movement between the movement of the human hand and the robot movement is maintained.

Overall, the system can be used reliably to solve a large number of tasks with different degrees of difficulty. In some examples, the ability to solve these tasks shows that the system can have the dexterity, precision and power grips, multi-finger gripping and non-gripping manipulations, finger movements of the hand and compound manipulations of the hand (e.g. gripping with two fingers while manipulating with the rest of the fingers).

In at least one exemplary embodiment, the system can enable a strongly moving hand-arm system to find a motorized solution for a large number of manipulation tasks by translating observed human hand and finger movements into robot arm and finger movements. In at least one embodiment, multiple tasks, such as pulling paper money out of a wallet and opening a cardboard box in a plastic container, can be so complex that developing a robotic solution by hand or directly applying learning methods is likely unsolvable. Solving these and other tasks by the robot being carried out can enable these solutions to be generated on demand for many demonstrations. In addition, creating these solutions on the system itself can read, access and store the various tactile signals in the robot's fingertips, various commanded and measured joint position and speed signals through the hand and arm, various torque commands throughout the system and any camera feeds associated with the system. In at least one embodiment, this rich data source can be used together with demonstrations of tasks to solve complex, multi-level tasks with a long time horizon.

In one embodiment, a system is developed to track objects in the hand during manipulation by a robot. In various embodiments, the system can be implemented by one or more systems, as shown in FIG 13th - 41 described / illustrated. As in 7th As illustrated, the system may include a GPU accelerated high fidelity physics simulator as a forward dynamics model with a sample based optimization framework to track object poses with feedback from contacts. In at least one embodiment, a simultaneous set of simulations is initialized with the initial states of a real robot and the initial pose of a real object, which can be obtained from a vision-based algorithm for pose registration, assuming that the object is not covered at the beginning. In at least one exemplary embodiment, the initial poses of the simulated objects are slightly perturbed and reflect the uncertainty of the vision-based pose registration algorithm. In at least one embodiment, the GPU-accelerated physics simulator runs many simultaneous simulations in real time on a single GPU. In at least one embodiment, a predetermined guideline is used that controls the real robot to approach, grip, and hand-manipulate the object, and the same robot control commands are executed on the simulated robots. In at least one embodiment, observations of the real robot and the simulated robots are collected, which include terms such as the size and direction of contacts on the touch sensors of the robot hand. In at least one embodiment, a sample-based optimization algorithm is used that periodically updates the states and parameters of the simulations according to a cost function that records how well the observations of each simulation match those of the real world. In addition, in at least one exemplary embodiment, the algorithm also updates simulation parameters such as mass and friction in order to further improve the dynamic models of the real world of the simulations. At any point in time, the object pose estimate can be the pose of the robot object system.

In various exemplary embodiments, a total of 24 trajectories for the manipulation in the hand with three different objects in the simulation can be used to evaluate the proposed algorithm and can be collected in the real world, although any number of trajectories can be collected. In at least one embodiment, a robot arm such as the Kuka IIWA7 arm with the 4-finger Wonik Robotics Allegro hand is used as the end effector, with each finger being equipped with a SynTouch BioTac touch sensor. In at least one exemplary embodiment, the object manipulation trajectories are human demonstrations that were collected via a teleoperation system with hand tracking. In various exemplary embodiments, on the basis of the object poses of the basic truths in the simulation, detailed ablation studies are carried out in simulation experiments in order to evaluate the properties of the proposed algorithm. In at least one embodiment, a vision-based algorithm is used to obtain the object pose in the first and last individual image of the collected trajectories in which the object is not concealed. In at least one embodiment, the pose in the first frame is used to initialize the simulations and the pose in the last frame is used to assess the accuracy of the proposed contact-based algorithm.

7th Figure 10 shows an embodiment of a framework for tracking poses of objects in the hand. In at least one embodiment, robot controls 702 sent to a GPU accelerated physics simulator that runs 708 many robot simulations in parallel, each with different physics parameters and changed (disturbed) object poses. In at least one embodiment, costs based on observations, such as B. Feedback through contacts from the real world and from the simulations to a sample-based, derivative-free optimizer 704 forwarded, which periodically updates the states and parameters of all simulations in order to better adapt them to the real world. In at least one exemplary embodiment, the pose of the simulation with the lowest cost is used as the current object pose estimate at any point in time 706 selected.

In one embodiment, a system tracks the pose of an object that is hand-held by a robotic manipulator during object manipulation. In some embodiments, for the time that can be represented by t, an object _{pose can be defined as p t} ∈ SE (3), and a physical dynamics model can be defined as s _{t + 1} = f (s _t , u _t , 0) where s _{t can be} the state of the real world (position and velocities of rigid bodies and of joint angles in movable bodies), u _t ∈ ℝ ^{M can be} the control of the robot (desired joint positions can be used as an action space), and θ ∈ ℝ ^{N can be} the fixed parameters of the simulation (eg, mass and friction).

In various exemplary embodiments, the pose estimation for a simulation model f, which with perfect initializations of p ₀ , s ₀ and θ exactly matches reality, can only _{require the return of the sequence of actions u t} that are applied to the robot in the simulation. However, since the forward models can be imperfect and the pose initializations can be noisy, feedback from observations can improve pose estimation.

(die sich an den Fingerspitzen befinden können), die Kraftvektoren der abgetasteten Kontakte $c_t^{\left(l\right)}\in {ℝ}^3,$

der Einheitsvektor in Richtung des translatorischen Schlupfes auf der Berührungsfläche $d_t^{\left(l\right)}\in$

ℝ², und der binären Richtung des rotatorischen Schlupfes auf der Berührungsfläche $r_t^{\left(l\right)}\in$

{0, 11, wobei / den l-ten Berührungssensor indizieren kann. In mindestens einem Ausführungsbeispiel wird zum allgemeinen Bestimmen der Posenabschätzung in der Hand anhand der aktuellen und vergangenen Beobachtungen o_1:t, der Steuerung des Roboters u_1:t und der initialen Pose p₀ die wahrscheinlichste aktuelle Objektpose p_t ermittelt.In some embodiments, D can be defined as the number of joints of the robot and L as the number of its touch sensors. An observation _{vector o t} can be defined as the concatenation of the joint position configuration of the robot q _t ∈ ℝ ^D , the position and rotation of the robot's touch sensors $P_t^{\left(l\right)}\in {ℝ}^3,{\mathrm{R.}}_t^{\left(l\right)}\in \mathrm{S.}O\left(3\right)$

(which may be on the fingertips), the force vectors of the contacts being sensed $c_t^{\left(l\right)}\in {ℝ}^3,$

the unit vector in the direction of the translational slip on the contact surface $d_t^{\left(l\right)}\in$

ℝ ² , and the binary direction of the rotational slip on the contact surface $r_t^{\left(l\right)}\in$

{0, 11, where / can index the lth touch sensor. In at least one embodiment, the most likely current object _{pose p t is} determined for the general determination of the pose estimation in the hand based on the current and past observations o _{1: t} , the control of the robot u _{1: t} and the initial pose p _0.

In various embodiments, a GPU-accelerated physics simulator can be used as a forward dynamics model to simulate many robot object environments simultaneously in order to track the object pose in hand, and derivative-free, sample-based optimizers can be used to collect the state and parameters of the simulations vote to improve tracking performance. 9 Figure 3 shows one embodiment of an algorithm that can be used.

und $o_t^{\left(i\right)}$

definiert werden, und die Beobachtungen der Grundwahrheit können als $o_t^{\left(gt\right)}$

definiert werden. In verschiedenen Ausführungsbeispielen kann bei einer Kostenfunktion C die aktuell beste Posenabschätzung zum Zeitpunkt t die Pose der i-ten Simulation, $p_t^{\left(i\ast \right)},$

sein, wobei die i-te Simulation diejenige sein kann, die die niedrigsten Durchschnittskosten über ein vergangenes Zeitfenster T verursacht: $$\begin{array}{l}{C}_i=\frac{1}{T}{\displaystyle \sum _{\mathrm{\Delta }t=0}^{T-1}C\left(o_{\left(t-\mathrm{\Delta }t\right)}^{\left(i\right)},o_{\left(t-\mathrm{\Delta }t\right)}^{\left(gt\right)}\right)}\\ i*=\text{arg min }{C}_i\end{array}$$

First, an estimate of an initial object pose can be obtained via a vision-based object pose estimator. It can be assumed that the pose estimator provides a reliable initial pose estimate p ₀ when the robot is not in contact with the object and when the object is not covered, e.g. B. before grasping. With the initial object pose estimation and the robot configuration, K simultaneous simulations can then be initialized, and the real robot actions u _t can be copied into all K simulations at each time step. In various examples, the object pose can change when the hand makes contact, and this can be modeled by the simulator. In various exemplary embodiments, the object pose and the observation of the i-th simulation can be used as $p_t^{\left(i\right)}$

and $O_t^{\left(i\right)}$

can be defined, and the observations of the fundamental truth can be called $O_t^{\left(Gt\right)}$

To be defined. In various exemplary embodiments, for a cost function C, the currently best pose estimate at time t can be the pose of the i-th simulation, $p_t^{\left(i\ast \right)},$

where the i-th simulation can be the one that causes the lowest average costs over a past time window T: $$\begin{array}{l}{\mathrm{C.}}_i=\frac{1}{T}{\displaystyle \sum _{\mathrm{\Delta }t=0}^{T-1}\mathrm{C.}\left(O_{\left(t-\mathrm{\Delta }t\right)}^{\left(i\right)},O_{\left(t-\mathrm{\Delta }t\right)}^{\left(Gt\right)}\right)}\\ i*=\text{bad min}{\mathrm{C.}}_i\end{array}$$

The costs can be used to periodically update the simulations and their parameters, which can enable an improved correspondence with the real robot object system.

In some embodiments, a cost function can be used that sufficiently correlates with differences between object poses during hand manipulation so that lower costs correspond to better estimates of the poses. The cost function can be represented by the following symbolic mathematical equation: $$\begin{array}{l}\mathrm{C.}\left(O_{\left(t-\mathrm{\Delta }t\right)}^{\left(i\right)},O_{\left(t-\mathrm{\Delta }t\right)}^{\left(Gt\right)}\right)\\ \begin{array}{l}=w_1\left|\left|q_t^{\left(i\right)}-q_t^{\left(Gt\right)}\right|\right|2\hfill \\ +{\displaystyle \sum _{l=1}^{\mathrm{L.}}(w_2\left|\left|P_t^{\left(i,l\right)}-P_t^{\left(Gt,l\right)}\right|\right|2+w_3\left|\mathrm{\Delta }\left({\mathrm{R.}}_t^{\left(i,l\right)},{\mathrm{R.}}_t^{\left(Gt,l\right)}\right)\right|+w_{\mathrm{4th}}\left(1-{\alpha }_{\left(i,l\right)}\right)}\hfill \\ +w_5\left({\alpha }_{\left(i,l\right)}\right)\left|\mathrm{\Delta }\mathrm{M.}\left(c_t^{\left(i,l\right)},c_t^{\left(Gt,l\right)}\right)\right|+w_{\mathrm{6th}}\left({\alpha }_{\left(i,l\right)}\right)\left|\mathrm{\Delta }\phi \left(c_t^{\left(i,l\right)},c_t^{\left(Gt,l\right)}\right)\right|+w_{\mathrm{7th}}\left(1-{\beta }_{\left(i,l\right)}\right)\hfill \\ +w_{\mathrm{8th}}\left({\beta }_{\left(i,l\right)}\right)\left|\mathrm{\Delta }\phi \left(d_t^{\left(i,l\right)},d_t^{\left(Gt,l\right)}\right)\right|+w_9\left(1-{\gamma }_{\left(i,l\right)}\right)+w_{10}\left({\gamma }_{\left(i,l\right)}\right)\left|\left(r_t^{\left(i,l\right)}-r_t^{\left(Gt,l\right)}\right)\right|)\hfill \end{array}\end{array}$$

For the first term in the cost function, a comparison of q _t s between the simulated and the real robots can be useful, even if they have the same u _t , since q _t depends on the collision restrictions imposed by the current object pose in contact with the robot hand can be different, which can make it physically impossible for a joint to achieve a commanded target angle.

In at least one embodiment, a touch sensor is considered to be in contact if its force magnitude is greater than a certain threshold value. In at least one embodiment, α ( _{i, l} ) is 1 if the binary contact state of the i-th touch sensor of the i-th simulation matches that of the real touch sensor, and otherwise it is 0. In at least one embodiment, β ( _{i, l} ) equals 1 if the l-th touch sensor of the i-th simulation agrees with the real touch sensor in terms of whether the sensor experiences translational slip or not, otherwise 0; γ _{(i, l)} is the same for rotating slip.

In at least one embodiment, ΔM (•, •) indicates the difference in their magnitudes for any two vectors, and Δφ (•, •) indicates the angle between them. In at least one exemplary embodiment, for any two rotations R _a and R _b , Δ (R _a , R _b ) specifies the angle of the axis angle representation of R _a ^-1 R _b . In at least one exemplary embodiment, the weights of the cost _{terms, w i} s, are determined in such a way that the corresponding mean amount of each term is approximately normalized to 1.

In various exemplary embodiments, two sources of uncertainty can occur in the estimation of the object pose by the simulation: 1) the initial pose estimation p ₀ from the vision-based pose estimator can be noisy, and 2) there can be a mismatch between the simulated and the real dynamics, partly caused by imperfect modeling and partly due to the unknown physical parameters θ of the real world.

zentriert ist, und die Anzahl der Simulationen K erhöht wird. In verschiedenen Ausführungsbeispielen, wenn K beliebig groß ist, kann es mit hoher Wahrscheinlichkeit sein, dass die wahre Ausgangspose in der Menge der Simulationen ausreichend repräsentiert ist, und eine gut entworfene Kostenfunktion die richtige Simulation mit der richtigen Pose auswählen kann. In mindestens einem Ausführungsbeispiel werden die Translation und die Rotation getrennt abgetastet, um eine Abtastung über die initialen Objektposen durchzuführen. In mindestens einem Ausführungsbeispiel wird die Translation aus einer isotropen Normalverteilung abgetastet, während die Rotation abgetastet werden kann, indem isotrope Tangentialvektoren mit Mittelwert Null in so(3) gezeichnet und dann auf die mittlere Rotation angewendet werden.In at least one embodiment, the first problem of the uncertainty of the initial pose estimate is addressed by perturbing the initial pose estimates across the various simulations by taking samples from a distribution that is around the vision-based estimated pose $p_0^{\left(i\right)}\sim N\left({\mathrm{<figure-callout\; id="0"\; label="p"\; filenames="DE102020127508A1\_0028.png,DE102020127508A1\_0046.png"\; state="\{\{state\}\}">p</figure-callout>}}_0,{\mathrm{\Sigma }}_p\right)$

is centered, and the number of simulations K is increased. In various exemplary embodiments, if K is arbitrarily large, there can be a high probability that the true starting pose is sufficiently represented in the set of simulations, and a well-designed cost function can select the correct simulation with the correct pose. In at least one exemplary embodiment, the translation and the rotation are scanned separately in order to carry out scanning over the initial object poses. In at least one embodiment, the translation is sampled from an isotropic normal distribution, while the rotation can be sampled by drawing isotropic tangential vectors with mean value zero in so (3) and then applying them to the mean rotation.

In at least one embodiment, a second problem of the mismatch between simulated and real physics (“sim to real” gap) is dealt with by using derivative-free, sample-based optimization algorithms for matching θ during pose tracking. In various exemplary embodiments, after each T time steps, the average costs of all simulations during this window can be transferred to a specific optimizer together with the simulation status and the parameters. In at least one embodiment, the optimizer determines the next set of simulations with their own updated parameters. In at least one embodiment, the simulations of the next sentence are taken from the simulations of the current sentence, with some additional disturbances (deviations) being added to the simulation parameters and the object pose. In at least one embodiment, such an approach can maintain the variety of simulations and prevent them from getting stuck in suboptimal simulation parameters or states due to noisy observations.

Although it may be desirable for θ ^{(i *)} to converge to the true θ ^(gt) , this may not be necessary to achieve a good pose estimate. In addition, due to the differences between the simulated and the real dynamics, the optimal 0 for reducing C to the corresponding values in the real world may have slight deviations from the calculated theoretical predictions.

The parameters of the K simulations can be optimized using three derivative-free, sample-based optimizers:

In one embodiment, a weighted resampling ("WRS") optimizer is used. In various exemplary embodiments, WRS can form a probability mass function (PMF) over the existing simulation ^{states s (1: K)} and sample K times with replacement from this distribution in order to form the next set of simulations. To form the PMF, WRS Softmax can apply to the simulation costs: $$P\left(i\right)=\frac{exp-\lambda \left({\mathrm{C.}}_i-mi{n}_j{\mathrm{C.}}_j\right)}{{\displaystyle {\sum }_{i=1}^{K}exp-\lambda \left({\mathrm{C.}}_i-mi{n}_j{\mathrm{C.}}_j\right)}}$$

Here, λ can be a temperature hyperparameter that determines the sharpness of the distribution. In at least one embodiment, after the resampling, an exploration of simulations is carried out by perturbing the simulation parameters θ and the object pose.

wobei Σ_θ vordefiniert sein kann. Der tiefgestellte Index τ kann den Aktualisierungsschritt des Optimierers bezeichnen (nach r Aktualisierungsschritten ist die Simulation insgesamt rT Zeitschritte lang gelaufen).In at least one embodiment, the simulation parameters are perturbed by samples from an isotropic normal distribution around the previous parameters: ${\theta }_{\mathrm{\tau }+1}^{\left(i\right)}\sim N\left({\theta }_{\mathrm{\tau }}^{\left(i\right)},{\mathrm{\Sigma }}_{\theta }\right),$

where Σ _θ can be predefined. The subscript τ can denote the update step of the optimizer (after r update steps, the simulation has run for a total of rT time steps).

In one embodiment, a relative entropy optimizer (Relative Entropy Policy Search, “REPS”) is used. In at least one exemplary embodiment, a sample-based variant of REPS is used which calculates weights for each simulation and takes samples from a distribution which is formed by the soft max of these weights. While WRS can use a fixed A-parameter to shape the distribution, REPS can look for an adaptive temperature parameter η that best improves the performance of the overall distribution as a function of ∈, which is a limitation of the KL divergence between the old and the updated sample distribution can be.

$$P\left(i\right)=\frac{\text{exp}\frac{R_i}{\eta *}}{{\displaystyle {\sum }_{j=1}^K\text{exp}\frac{R_i}{\eta *}}}$$

To use REPS, in at least one embodiment, the costs are reformulated as rewards by setting R _i = max _j C _j + mn _j C _{j −C i} . The parameter η can be calculated in each step by optimizing the dual function g (η), and then η can be used to form the PMF: $$\eta *=\text{bad min}\eta \epsilon +\eta lOG\frac{1}{K}{\displaystyle \sum _{i=1}^K\text{exp}\frac{{\mathrm{R.}}_i}{\eta }}$$

$$P\left(i\right)=\frac{\text{exp}\frac{{\mathrm{R.}}_i}{\eta *}}{{\displaystyle {\sum }_{j=1}^K\text{exp}\frac{{\mathrm{R.}}_i}{\eta *}}}$$

After resampling, a simulation can be perturbed in the same way as with WRS.

A population-based optimization algorithm (“PBO”) can be used in various exemplary embodiments. In at least one exemplary embodiment, the PBO algorithm ranks all simulations according to their average costs and finds the best K _best simulations with the lowest costs. In at least one embodiment, the algorithm exploits by replacing the remaining K - K _best simulations with copies of the K _best simulations sampled with replacement and explored by performing the K _best simulations in the same way as at WRS disturbs (English perturb). In at least one embodiment, PBO uses effectively shaped costs that depend only on the relative placement of the simulation costs and not on their magnitudes, which may make the optimizer more robust to noisy costs.

• • T, as die Zeitschritte repräsentieren kann, die ein Algorithmus bei jeder Aktualisierung wartet.
• • K, das die Anzahl der gleichzeitigen Simulationen repräsentieren kann.
• • θ₀, das die initiale Normalverteilung der Simulationsparameter repräsentieren kann.
• • Σ_p, das die diagonale Kovarianzmatrix für die Normalverteilung über die initiale Posenstörung repräsentieren kann.
• • Σ_θ und Σ_ν, die die diagonalen Kovarianzen der Normalverteilungen der für die Exploration verwendeten Störungen repräsentieren können.

In at least one embodiment, the optimizers described above use a distribution-shaping hyperparameter that is used to strike a balance between exploration and exploitation. In at least one embodiment, various embodiments may use combinations of additional hyperparameters such as the following:

• • T, as can represent the time steps that an algorithm waits for each update.
• • K, which can represent the number of simultaneous simulations.
• • θ ₀ , which can represent the initial normal distribution of the simulation parameters.
• • Σ _p , which can represent the diagonal covariance matrix for the normal distribution over the initial pose disturbance.
• • Σ _θ and Σ _ν , which can represent the diagonal covariances of the normal distributions of the perturbations used for exploration.

A larger K can generally be better than a smaller K, with the caveat that the resulting simulation can be slower and not practical in the application. Σ _p can be so large that the actual initial pose is well represented in the initial pose distribution. However, with a larger Σ _p and the convariance of θ ₀ , K can be increased to ensure that the sample density is high enough to cover a wider distribution.

In at least one embodiment, there are two additional tradeoffs with these hyperparameters. In at least one embodiment, one trade-off is the trade-off between exploration and exploitation in connection with the optimization for θ, and the other trade-off is the trade-off between the optimization of θ and p _t ^{(i *)} . In at least one embodiment, expanding Σ _θ or Σ _{ν increases} the speed at which the set of simulation parameters "moves" and the optimizer can explore more than exploit. In at least one embodiment, increasing T improves the optimization for θ because the optimizer can have more samples to evaluate each simulation. However, updating the simulation parameters too slowly can lead to a drift in the pose estimation if the simulation with the lowest cost differs sufficiently from the real world, which in some examples can lead to divergent behavior. In some examples, deviating behavior may occur when a force pertubation or some simulation parameters result in an unrecoverable configuration in which the object falls out of the hand or the object is posed in such a way that small force pertubations no longer put it in the correct pose can bring back. In some examples this may be acceptable if a few samples are divergent. Their cost can be high, so in some embodiments they can be discarded and replaced with ones that are not divergent during the optimization update.

In various exemplary embodiments, a physics simulator can be used as a forward model instead of a constant model. In addition to tracking the object pose, the proposed algorithm can also identify the context of the forward model by setting the simulation parameters θ that cannot be influenced by the forward or observation model. The optimizers can be based on discrete sampling because poses of objects in the hand from a given distribution cannot be easily sampled due to complicated network penetration restrictions by contacts.

In various exemplary embodiments, the system can be evaluated both with simulations and with experiments in the real world, using an Allegro hand on a Kuka IIWA7 robot arm, although in various exemplary embodiments any hand, any robot arm, any robot component and / or variations thereof can be used. The trajectories of the manipulation in the hand can first be collected with a teleoperation system for hand tracking, and the errors in the pose estimation can be evaluated by running the proposed algorithms offline on the basis of the collected trajectories. 9 Figure 3 shows an example of an algorithm that can be used by the system. These trajectories can begin and end with the object not in the hand and not in obscurity. Due to the presence of object poses of a fundamental truth in simulation experiments, detailed ablation studies can be performed in the simulation to examine the effects of various hyperparameters on algorithm performance. In various exemplary embodiments, PoseRBPF, a newer algorithm for pose estimation based on RGB D and particle filters, can be used to obtain the initial and final object pose. The initial and final object poses can be treated as basic truths and the final pose can be compared with the pose predicted by the system.

In various embodiments, the Allegro hand with 4 fingers and 16 DoF can be mounted on the Kuka IIWA7 robot arm with 7 DoF. To get feedback through contacts in the real world, SynTouch BioTac sensors or a variation of them can be attached to each of the fingertips. One or more processes performed in conjunction with various sensors and the raw electrode readings from the sensors can be used to predict the contact force, slip direction, and grip stability. A trained model can be used to estimate _{force vectors c t.} In various examples, the cost functions may not include terms for slip. The simulations can be carried out on a computer with one or more graphic processing units, one or more central processing units and one or more storage units.

In various exemplary embodiments, three objects from the Yale Columbia Berkeley (“YCB”) object data set (spam can, foam block and toy banana) with models, textures and point clouds can be used. The objects can be selected based on constraints on the size of the robotic hand and that the objects are light enough for the robotic hand (e.g., the Allegro hand) to perform robust precision grips with its fingertips.

In some examples, for each object, both in simulation and in real experiments, 2 demonstrations of 2 types of trajectories can be used for manipulation: 1) Record and placing (pick-and-place) by means of a finger grip and rotating the object in the hand and 2) the same, but with fingertips that interrupt and re-establish contact during the grip (finger movements). This can result in a total of 24 trajectories that can be analyzed for both simulations and experiments in the real world. With both trajectories, the object can be displaced translationally and rotatingly by inertial forces as well as by pressure contact with the table. Each trajectory can take about a minute. In various exemplary embodiments, the algorithm for estimating the pose can be carried out at approx. 30 Hz, which can result in a total of approx.

In at least one embodiment, the input to the system is a point cloud of the hand of the human demonstrator. In at least one embodiment, a neural network, e.g. a neural network based on the PointNet ++ neural network, assigns the point cloud to an estimate of the hand pose relative to the camera and the joint angles of the hand. In at least one embodiment, these estimates, along with a moving hand model and the original point cloud, are then passed to DART, which carries out the tracking by refining the neural network estimates. In order to finally perform a kinematic realignment, an optimization problem is solved that finds the joint angles of the Allegro hand that lead to fingertip poses that come close to those of the human hand, according to one embodiment.

In addition to the optimizers mentioned above (WRS, REPS, PBO), the following two basic functions can also be evaluated: Open Loop (“OLP”) and Identity (“EYE”). OLP can track the object pose with 1 simulation. EYE can be initialized with a set of noisy initial poses and can always choose the pose of the simulation with the lowest cost, but cannot resample or perform optimization updates. In various exemplary embodiments, an average distance deviation (“ADD”) can be used as an evaluation standard. ADD can calculate the average distance between corresponding points in the object point cloud that are in the fundamental truth pose and in the predicted pose.

In various exemplary embodiments, the arm and hand in the simulation can be controlled via a joint angle PD controller, and the gains of the controller can be set so that the joint angle step responses are similar to those of the real robot. To speed up the simulation, the collision nets of the robot and the objects can be simplified. This can be done with various tools, e.g. By applying an algorithm such as the TetWild algorithm which yields a mesh with triangles that are more equilateral than other algorithms such as e.g. B. the Quadric Edge Collapse Decimation from MeshLab 1. In total, each simulation can generate a maximum of 200 contacts during the manipulation, and K = 40 simulations with 30 Hz can be carried out.

The simulation experiments can be carried out with different amounts of initial noise of the initial pose. Three levels can be tested: “Low” can have a standard deviation of translation of 1 mm and a standard deviation of rotation of 0.01 radians. “Medium” can be 5 mm and 0.1 radians, and “High” can be 10 mm and 1 radian.

8th shows an example of a comparison of the optimizers when tracking object poses in hand across all trajectories of the simulation. ADD 802 can increase with increasing initial pose error, and the mean ADD for the optimizer-based methods can be lower. While EYE can achieve a comparable mean ADD with the optimizer methods, the latter can have a much smaller error variance and a smaller maximum error. The optimizers 804 can focus the distribution of simulations over time on the more powerful. In the case of medium noise, REPS and PBO can achieve the best ADD with a mean value of 5.8 mm and 5.9 mm, respectively.

Show in one embodiment 10 and 11 Examples of results of ablation studies in simulations performed with the hyperparameters that include the exploration distance (how much the simulations are perturbed), the number of parallel simulations, and whether or not the feedback for contact and slip detection is used in the cost function do not determine.

The algorithms can be evaluated on trajectories from the real world that are similar to those from the simulation. PoseRBPF can be used to register the object pose in the first and last frame of a trajectory. The initial pose estimate can be used to initialize the simulations, while the last can be used to evaluate the accuracy of the contact-based pose tracking algorithm. In contrast to the simulation experiments, the In experiments in the real world, the object pose can be initialized by samples from the distribution over the object poses of PoseRBPF, so that the initial pose samples correspond to the uncertainties of the vision-based algorithm for the pose estimation.

12th shows an example of real test results. The ADDs can be higher than those from simulation experiments. This can be due to the fact that the dynamics in the real world are more dissimilar to the simulations than the simulations with other parameters, as well as to the fact that the observations in the real world can be more noisy than those in the simulations. In various embodiments, few or no optimizers may be able to keep track of the toy banana for the real world data. The long lever arm of the object and the low coefficient of friction can make the slip behavior of the object difficult to precisely model. This can be an algorithm failure mode where if all simulations diverge (e.g. the banana rotates in the wrong direction or falls out of hand), the algorithm cannot recover in subsequent optimizer updates. The best ADD achieved with foam can be 14.1mm with PBO, and with Spam 12.2mm with REPS.

In at least one embodiment, a sample-based optimization algorithm is developed for tracking poses of objects in the hand during manipulation through feedback through contacts and GPU-accelerated robot simulation. In at least one embodiment, many principles about the real world are simultaneously maintained in parallel simulations and changes in the object pose are modeled that can be caused by complex contact dynamics. In at least one embodiment, the optimization algorithm adjusts the simulation parameters while tracking the object pose to further improve tracking performance. In various exemplary embodiments, the detection of contacts in the real world can be improved through the use of vision in the loop.

13th Figure 13 shows an example of a process that, as a result of being executed by a computer system, determines the pose of an object that is manipulated by a robotic hand equipped with tactile force sensors. In at least one embodiment, a computer system having one or more processors executes executable instructions that are stored in computer-readable memory and, as a result of execution, cause the computer system to be included in 13th operations shown and described below.

In at least one embodiment, the computer system receives in step 1302 tactile information from one or more touch sensors on a robotic hand that is used to manipulate an object. In at least one embodiment, the touch sensors are BioTac sensors that are provided with a friction coating. In at least one embodiment, the touch sensor data is a two-dimensional array of force data that represents forces distributed over the surface of the touch sensor.

In at least one embodiment, the computer system generates in step 1304 a set of simulations that simulate the manipulation performed by the robot and determines a set of simulated touch-sensor information 1306 for every simulation. In at least one embodiment, each simulation uses a different pose for the object. In at least one embodiment, an estimated pose for the object is obtained from one or more images of the object.

In at least one embodiment, the computer system determines in step 1308 a cost value for each simulation based at least in part on the differences between the touch sensor information measured in the real world and the touch information determined in the simulation. In one embodiment, the cost can be determined as described above. Based at least in part on the costs determined for each simulation, a simulation is identified that most closely matches the real world observations 1310. In at least one embodiment, the pose of the object in the identified simulation is identified as the pose of the object in the real world determined 1312.

In at least one embodiment, the pose can be used when performing a task that involves manipulating the object in the hand. For example, the pose of a nut in the hand can be determined so that the robot can position the nut on a threaded bolt. In another example, the pose of an object in hand can be used when the object is on a horizontal surface is set to ensure that the object is upright and stable when the robot hand releases it.

Interference and training logic

14A shows the inference and / or training logic 1415 used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are related to below 14A and / or 14B provided.

In at least one embodiment, the inference and / or training logic 1415 a code and / or data memory without restriction 1401 for storing forward and / or output weight and / or input / output data and / or other parameters for configuring neurons or layers of a neural network that are trained and / or used for inference, according to aspects of one or more exemplary embodiments. In at least one embodiment, the training logic 1415 the code and / or the data memory 1401 or be coupled to, to store graph code or other software to control the timing and / or order in which weight and / or other parameter information is loaded to configure the logic, including integer and / or floating point units (collectively, arithmetic -Logical units (ALUs)). In at least one embodiment, code, such as graph code, weight or other parameter information, loads into processor ALUs based on a neural network architecture to which the code corresponds. In at least one embodiment, the code and / or data memory stores 1401 Weight parameters and / or input / output data of each layer of a neural network that are trained or used during the forward propagation of input / output data and / or weight parameters during training and / or inferencing using aspects of one or more exemplary embodiments. In at least one embodiment, any part of the code and / or data memory 1401 be grouped with other on-chip or off-chip data storage, including the L1, L2, or L3 cache or system memory of a processor.

In at least one embodiment, any part of the code and / or data memory 1401 be implemented internally or externally by one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or the code and / or data memory 1401 a cache memory, a dynamic, randomly addressable memory ("DRAM"), a static, randomly addressable memory ("SRAM"), a non-volatile memory (e.g. flash memory) Memory) or another memory. In at least one embodiment, the choice of whether the code and / or the code and / or data memory 1401 For example, it is internal or external to the processor or consists of DRAM, SRAM, Flash or another type of memory, the available memory on the chip compared to the memory outside the chip, the latency requirements of the training and / or inference functions carried out, the stack size of the inference and / or the training of a neural network data used or a combination of these factors depend.

In at least one embodiment, the inference and / or training logic 1415 without limitation a code and / or a data memory 1405 to store backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the code and / or data memory stores 1405 Weight parameters and / or input / output data of each layer of a neural network that are trained or used during backward propagation of input / output data and / or weight parameters during training and / or inferencing using aspects of one or more exemplary embodiments. In at least one embodiment, the training logic 1415 a code and / or data memory 1405 or be coupled to, to store graph code or other software to control the timing and / or order in which weight and / or other parameter information is to be loaded to configure the logic, including integer and / or floating point units ( together of arithmetic-logical units (ALUs)). In at least one embodiment, code, such as graph code, weight or other parameter information, loads into processor ALUs based on a neural network architecture to which the code corresponds. In at least one embodiment, anyone can Part of the code and / or data memory 1405 with other on-chip or off-chip data memories, including the L1, L2 or L3 cache memory or the system memory of a processor. In at least one embodiment, any part of the code and / or data memory 1405 be contained internally or externally in one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data memory 1405 Cache memory, DRAM, SRAM, non-volatile memory (e.g. flash memory) or another memory. In at least one embodiment, the choice of whether the code and / or data memory 1405 For example, processor-internal or processor-external or consists of DRAM, SRAM, Flash or another type of memory, of the available memory on-chip or off-chip, of the latency requirements of the training and / or inference functions carried out, of the stack size of the inference and / or the training of a neural network data used or depend on a combination of these factors.

In at least one embodiment, the code and / or data memory 1401 and the code and / or data memory 1405 be separate storage structures. In at least one embodiment, the code and / or data memory 1401 and the code and / or data memory 1405 be the same memory structure. In at least one embodiment, the code and / or data memory 1401 and the code and / or data memory 1405 partly the same memory structure and partly separate memory structures. In at least one embodiment, any part of the code and / or data memory 1401 and the code and / or data memory 1405 be grouped with other on-chip or off-chip data storage, including the L1, L2, or L3 cache memory or the system memory of a processor.

In at least one embodiment, the inference and / or training logic 1415 comprise, without limitation, one or more arithmetic-logical unit (s) ("ALU (s)") 1410, including integer and / or floating point units, in order to perform logical and / or mathematical operations that are at least partially based on a training and / or or inference code based on or indicated by this (e.g. graph code), the result of which is stored in an activation memory 1420 stored activations (eg output values of layers or neurons within a neural network) can generate the functions of in code and / or data memory 1401 and / or code and / or data memory 1405 stored input / output and / or weight parameter data. In at least one embodiment, an activation memory 1420 generated stored activations according to linear algebraic and / or matrix-based mathematics executed by ALU (s) 1410 in response to execution of instructions or other code, in code and / or data memory 1405 and / or data 1401 Stored weight values as operands together with other values, such as, for example, default values (bias values), gradient information, pulse values or other parameters or hyperparameters, which can be stored as desired or completely in code and / or data memories 1405 or code and / or data storage 1401 or another memory on or off-chip can be used.

In at least one embodiment, the ALU (s) 1410 are included in one or more processors or other hardware logic devices or circuits, while in another embodiment the ALU (s) 1410 are external to a processor or other hardware logic device or circuit they can use (e.g. a co-processor). In at least one embodiment, the ALU (s) 1410 can be contained in the execution units of a processor or otherwise in a group of ALUs to which the execution units of a processor either within the same processor or distributed among different processors of different types (e.g. central processing units, graphics processing units, units with fixed functionality, etc.). In at least one embodiment, the data memory 1401 , the code and / or data memory 1405 and the activation memory 1420 reside on the same processor or other hardware logic device or circuit, while in another embodiment they reside in different processors or other hardware logic devices or circuits, or in a combination of the same and different processors or other hardware logic devices or circuits can. In at least one embodiment, any part of the activation memory 1420 be grouped with other on-chip or off-chip data storage, including the L1, L2, or L3 cache, or the system memory of a processor. Furthermore, the inference and / or training code can be stored along with other code accessible by a processor or other hardware logic or circuitry and using the retrieval, decoding, scheduling, execution, shutdown and / or other logical circuits of a processor can be called up and / or processed.

In at least one embodiment, the activation memory 1420 Cache memory, DRAM, SRAM, non-volatile memory (e.g. flash memory) or another memory. In at least one embodiment, the activation memory 1420 wholly or partially inside or outside of one or more processors or other logical circuits. In at least one Embodiment can choose whether the activation memory 1420 For example, processor-internal or processor-external or consists of DRAM, SRAM, Flash or another type of memory, of the available memory on the chip or off-chip, of the latency requirements of the training and / or inference functions carried out, of the stack size of the inference and / or the training of a neural network data used or depend on a combination of these factors. In at least one embodiment, the in 14A illustrated inference and / or training logic 1415 can be used in conjunction with an application-specific integrated circuit ("ASIC"), such as the Tensorflow® Processing Unit from Google, an Inference Processing Unit (IPU) from Graphcore ™ or a Nervana® (eg “Lake Crest”) processor from Intel Corp. . In at least one embodiment, the in 14A illustrated inference and / or training logic 1415 can be used in conjunction with hardware of the Central Processing Unit (“CPU”), hardware of the Graphics Processing Unit (“GPU”) or other hardware such as Field Programmable Gate Arrays (“FPGAs”).

14B shows the inference and / or training logic 1415 different according to at least one embodiment. In at least one embodiment, the inference and / or training logic 1415 include, without limitation, hardware logic in which computing resources are dedicated or otherwise used solely in conjunction with weight values or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, the in 14B illustrated inference - and / or training logic 1415 can be used in conjunction with an application-specific integrated circuit (ASIC), such as the Tensorflow® Processing Unit from Google, an Inference Processing Unit (IPU) from Graphcore ™ or a Nervana® (eg “Lake Crest”) processor from Intel Corp. In at least one embodiment, the in 14B illustrated inference and / or training logic 1415 can be used in conjunction with the hardware of the central processing unit (CPU), the hardware of the graphics processing unit (GPU) or other hardware such as field programmable gate arrays (FPGAs). In at least one embodiment, the inference and / or training logic includes 1415 the code and / or data memory without restriction 1401 and the code and / or data memory 1405 that can be used to store code (e.g. graph code), weight values, and / or other information, including default values, gradient information, pulse values, and / or other parameter or hyper-parameter information. In at least one embodiment described in 14B is shown is both the code and / or data memory 1401 as well as the code and / or data memory 1405 associated with a dedicated computing resource, such as computing hardware 1402 or the computing hardware 1406 . In at least one embodiment, both include computing hardware 1402 as well as the computing hardware 1406 one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information contained in the code and / or data memory 1401 or in the code and / or data memory 1405 are stored and their result in the activation memory 1420 is saved.

In at least one embodiment, each of the code and / or data stores corresponds 1401 and 1405 and the corresponding computing hardware 1402 or. 1406 different layers of a neural network, so that the "memory / calculation pair 1401/1402 “Resulting activation of the code and / or data memory 1401 and the computing hardware 1402 as input for the next “memory / calculation pair” 1405/1406 “Of the code and / or data memory 1405 and the computing hardware 1406 is provided to reflect the conceptual organization of a neural network. In at least one embodiment, each of the memory / computation pairs 1401/1402 and 1405/1406 correspond to more than one layer of a neural network. In at least one exemplary embodiment, additional storage / computation pairs (not shown) can follow or in parallel with the storage / computation pairs 1401/1402 and 1405/1406 in inference and / or training logic 1415 be included.

TRAINING AND USE OF NEURONAL NETWORKS

15th shows the training and use of a deep neural network, according to at least one embodiment. In at least one embodiment, the untrained neural network 1506 using a training set 1502 trained. In at least one embodiment, this is the training framework (frame structure) 1504 a PyTorch framework, while in other exemplary embodiments the training framework 1504 is a Tensorflow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j or another training framework. In at least one embodiment, the training framework trains 1504 an untrained neural network 1506 and enables its training using the processing resources described here to create a trained neural network 1508 to create. In at least one embodiment, the weights can be selected randomly or through pre-training using a deep believe network. In In at least one embodiment, the training can be performed either supervised, partially supervised, or unsupervised.

In at least one embodiment, the untrained neural network 1506 trained using supervised learning, the training data set 1502 comprises an input paired with a desired output for an input, or wherein the training data set 1502 comprises an input with a known output and an output of the neural network 1506 is graded manually. In at least one embodiment, the untrained neural network 1506 trained under supervision, processes inputs from the training data set 1502 and compares the resulting outputs to a set of expected or desired outputs. In at least one embodiment, errors are then caused by the untrained neural network 1506 propagated back. In at least one embodiment, the training framework fits 1504 Weights that the untrained neural network 1506 Taxes. In at least one embodiment, the training framework comprises 1504 Tools that can be used to monitor how well the untrained neural network is doing 1506 to a model like the trained neural network 1508 converges which is appropriate based on known input data such as new data 1512 correct answers as in the result 1514 to generate. In at least one embodiment, the training framework trains 1504 the untrained neural network 1506 repeatedly while adjusting the weights to provide an output of the untrained neural network 1506 using a loss function and an adaptation algorithm such as stochastic gradient descent. In at least one embodiment, the training framework trains 1504 the untrained neural network 1506 until the untrained neural network 1506 achieved a desired accuracy. In at least one embodiment, the trained neural network 1508 then used to implement any number of machine learning operations.

In at least one embodiment, the untrained neural network 1506 trained using unsupervised learning, the untrained neural network 1506 tries to train itself using unlabeled data. In at least one embodiment, the data set contains 1502 for unsupervised learning input data without associated output data or ground truth data. In at least one embodiment, the untrained neural network 1506 Groupings within the training data set 1502 learn and determine how individual entries are made with the untrained data set 1502 are related. In at least one exemplary embodiment, a self-organizing map can be generated by unsupervised training, which map is a type of trained neural network 1508 that is capable of performing operations that reduce the dimensionality of new data 1512 are useful. In at least one embodiment, unsupervised training can also be used to perform anomaly detection, which is the identification of data points in a new data set 1512 allows that from the normal patterns of the new record 1512 differ.

In at least one embodiment, semi-supervised learning, a technique in which the data set 1502 includes a mix of marked and unmarked data in training. In at least one embodiment, the training framework 1504 used to perform incremental learning, e.g. through transferred learning techniques. In at least one embodiment, the incremental learning enables the trained neural network 1508 to remember new data 1512 without forgetting the knowledge that entered the network during the initial training.

DATA CENTER

16 shows an example of the data center 1600 , in which at least one embodiment can be applied. In at least one embodiment, the data center comprises 1600 a data center infrastructure layer 1610 , a framework layer 1620 , a software layer 1630 and an application layer 1640 .

In at least one embodiment, as in 16 shown, the data center infrastructure layer 1610 a resource orchestrator 1612 , grouped computing resources 1614 and Node Computing Resources ("CRs") 1616 (1) -1616 (N), where "N" represents any positive whole number. In at least one embodiment, the node CRs 1616 (1) -1616 (N) can include any number of central processing units ("CPUs") or other processors (including accelerators, field programmable gate arrays (FPGAs), Graphics processors, etc.), storage devices (e.g., dynamic read-only memory), storage devices (e.g. Solid state or disk drives), devices for network input / output ("NW I / O"), network switches, virtual machines ("VMs"), power and cooling modules, etc. In at least one exemplary embodiment, one or more node CRs from the node CRs 1616 (1) -1616 (N) be a server with one or more of the computing resources mentioned above.

In at least one embodiment, grouped computing resources 1614 may include separate groupings of node CRs housed in one or more shelves (not shown) or multiple racks housed in data centers at different geographic locations (also not shown). Separate groupings of node CRs within grouped computing resources 1614 can include clustered compute, network, storage, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs including CPUs or processors may be grouped within one or more cabinets to provide computing resources to support one or more workloads. In at least one exemplary embodiment, one or more cabinets can also include any number of power modules, cooling modules and network switches in any combination.

In at least one embodiment, the resource orchestrator 1612 one or more node CRs 1616 (1) -1616 (N) and / or clustered computing resources 1614 configure or otherwise control. In at least one embodiment, the resource orchestrator 1612 a software design infrastructure management unit ("SDI") for the data center 1600 include. In at least one embodiment, the resource orchestrator can include hardware, software, or a combination thereof.

In at least one embodiment, as in 16 shown includes the framework layer 1620 a task planner 1632 , a configuration manager 1634 , a resource manager 1636 and a distributed file system 1638 . In at least one embodiment, the framework layer 1620 a framework to support the software 1632 the software layer 1630 and / or one or more application (s) 1642 the application layer 1640 include. In at least one embodiment, the software 1632 or the application (s) 1642 each include web-based service software or applications such as those provided by Amazon Web Services, Google Cloud and Microsoft Azure. In at least one embodiment, the framework layer 1620 be some kind of free and open source software web application framework such as Apache Spark ™ (hereafter "Spark"), which is the distributed file system 1638 for processing large amounts of data (eg "large amounts of data"), but is not limited to this. In at least one embodiment, the task scheduler may include a Spark driver to facilitate the planning of workloads from different layers of the data center 1600 get supported. In at least one embodiment, the configuration manager 1634 be able to have different layers like the software layer 1630 and the framework layer 1620 including Spark and the distributed file system 1638 to be configured to support the processing of large amounts of data. In at least one embodiment, the resource manager 1636 be able to manage clustered or grouped computing resources belonging to the distributed file system 1638 and the task planner 1632 assigned or assigned for their support. In at least one embodiment, clustered or grouped computing resources may be the grouped computing resource 1614 at the data center infrastructure layer 1610 include. In at least one embodiment, the resource manager 1636 with the resource orchestrator 1612 coordinate to manage these allocated or allocated computing resources.

In at least one embodiment, software 1632 that are in the software layer 1630 is comprised of software comprised of at least portions of the node CRs 1616 (1) -1616 (N), grouped computing resources 1614 and / or the distributed file system 1638 the framework layer 1620 is used. One or more types of software may include Internet webpage browsing software, email virus scanning software, database software, and video streaming software, among others.

In at least one embodiment, the application (s) can 1642 that are in the application layer 1640 is / are comprised of one or more types of applications managed by at least portions of the node CRs 1616 (1) -1616 (N), clustered computing resources 1614 and / or the distributed file system 1638 the framework layer 1620 be used. One or more types of applications can include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.) .) or other machine learning applications used in connection with one or more embodiments.

In at least one embodiment, anyone can be supported by the configuration manager 1634 , Resource manager 1636 and resource orchestrator 1612 Perform any number and type of self-modifying actions based on any amount and type of data collected in any technically appropriate manner. In at least one embodiment, self-modifying actions can allow an operator of the data center 1600 Relieve the burden of making potentially poor configuration decisions and avoiding potentially underutilized and / or underperforming parts of a data center.

In at least one embodiment, the data center 1600 Tools, services, software or other resources to train one or more machine learning models or to predict or derive information using one or more machine learning models according to one or more exemplary embodiments described herein. For example, according to at least one exemplary embodiment, a machine learning model can be created by calculating weighting parameters according to a neural network architecture using the above in relation to the data center 1600 described software and computing resources are trained. In at least one embodiment, trained machine learning models corresponding to one or more neural networks can be used to inference or predict information using the above in relation to the data center 1600 resources described may be used using weight parameters calculated by one or more training techniques described herein.

In at least one exemplary embodiment, a data center can use CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs or other hardware in order to carry out training and / or inferencing using the resources described above. In addition, one or more of the software and / or hardware resources described above can be configured as a service to enable users to train or perform inferencing information, such as image recognition, speech recognition or other services with artificial intelligence.

Inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more exemplary embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in the system 16 for inference or prediction of operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and / or architectures of neural networks, or use cases of neural networks described herein.

At least one embodiment can be carried out using the techniques described above. For example, a generation network or an evaluation network as in 14-16 described.

AUTONOMOUS VEHICLE

17A shows an example of an autonomous vehicle 1700 , according to at least one embodiment. In at least one embodiment, the autonomous vehicle 1700 (here alternatively referred to as “vehicle 1700”) can be a passenger vehicle without restriction, such as a car, a truck, a bus and / or another type of vehicle that takes one or more passengers. In at least one embodiment, the vehicle can 1700 be a semi-trailer truck used to transport goods. In at least one embodiment, vehicle can 1700 be an airplane, robotic vehicle, or other type of vehicle.

Autonomous vehicles can be described in terms of the level of automation provided by the National Highway Traffic Safety Administration (“NHTSA”), a division of the US Department of Transportation, and the Society of Automotive Engineers (“SAE”) “taxonomy and definitions of terms in the Connection with drive automation systems for road vehicles "is defined (e.g. Standard No. J3016-201806, published June 15, 2018, Standard No. J3016-201609, published September 30, 2016 , as well as earlier and future versions of this standard). In one or more exemplary embodiments, the vehicle can 1700 a functionality according to one or more of the levels 1 - 5 of the autonomous driving steps. For example, vehicle can 1700 in at least one embodiment, depending on the embodiment conditional automation (level 3 ), a high level of automation (level 4th ) and / or full automation (level 5 ) exhibit.

In at least one embodiment, the vehicle can 1700 include, without limitation, components such as the chassis, vehicle body, wheels (e.g. 2, 4, 6, 8, 18, etc.), tires, axles and other components of a vehicle. In at least one embodiment, vehicle can 1700 a drive system without restriction 1750 include, such as an internal combustion engine, a hybrid electric power plant, an all-electric motor and / or some other type of propulsion system. In at least one embodiment, the drive system 1750 with a drive train of the vehicle 1700 be connected, which may include a transmission to drive the vehicle 1700 to enable. In at least one embodiment, the drive system 1750 in response to receiving signals from one or more throttle / accelerators 1752 being controlled.

In at least one embodiment, a steering system 1754 , which may include a steering wheel, among other things, used to drive a vehicle 1700 to steer (e.g. along a desired path or route) if a propulsion system 1750 is in operation (e.g. when the vehicle is in motion). In at least one embodiment, a steering system 1754 Signals from steering actuator (s) 1756 receive. The steering wheel can be used for the full functionality of the automation (level 5 ) be optional. In at least one embodiment, a brake sensor system 1746 used to apply vehicle brakes in response to receiving signals from brake actuator (s) 1748 and / or actuate brake sensors.

In at least one embodiment, the controller (s) 1736 who, among other things, have one or more system-on-chips (“SoCs”) (not in 17A shown) and / or graphics processing unit (s) (“GPU (s)”) may include signals (eg representative of commands) to one or more components and / or systems of the vehicle 1700 ready. In at least one embodiment example, controller (s) 1736 Signals for actuating the vehicle brakes via brake actuators 1748 , to operate the steering system 1754 via steering actuator (s) 1756 , to operate the drive system 1750 via throttle / accelerator 1752 send. Control (s) 1736 may include one or more on-board (e.g. integrated) computing devices (e.g. supercomputers) that process sensor signals and issue operating commands (e.g. signals that represent commands) to enable autonomous driving and / or a human driver when driving the vehicle 1700 to support. In at least one embodiment, control (s) 1736 a first control 1736 for autonomous driving functions, a second control 1736 for functional safety functions, a third controller 1736 for functionality of artificial intelligence (e.g. computer vision), a fourth control 1736 for infotainment functionality, a fifth control 1736 for redundancy in emergencies and / or other controls. In at least one embodiment, a single controller 1736 handle two or more of the above functionalities, two or more controls 1736 can handle a single functionality and / or any combination thereof.

In at least one embodiment, control (s) 1736 in response to sensor data received from one or more sensors (eg sensor inputs), signals for controlling one or more components and / or systems of the vehicle 1700 ready. In at least one embodiment, sensor data can be received, for example and without limitation, from the sensor (s) of a global navigation satellite system (“GNSS”) 1758 (e.g. sensor (s) of the global positioning system, RADAR sensor (s) ) 1760 , Ultrasonic sensor (s) 1762 , LIDAR sensor (s) 1764 , Sensor (s) of an inertial measurement unit ("IMU") 1766 (e.g. accelerometer, gyroscope (s), magnetic compass (s), magnetometer etc.), microphone (s) 1796 , Stereo camera (s) 1768, wide-angle camera (s) 1770 (e.g. fisheye cameras), infrared camera (s) 1772 , Surround camera (s) 1774 (e.g. 360 degree cameras), remote camera (s) (not in 17A shown), mid-range camera (s) (not in 17A shown), speed sensor (s) 1744 (e.g. to measure the speed of the vehicle 1700 ), Vibration sensor (s) 1742 , Steering sensor (s) 1740 , Brake sensor (s) (e.g. as part of the brake sensor system 1746 ) and / or other types of sensors.

In at least one embodiment, one or more control (s) 1736 Inputs (e.g. represented by input data) from an instrument cluster 1732 of the vehicle 1700 receive and output (e.g. represented by output data, display data, etc.) via a man-machine interface (“HMI”) display 1734, an acoustic signal generator, a loudspeaker and / or via other components of the vehicle 1700 provide. In at least one embodiment, the outputs can include information such as vehicle speed, speed, time, map data (e.g. a high-resolution map (not in 17A shown), location data (e.g. the position of the vehicle 1700 like on a card), Direction, position of other vehicles (e.g. an occupancy grid), information about objects and the status of objects as provided by control (s) 1736 perceived, etc. include. For example, the HMI display 1734 In at least one embodiment, display information about the presence of one or more objects (e.g. a street sign, a warning sign, a traffic light switch, etc.) and / or information about driving maneuvers that the vehicle has performed, is currently performing or is about to perform (e.g. lane change now, exit 34B in two miles, etc.).

In at least one exemplary embodiment, the vehicle comprises 1700 further a network interface 1724 , the wireless antenna 1726 and / or modem (s) to communicate over one or more networks. In at least one embodiment, the network interface 1724 for example communication via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communication (“GSM”), IMT-CDMA Enable multi-carrier ("CDMA2000") etc. In at least one embodiment, wireless antenna (s) 1726 also communication between objects in the vicinity (e.g. vehicles, mobile devices etc.) using local networks such as Bluetooth, Bluetooth Low Energy ("LE"), Z-Wave, ZigBee etc. and / or broadband networks with low power (" LPWANs ") such as LoRaWAN, SigFox etc. enable.

Inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in the system 17A for inference or prediction of operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

At least one embodiment can be carried out using the techniques described above. In at least one embodiment, the robot can be an autonomous vehicle, and the evaluation and generation of networks can be implemented in the autonomous vehicle using a computer system.

17B shows an example with camera positions and fields of view for the autonomous vehicle 1700 the 17A according to at least one embodiment. In at least one exemplary embodiment, the camera positions and the respective fields of view are an exemplary representation and should not be interpreted in a restrictive manner. For example, additional and / or alternative cameras can be included in at least one exemplary embodiment and / or the cameras can be at different positions on the vehicle 1700 are located.

In at least one embodiment, camera types for cameras may include, but are not limited to, digital cameras that are designed for use with vehicle components and / or systems 1700 are trained. The camera (s) can be operated on Safety Integrity Level ("ASIL") B and / or on another ASIL level. In at least one embodiment, the camera types can achieve any desired image recording rate, such as 60 images per second (fps), 1220 fps, 240 fps, etc., depending on the embodiment. In at least one embodiment, cameras can be deployed using scrolling bezels, global bezels, a different type of bezel, or a combination thereof. In at least one embodiment, a color filter array can be a red-clear-clear-clear color filter array ("RCCC"), a red-clear-clear-clear-blue color filter array ("RCCB"), a red Blue-Green-Clear Color Filter Array ("RBGC"), a Foveon X3 Color Filter Array, a Bayer Sensor Color Filter Array ("RGGB"), a Monochrome Sensor Color Filter Array, and / or another Include type of color filter array. In at least one embodiment, cameras with clear pixels, such as cameras with an RCCC, an RCCB, and / or an RBGC color filter array, can be used to increase the sensitivity to light.

In at least one exemplary embodiment, one or more cameras can be used to carry out advanced functions of advanced driver assistance systems (“ADAS”) (for example as part of a redundant or fail-safe design). For example, a multifunction mono camera can be installed in at least one embodiment, which includes functions such as lane departure warning, traffic sign assistant and intelligent headlight control. In at least one exemplary embodiment, one or more cameras (for example all cameras) can simultaneously record and provide image data (for example video).

In at least one embodiment, one or more cameras can be mounted in a bracket, e.g. B. in a custom (three-dimensional ("3D") printed) assembly to eliminate stray light and reflections from inside the vehicle (e.g., dashboard reflections reflected in windshield mirrors) that could interfere with the camera's image data acquisition capabilities. With respect to side mirror mounting assemblies, in at least one embodiment, side mirror assemblies can be custom 3D printed so that the camera mounting plate conforms to the shape of the side mirror. In at least one exemplary embodiment, the camera (s) can be integrated into the side mirror. In the case of side view cameras, the cameras can also be integrated in four columns at each corner of the cabin, according to at least one exemplary embodiment.

In at least one embodiment, cameras can have a field of view that covers parts of the area in front of the vehicle 1700 includes (e.g. front-facing cameras), can be used for a 360 ° view, to identify forward-facing paths and obstacles, and to use one or more controls 1736 and / or to provide control SoCs with information that is decisive for the creation of an occupancy grid and / or the determination of preferred vehicle paths. In at least one embodiment, front-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including, without limitation, emergency braking, pedestrian detection, and collision avoidance. In at least one embodiment, forward-facing cameras can also be used for ADAS functions and systems, including, but not limited to, Lane Departure Warnings (“LDW”), autonomous cruise control (“ACC “) And / or other functions such as traffic sign recognition.

In at least one embodiment, a plurality of cameras may be used in a front-facing configuration, including, for example, a monocular camera platform that includes a Complementary Metal Oxide Semiconductor (CMOS) color image sensor. In at least one embodiment, a wide angle camera 1770 can be used to perceive objects that can be seen from the periphery (e.g. pedestrians, crossing traffic or bicycles). Though just a wide-angle camera 1770 in 17B As shown, in other embodiments there may be any number (including zero) wide angle cameras 1770 on the vehicle 1700 give. In at least one embodiment, any number of remote camera (s) 1798 (eg a pair of remote stereo cameras) can be used for depth-based object recognition, in particular for objects for which a neural network has not yet been trained. In at least one embodiment, remote cameras 1798 can also be used for object detection and classification as well as basic object tracking.

In at least one embodiment, any number of stereo camera (s) 1768 can also be included in a front-facing configuration. In at least one embodiment, one or more of the stereo camera (s) 1768 may include an integrated control unit that includes a scalable processing unit that includes programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) - or Can provide an Ethernet interface on a single chip. In at least one exemplary embodiment, such a unit can be used to create a 3D map of the surroundings of the vehicle 1700 including a distance estimate for all pixels. In at least one embodiment, one or more stereo cameras 1768 may include, among other things, one or more compact stereo vision sensor (s) that may include, among other things, two camera lenses (one left and one right) and an image processing chip that the distance from the vehicle 1700 measure to the target object and be able to use the information generated (e.g. metadata) to activate autonomous emergency braking and lane departure warning functions. In at least one embodiment, other types of stereo camera (s) 1768 may be used in addition to or as an alternative to those described herein.

In at least one embodiment, cameras can have a field of view that covers parts of the area to the side of the vehicle 1700 includes (e.g. side-view cameras), for the all-round view, information is provided that is used to create and update the occupancy grid and to generate warnings of side impact collisions. For example, in at least one embodiment, surround camera (s) 1774 (e.g., four surround cameras 1774 as in 17B shown) on vehicle 1700 be positioned. The surround camera (s) 1774 can, without limitation, have any number and combination of wide-angle camera (s) 1770 , Fisheye camera (s), 360 degree camera (s) and / or the like. For example, in at least one embodiment, there can be four fisheye cameras on the front, rear and sides of the vehicle 1700 be arranged. In at least one embodiment, the vehicle can 1700 three surround camera (s) 1774 (e.g. left, right and back) and use one or more other camera (s) (e.g. a front-facing camera) as the fourth surround camera.

In at least one embodiment, cameras can have a field of view that covers parts of the area behind the vehicle 1700 includes (e.g. reversing cameras), used for parking assistance, all-round visibility, warning of rear-end collisions and the creation and updating of the occupancy grid. In at least one embodiment, a variety of cameras may be used, including, but not limited to, cameras that are also suitable as the front-facing camera (s) (eg, remote cameras) 1798 and / or medium range camera (s) 1776, stereo camera (s) 1768), infrared camera (s) 1772 etc.) as described here.

Inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on inference and / or training logic 1415 are here in connection with 14A and / or 14B. In at least one embodiment, the inference and / or training logic 1415 in the system 17B for inference or prediction of operations based, at least in part, on weight parameters that are calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

At least one embodiment can be carried out using the techniques described above. In at least one embodiment, the robot can be an autonomous vehicle and the evaluation and generation networks can be implemented in the autonomous vehicle using a computer system. In at least one exemplary embodiment, the depth camera that is used to capture the point cloud of an object to be gripped is a camera, sonar, radar or lidar in an autonomous vehicle.

17C Figure 13 is a block diagram illustrating an exemplary system architecture for the autonomous vehicle 1700 the 17A shows, according to at least one embodiment. In at least one embodiment, all of the components, features, and systems of the vehicle 1700 in 17C than via a bus 1702 shown connected. In at least one embodiment, the bus 1702 without restriction include a CAN data interface (here alternatively referred to as "CAN bus"). In at least one embodiment, a CAN can be a network within the vehicle 1700 be that in support of the control of various features and functions of vehicle 1700 is used, such as brake application, acceleration, braking, steering, windshield wipers, etc. In at least one embodiment, the bus can 1702 be configured so that it has dozens or even hundreds of nodes, each of which has its own unique identifier (e.g. a CAN ID). In at least one embodiment, Bus 1702 can be read to determine steering wheel angle, vehicle speed, engine speed per minute ("RPMs"), button positions, and / or other vehicle health indicators. In at least one embodiment, Bus 1702 be a CAN bus that is ASIL B compliant.

In at least one exemplary embodiment, in addition to or as an alternative to CAN, FlexRay and / or Ethernet can be used. In at least one embodiment, there can be any number of buses 1702 which may include, without limitation, zero or more CAN buses, zero or more FlexRay buses, zero or more Ethernet buses, and / or zero or more other types of buses using a different protocol. In at least one embodiment, two or more buses 1702 can be used to perform various functions and / or for redundancy. For example, a first bus 1702 for the functionality of collision avoidance and a second bus 1702 used to control the actuation. In at least one embodiment, each bus can 1702 with any component of the vehicle 1700 communicate, and two or more buses 1702 can communicate with the same components. In at least one embodiment, any of any number of systems on chip (s) ("SoC (s)") 1704 , each control (s) 1736 and / or every computer in the vehicle has access to the same input data (e.g. inputs from sensors in the vehicle 1700 ) and be connected to a common bus, a CAN bus.

In at least one embodiment, the vehicle can 1700 one or more controls 1736 include, like the ones here regarding 17A described. Control (s) 1736 can be used for a variety of functions. In at least one embodiment, control (s) 1736 with any other components and systems of the vehicle 1700 coupled and for controlling vehicle 1700 , artificial intelligence of vehicle 1700 , Infotainment for vehicle 1700 and / or similar functions are used.

In at least one embodiment, the vehicle can 1700 any number of SoCs 1704 include. Each of the SoCs 1704 can, without restriction, central units ("CPU (s)") 1706, graphics processing units ("GPU (s)") 1708 , Processor (s) 1710 , Cache (s) 1712, accelerator 1714, data storage 1716 and / or other components and features not shown. In at least one embodiment, SoC (s) can 1704 for controlling the vehicle 1700 be used in a variety of platforms and systems. For example, SoC (s) 1704 in at least one embodiment in a system (e.g. system of the vehicle 1700 ) with a high definition ("HD") map 1722 combined, the map updates and / or updates via the network interface 1724 from one or more servers (in 17C not shown).

In at least one embodiment, the CPU (s) 1706 may comprise a CPU cluster or CPU complex (alternatively referred to herein as “CCPLEX”). In at least one embodiment, the CPU (s) 1706 may include multiple cores and / or level 2 caches (“L2”). For example, in at least one embodiment, the CPU (s) 1706 may include eight cores in a coherent multiprocessor configuration. In at least one embodiment, the CPU (s) 1706 can include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2MB L2 cache). In at least one embodiment, the CPU (s) 1706 (e.g., CCPLEX) can be configured to support simultaneous clustering so that any combination of clusters of the CPU (s) 1706 can be active at any point in time.

In at least one exemplary embodiment, one or more of the CPU (s) 1706 can implement power management functions that include, without limitation, one or more of the following features: Individual hardware blocks can be automatically decoupled from the clock signal in a gate when idle (clock-controlled). clock-gated ”) to save energy dynamically; Each core cycle can be decoupled if the core is not actively executing commands due to the execution of Wait-For-Interrupt (“WFI”) / Wait-For-Event (“WFE”) instructions; each core can be independently power gated; each core cluster can be independently clock controlled if all cores are clock controlled or power controlled; and / or each core cluster can be independently power driven if all cores are power driven. In at least one embodiment, the CPU (s) 1706 may further implement an improved performance state management algorithm, specifying the allowable performance states and expected wake-up times, and the hardware / microcode determining the best performance state to enter the core, cluster, and CCPLEX can. In at least one embodiment, processing cores can support simplified sequences for entering performance states in software, with the work shifting to the microcode.

In at least one embodiment, GPU (s) can 1708 an integrated GPU (here alternatively referred to as "iGPU"). In at least one embodiment, GPU (s) can 1708 programmable and efficient for parallel workloads. In at least one embodiment, GPU (s) can 1708 use an improved tensor command set. In at least one embodiment, GPU (s) can 1708 one or more streaming microprocessors, each streaming microprocessor having an L1 cache (e.g. an L1 cache with at least 96 KB storage capacity) and two or more streaming microprocessors an L2 cache (e.g. an L2 cache with 512 KB storage capacity ) can use together. In at least one embodiment, GPU (s) can 1708 Include at least eight streaming microprocessors. In at least one embodiment, GPU (s) can 1708 Use Compute Application Programming Interface (s) (API (s)). In at least one embodiment, GPU (s) can 1708 use one or more parallel computer platforms and / or programming models (e.g. CUDA from NVIDIA).

In at least one embodiment, one or more of the GPU (s) 1708 be performance optimized for the best performance in automotive and embedded use cases. In one embodiment, GPU (s) could 1708 for example on a fin field effect transistor (“FinFET”). In at least one embodiment, each streaming microprocessor may include a number of mixed precision processing cores that are divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores could be divided into four processing blocks. In at least one embodiment, each processing block could have 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two NVIDIA TENSOR-CORES for deep learning matrix arithmetic with mixed precision, a level zero ("L0") instruction cache , a warp planner, a shipping unit, and / or a 64K register file. In In at least one embodiment, streaming microprocessors may include independent parallel integer and floating point data paths to enable efficient execution of workloads with a mixture of computational and addressing computations. In at least one embodiment, streaming microprocessors can include independent thread scheduling functions to enable finer-grain synchronization and collaboration between parallel threads. In at least one embodiment, streaming microprocessors can include a combined LI data cache and shared memory device to improve performance while simplifying programming.

In at least one embodiment, one or more of the GPU (s) 1708 high bandwidth storage (“HBM”) and / or a 16 GB HBM2 storage subsystem to provide, in some examples, a peak storage bandwidth of approximately 900 GB / second. In at least one embodiment, a synchronous graphics random access memory (“SGRAM”) can be used in addition to or as an alternative to the HBM memory, for example a synchronous graphics random access memory with double data rate of type 5 (“GDDR5”).

In at least one embodiment, GPU (s) can 1708 include a unified storage technology. In at least one embodiment, Address Translation Services ("ATS") support can be used to provide GPU (s) 1708 to enable direct access to CPU (s) 1706 page tables. In at least one embodiment, an address translation request can be transmitted to the CPU (s) 1706 when the memory management unit ("MMU") GPU (s) 1708 experiences a bad access. In response to this, the CPU (s) 1706 can search in their page tables for a virtual-to-physical assignment of the address and send the translation back to the GPU (s) 1708 transmitted, according to at least one embodiment. In at least one embodiment, the unified memory technology can use a single, uniform virtual address space for the memory of both the CPU (s) 1706 and the GPU (s) 1708 enable programming of the GPU (s) 1708 and porting applications to the GPU (s) 1708 is simplified.

In at least one embodiment, the GPU (s) can 1708 include any number of access counters showing the frequency of access by GPU (s) 1708 can track the memory of other processors. In at least one embodiment, access counters can help ensure that pages of memory are moved into physical memory of the processor that accesses pages most often, thereby improving the efficiency of memory areas shared by processors.

In at least one embodiment, one or more of the SoC (s) 1704 include any number of cache (s) 1712, including those described herein. For example, in at least one embodiment, the cache (s) 1712 could include a level three cache that serves both the CPU (s) 1706 and the GPU (s) (s) 1708 is available (i.e. the CPU (s) 1706 as well as the GPU (s) 1708 is / are connected). In at least one embodiment, cache (s) 1712 may include a write-back cache that can track states of lines, e.g., using a cache coherency protocol (e.g., MEI, MESI, MSI, etc.). In at least one embodiment, the L3 cache can be 4 MB or more, depending on the embodiment, although smaller cache sizes can be used.

In at least one embodiment, one or more of the SoC (s) 1704 one or more accelerators 1714 include (e.g., hardware accelerators, software accelerators, or a combination thereof). In at least one embodiment, SoC (s) can 1704 comprise a hardware acceleration cluster that may contain optimized hardware accelerators and / or large on-chip memory. In at least one embodiment, a large on-chip memory (eg 4 MB SRAM) can enable a hardware acceleration cluster to accelerate neural networks and other computations. In at least one embodiment, a hardware acceleration cluster using GPU (s) 1708 used to control the graphics processor (s) 1708 to supplement and the graphics processor (s) 1708 to relieve some tasks (e.g. to have more cycles of the graphics processor (s) 1708 for the execution of other tasks). In at least one embodiment, the accelerator (s) could 1714 be used for targeted workloads (e.g. perception, convolutional neural networks ("CNNs"), recurrent neural networks ("RNNs"), etc.) that are stable enough to be amenable to acceleration. In at least one embodiment, a CNN may include region-based or regional convolutional neural networks (“RCNNs”) and fast RCNNs (eg, such as those used for object recognition) or other types of CNNs.

In at least one embodiment, the accelerator (s) can 1714 (e.g. hardware acceleration clusters) comprise one or more Deep Learning Accelerator (s) ("DLA"). DLA (s) can, include, without limitation, one or more Tensor Processing Units ("TPUs") that may be configured to provide an additional ten trillion operations per second for deep learning applications and inferencing. In at least one embodiment, TPUs can be accelerators that are configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). DLA (s) can be further optimized for a particular set of neural network types and floating point operations, as well as for inferencing. In at least one embodiment, the design of DLA (s) can provide more performance per millimeter than a typical general purpose GPU and typically far outperforms a CPU. In at least one embodiment, TPU (s) can perform multiple functions, including a single instance convolution function that supports, for example, INT8, INT16, and FP16 data types for both features and weights and post-processor functions. In at least one embodiment, DLA (s) can quickly and efficiently execute neural networks, particularly CNNs, on processed or unprocessed data for any variety of functions, including, for example and without limitation: a CNN for object identification and recognition using data from Camera sensors; a CNN for estimating distances using data from camera sensors; a CNN for emergency vehicle detection and identification and recognition using data from microphones 1796 ; a CNN for face recognition and vehicle owner identification using data from camera sensors; and / or a CNN for security-related events.

In at least one embodiment, DLA (s) can perform any function of GPU (s) 1708 and using an inference accelerator, for example, a designer can use either DLA (s) or GPU (s) 1708 use for any function. For example, in at least one embodiment, the designer can focus the processing of CNNs and floating point operations on DLA (s) and other functions GPU (s) 1708 and / or other accelerator (s) 1714.

In at least one embodiment, the accelerator (s) can 1714 (eg hardware acceleration clusters) comprise one / more programmable vision accelerator (s) ("PVA"), which / which can alternatively be referred to here as computer vision accelerator (s). In at least one embodiment, the PVA (s) can be designed and configured to use computer vision algorithms for Advanced Driver Assistance System (“ADAS”) 1738, autonomous driving, augmented reality (“AR”) ) Applications and / or virtual reality ("VR") applications accelerated. PVA (s) can provide a balance between performance and flexibility. In at least one embodiment, each PVA can include, for example and without limitation, any number of reduced instruction set computing cores (“RISC”), direct memory access (“DMA”), and / or any number of vector processors.

In at least one embodiment, RISC cores can interact with image sensors (e.g., image sensors from any of the cameras described herein), image signal processor (s), and / or the like. In at least one embodiment, each RISC core can include any memory location. In at least one exemplary embodiment, RISC cores can use one of several protocols, depending on the exemplary embodiment. In at least one embodiment, RISC cores can execute a real-time operating system (“RTOS”). In at least one embodiment, RISC cores can be implemented using one or more integrated circuit devices, application specific integrated circuits (“ASICs”), and / or memory devices. In at least one embodiment, RISC cores could include an instruction cache and / or a tightly coupled RAM, for example.

In at least one embodiment, DMA can enable components of PVA (s) to access system memory independently of CPU (s) 1706. In at least one embodiment, DMA can support any number of functions provided to optimize PVA, including, but not limited to, support for multi-dimensional addressing and / or circular addressing. In at least one embodiment, DMA can support up to six or more addressing dimensions, which can include block width, block height, block depth, horizontal block grading, vertical block grading, and / or depth grading, among others.

In at least one embodiment, vector processors can be programmable processors that can be configured to efficiently and flexibly program algorithms for image processing and to provide signal processing capabilities. In at least one embodiment, PVA can include a PVA core and two vector processing subsystem partitions. In at least one embodiment, the PVA core can include a processor subsystem, DMA machine (s) (eg, two DMA machines), and / or other peripheral devices. In In at least one embodiment, the vector processing subsystem may operate as a primary PVA processing unit and may include a vector processing unit (“VPU”), an instruction cache, and / or a vector memory (eg, “VMEM”). In at least one embodiment, the VPU core can include a digital signal processor, such as a single instruction, multiple data (“SIMD”), very long instruction word (“VLIW”) digital signal processor. In at least one embodiment, a combination of SIMD and VLIW can improve throughput and speed.

In at least one embodiment, each of the vector processors may include an instruction cache and be coupled to dedicated memory. As a result, in at least one embodiment, each vector processor can be configured to operate independently of other vector processors. In at least one embodiment, vector processors included in a particular PVA can be configured to use data parallelism. For example, in at least one embodiment, multiple vector processors included in a single PVA may execute the same algorithm for computer imaging, but in different areas of an image. In at least one embodiment, vector processors included in a particular PVA may concurrently execute different algorithms for computer vision on the same image, or even execute different algorithms on sequential images or portions of an image. In at least one embodiment, any number of PVAs may be included in a hardware acceleration cluster and any number of vector processors may be included in each of the PVAs, among other things. In at least one embodiment, PVA (s) may include additional error correction code (ECC) memory to improve overall system security.

In at least one embodiment, the accelerator (s) can 1714 (e.g. hardware acceleration cluster) include a computer vision network on a chip and static random-access memory ("SRAM") to provide high bandwidth, low latency SRAM for the accelerator (s) 1714 provide. In at least one embodiment, the on-chip memory can include at least 4 MB SRAM, which for example and without restriction consists of eight freely configurable (field-configurable) memory blocks that can be accessed by both PVA and DLA. In at least one exemplary embodiment, each pair of memory blocks can comprise an Advanced Peripheral Bus (“APB”) interface, a circuit arrangement for configuration, a controller and a multiplexer. In at least one embodiment, any type of memory can be used. In at least one embodiment, PVA and DLA can access memory through a framework that enables PVA and DLA to access memory at high speed. In at least one embodiment, the backbone may include a computer vision network on-chip that connects PVA and DLA to memory (eg, using APB).

In at least one embodiment, an on-chip computer vision network may include an interface that determines, prior to the transmission of control signals / addresses / data, that both PVA and DLA are delivering finished and valid signals. In at least one embodiment, an interface can provide separate phases and separate channels for the transmission of control signals / addresses / data as well as burst communications for the continuous data transmission. In at least one embodiment, an interface may conform to International Standardization Organization (“ISO”) 26262 or International Electrotechnical Commission (“IEC”) 61508 standards, although other standards and protocols may be used.

In at least one embodiment, one or more of the SoC (s) 1704 a hardware accelerator for real-time ray tracing. In at least one embodiment, a hardware accelerator can be used for fast and efficient determination of positions and dimensions of objects (e.g. within a model of the real world), for generating real-time visualization simulations, for interpreting RADAR signals, for synthesis and / or analysis sound propagation, for simulating SONAR systems, for general wave propagation simulation, for comparison with LIDAR data for the purpose of localization and / or other functions and / or for other purposes.

In at least one embodiment, the accelerator (s) have 1714 (e.g. hardware accelerator cluster) a broad field of application for autonomous driving. In at least one embodiment, PVA can be a programmable vision accelerator that can be used for important processing stages in ADAS and autonomous vehicles. In at least one embodiment, the capabilities of PVA are well suited to algorithmic domains that require predictable processing at low power, low latency. In other words, the PVA is at half or dense Regular calculations can also be used for small data sets that require predictable runtimes with low latency and low performance. In at least one embodiment, PVAs are for autonomous vehicles, such as vehicle 1700 , designed in such a way that they can execute classical algorithms for computer-aided vision, as they are efficient at object recognition and work with integer mathematics.

For example, in accordance with at least one embodiment of the technology, PVA is used to perform computer stereo vision. In at least one embodiment, a semi-global matching based algorithm may be used in some examples, although this is not intended to be a limitation. In at least one embodiment, applications for level 3-5 autonomous driving use on-the-fly motion estimation / stereo matching (e.g. structure of motion, pedestrian detection, lane detection, etc.). In at least one embodiment, the PVA can perform a computer stereo vision function upon inputs from two monocular cameras.

In at least one embodiment, PVA can be used to perform dense optical flow. For example, in at least one embodiment, PVA could process raw RADAR data (e.g., using a 4D Fast Fourier Transform) to provide processed RADAR data. In at least one embodiment, PVA is used to process a runtime depth, e.g., by processing raw runtime data to provide processed runtime data.

In at least one exemplary embodiment, DLA can be used to operate any type of network in order to improve control and driving safety, for example and without limitation a neural network that outputs a degree of confidence for each object recognition. In at least one embodiment, confidence can be represented or interpreted as probability or as providing a relative “weight” of each detection compared to other detections. In at least one embodiment, the confidence enables a system to make further decisions about which discoveries should be considered true positive discoveries rather than false positive discoveries. For example, in at least one embodiment, a system may set a threshold for confidence and consider only those detections as true positive detections that exceed the threshold. In an exemplary embodiment in which an automatic emergency braking system (“AEB”) is used, false-positive detections would cause the vehicle to automatically perform emergency braking, which is obviously undesirable. In at least one embodiment, very reliable detections can be considered as triggers for AEB. In at least one embodiment, DLA can operate a neural network for regression of the confidence value. In at least one embodiment, the neural network can use at least a subset of parameters as input, such as the dimensions of the bounding box, the estimate of the floor area obtained (e.g. from another subsystem), the output from IMU sensor (s) 1766 that with the orientation of the vehicle 1700 correlates the distance, 3D position estimates of the object, which are provided by the neural network and / or other sensors (e.g. LIDAR sensor (s) 1764 or RADAR sensor (s) 1760 ) and others.

In at least one embodiment, one or more of the SoC (s) 1704 the data store (s) 1716 (e.g. memory). In at least one embodiment, the data store (s) can 1716 an on-chip memory from SoC (s) 1704 that can store neural networks that run on GPU (s) 1708 and / or DLA are to be executed. In at least one embodiment, the data store (s) can 1716 be large enough to store multiple instances of neural networks for redundancy and security. In at least one embodiment, the data store (s) can 1712 Include L2 or L3 cache (s).

In at least one embodiment, one or more of the SoC (s) 1704 any number of processor (s) 1710 (e.g. embedded processors) include. The processor (s) 1710 may include a boot and power management processor, which may be a dedicated processor and subsystem that provides boot performance and management functions and related security enforcement. In at least one embodiment, the boot and power management processor can be part of the boot sequence of the SoC (s) 1704 and provide power management services at run time. In at least one embodiment, the boot performance and management processor can program clock and voltage, support for state transitions in the system with lower power consumption, management of the thermal and temperature sensors of the SoC (s) 1704 and or Management of the power status of the SoC (s) 1704 provide. In at least one embodiment, each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to temperature, and SoC (s) 1704 can use ring oscillators to measure temperatures from CPU (s) 1706, GPU (s) 1708 and / or accelerator (s) 1714. In at least one embodiment, if the temperatures are found to exceed a threshold, then the boot and power management processor may enter a temperature fault routine and the SoC (s) 1704 put into a state of lower power and / or the vehicle 1700 Put the vehicle in a safe stop mode (e.g. the vehicle 1700 put into a safe stop mode).

In at least one embodiment, the processor (s) can 1710 further comprise a set of embedded processors that can serve as an audio processing engine. In at least one embodiment, an audio processing engine can be an audio subsystem that enables full hardware support for multi-channel audio over multiple interfaces and a wide and flexible range of audio I / O interfaces. In at least one embodiment, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

In at least one embodiment, the processor (s) can 1710 further include an always-on processor engine that can provide the necessary hardware features to support the management of low-power sensors and wake-up use cases. In at least one embodiment, the always-on processor engine may include, without limitation, a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controls), various I / O control peripherals, and routing logic.

In at least one embodiment, the processor (s) can 1710 further include a security cluster engine that includes, without limitation, a dedicated processor subsystem for handling security management for automotive applications. In at least one embodiment, the security cluster engine may include, without limitation, two or more processor cores, tightly coupled RAM memory, supporting peripheral devices (eg, timers, interrupt controls, etc.), and / or routing logic. In a security mode, in at least one embodiment, two or more cores can operate in a lockstep mode and act as a single core with comparison logic to detect differences between their operations. In at least one embodiment, the processor (s) can 1710 further comprise a real-time camera engine which, among other things, can comprise a dedicated processor subsystem for handling the real-time camera management. In at least one embodiment, the processor (s) can 1710 further comprise a high dynamic range signal processor which may include, among other things, an image signal processor which is a hardware engine that is part of the camera processing pipeline.

In at least one embodiment, the processor (s) can 1710 a video image composer, which may be a processing block (e.g. implemented on a microprocessor) that implements video post-processing functions required by a video playback application to generate the final image for the display window. In at least one embodiment, the video composer can correct the lens distortion on the wide-angle camera (s) 1770 , the surround camera (s) 1774 and / or the sensor (s) of the surveillance camera (s) in the cabin. In accordance with at least one exemplary embodiment, in-cabin surveillance camera sensors are preferably monitored by a neural network that is located on another instance of SoC 1704 is running and configured to identify and respond appropriately to events in the cabin. In at least one embodiment, a system in the cabin can perform lip reading without limitation to activate cellular service and make a phone call, dictate emails, change the vehicle's destination, activate or deactivate the vehicle's infotainment system and settings change or provide voice-activated web surfing. In at least one exemplary embodiment, certain functions are available to the driver when the vehicle is operated in an autonomous mode and are otherwise deactivated.

In at least one embodiment, the video compositor may include enhanced temporal noise suppression for both spatial and temporal noise suppression. For example, in at least one embodiment where motion occurs in video, the noise suppression may weight spatial information accordingly, thereby reducing the weight of the information provided by adjacent frames. In at least one embodiment in which an image or part of an image does not include motion, the one performed by the video image composer may Temporal noise reduction Use information from the previous image to reduce the noise in the current image.

In at least one exemplary embodiment, the video image composer can also be configured in such a way that it carries out stereo rectification on the individual images of the input stereo lenses. In at least one embodiment, the video compositor can continue to be used for composing user interfaces when the desktop of the operating system is used and the GPU (s) 1708 is / are not required to continuously render new surfaces. In at least one embodiment, the video image composer can use the video image composer to relieve the graphics processor (s) 1708 used when the graphics processor (s) 1708 is turned on and actively performing 3D rendering to improve performance and responsiveness.

In at least one embodiment, one or more SoC (s) 1704 further comprise a serial MIPI interface (Mobile Industry Processor Interface) for cameras for receiving video and inputs from cameras, a high-speed interface and / or a video input block that can be used for camera and associated pixel input functions. In at least one embodiment, one or more SoC (s) 1704 further comprise an input / output controller (s) which can be controlled by software and used to receive I / O signals which are not intended for a particular role.

In at least one embodiment, one or more of the SoC (s) 1704 further include a wide variety of peripheral interfaces to enable communication with peripheral devices, audio encoders / decoders ("codecs"), power management, and / or other devices. SoC (s) 1704 can / can be used to receive data from cameras (e.g. connected via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g. LIDAR sensor (s) 1764 , RADAR sensor (s) 1760 etc., which can be connected via Ethernet), data from bus 1702 (e.g. speed of the vehicle 1700 , Steering wheel position, etc.), data from GNSS sensor (s) 1758 (e.g. connected via Ethernet or CAN bus) etc. to be processed. In at least one embodiment, one or more SoC (s) 1704 further include high performance dedicated mass storage controllers which may contain their own DMA engines and which may be used to offload routine data management tasks from the CPU (s) 1706.

In at least one embodiment, SoC (s) can 1704 be an end-to-end platform with a flexible architecture that includes automation levels 3-5 and thus provides a comprehensive architecture of functional safety that uses and efficiently uses computer vision and ADAS techniques for diversity and redundancy, a platform for provides a flexible, reliable drive software stack along with deep learning tools. In at least one embodiment, SoC (s) 1704 be faster, more reliable and even more energy and space-saving than conventional systems. For example, in at least one embodiment, the accelerator (s) can 1714 in combination with CPU (s) 1706, GPU (s) 1708 and data memory (s) 1716 provide a fast, efficient platform for level 3-5 autonomous vehicles.

In at least one embodiment, computer vision algorithms may be executed on CPUs that can be configured using a high level programming language, such as the C programming language, to perform a variety of processing algorithms on a variety of visual data. However, in at least one embodiment, CPUs are often unable to meet the performance requirements of many computer vision applications, such as those related to run time and power consumption. In at least one embodiment, many CPUs are unable to execute complex, real-time object detection algorithms that are used in in-vehicle ADAS applications and in practical level 3-5 autonomous vehicles.

The exemplary embodiments described here make it possible to execute several neural networks simultaneously and / or sequentially and to combine the results with one another in order to enable an autonomous functionality of the autonomous driving of level 3-5. For example, in at least one embodiment, a CNN based on DLA or discrete GPU (e.g. GPU (s) 1720 ) include text and word recognition so that a supercomputer can read and understand traffic signs, including those for which the neural network has not been specially trained. In at least one embodiment, DLA may further comprise a neural network capable of identifying, interpreting and providing a semantic understanding of characters and conveying this semantic understanding to path planning modules running on a CPU complex.

In at least one embodiment, several neural networks can be operated at the same time, such as when driving on level 3, 4 or 5. For example, in at least one embodiment a warning sign with the inscription "Caution: flashing light indicates icing" together with an electric light from several neural networks Networks can be interpreted independently or together. In at least one embodiment, the sign itself can be identified as a traffic sign by a first neural network used (e.g. a trained neural network), the text "Flashing lights indicate icing conditions" can be interpreted by a second neural network used, which the vehicle's route planning software ( which is preferably run on CPU Complex) informs you that icing conditions exist when flashing lights are detected. In at least one exemplary embodiment, “flashing lights” can be identified by operating a third neural network that is used via several individual images, which informs the vehicle's route planning software about the presence (or absence) of flashing lights. In at least one embodiment, all three neural networks can be operated simultaneously, e.g. within DLA and / or on GPU (s) 1708 .

In at least one embodiment, a CNN for face recognition and vehicle owner identification can use data from camera sensors to determine the presence of an authorized driver and / or vehicle owner of the vehicle 1700 to identify. In at least one embodiment, an always-on sensor processing engine can be used to unlock the vehicle when the owner approaches the driver's door and turns on the lights, and to disable the vehicle in safety mode when the owner leaves the vehicle. In this way the SoC (s) 1704 intended for security against theft and / or violent car hijacking.

In at least one exemplary embodiment, the CNN can use the data from microphones for the detection and identification of emergency vehicles 1796 Use to detect and identify emergency vehicle sirens. In at least one embodiment, the SoC (s) use 1704 a CNN for classifying environmental and city sounds as well as classifying visual data. In at least one embodiment, the CNN running on DLA is trained to identify the relative approach speed of the emergency vehicle (eg, using the Doppler effect). In at least one embodiment, the CNN can also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, such as by GNSS sensor (s) 1758 identified. In at least one embodiment, when operating in Europe, the CNN will attempt to identify European sirens, and in the United States, the CNN will attempt to identify only North American sirens. In at least one embodiment, a control program using an ultrasonic sensor (of ultrasonic sensors) 1762 as soon as an emergency vehicle is detected, used to execute a safety routine for the emergency vehicle, slowing the vehicle down, driving to the side, parking the vehicle and / or keeping the vehicle idling until the emergency vehicle (s) passes ( drive).

In at least one embodiment, vehicle can 1700 CPU (s) 1718 (e.g. discrete CPU (s) or dCPU (s)) connected to SoC (s) via a high-speed connection (e.g. PCIe) 1704 can be coupled. In at least one embodiment, the CPU (s) can 1718 for example, include an X86 processor. CPU (s) 1718 can be used using any function, including arbitrating potentially inconsistent results between ADAS sensors and SoC (s) 1704 and / or the monitoring of the status and condition of the controller (s) 1736 and / or an infotainment system on a chip ("Infotainment SoC") 1730 , for example.

In at least one embodiment, vehicle can 1700 GPU (s) 1720 (e.g. discrete GPU (s) or dGPU (s)) connected to SoC (s) over a high-speed connection (e.g. NVIDIA's NVLINK) 1704 can be coupled. In at least one embodiment, GPU (s) can 1720 provide additional functionality of artificial intelligence, for example by executing redundant and / or different neural networks, and can / can for training and / or updating neural networks based at least partially on inputs (eg sensor data) from sensors of the vehicle 1700 be used.

In at least one embodiment, the vehicle can 1700 further a network interface 1724 include, among other things, the wireless antenna (s) 1726 may include (e.g., one or more wireless antennas 1726 for various communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). In at least one embodiment, the network interface 1724 be used to establish wireless connectivity over the Internet with a cloud (e.g. with server (s) and / or other network devices), with other vehicles and / or with computing devices (e.g. client devices from Passengers). In at least one exemplary embodiment, a direct connection between vehicles can be used for communication with other vehicles 170 and other vehicles and / or an indirect connection (e.g. via networks and via the Internet). In at least one embodiment, direct connections can be provided using a vehicle-to-vehicle communication link. A vehicle-to-vehicle communication link can be vehicle 1700 Information about vehicles near vehicle 1700 provide (e.g. vehicles in front of, to the side and / or behind the vehicle 1700 ). In at least one exemplary embodiment, the aforementioned functionality can be part of a cooperative adaptive cruise control functionality of the vehicle 1700 be.

In at least one embodiment, the network interface 1724 comprise a SoC that provides modulation and demodulation functionality and the control (s) 1736 enabled to communicate over wireless networks. In at least one embodiment, the network interface 1724 include a radio frequency front end for upconversion from baseband to radio frequency and downconversion from radio frequency to baseband. In at least one embodiment, frequency conversions can be performed in any technically suitable manner. For example, frequency conversions can be performed by known methods and / or using super heterodyne methods. In at least one exemplary embodiment, the high-frequency front-end functionality can be provided by a separate chip. In at least one embodiment, the network interface can include wireless functionality for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN and / or other protocols for wireless data transmission.

In at least one embodiment, the vehicle can 1700 further one / more data memories 1728 include, without restriction, an off-chip memory (e.g. off-SoC (s) 1704 ) can include. In at least one embodiment, the data store (s) can 1728 include, without limitation, one or more storage elements including RAM, SRAM, dynamic random-access memory ("DRAM"), video random-access memory ("VRAM"), flash, hard drives and / or other components and / or devices that can store at least one data bit.

In at least one embodiment, vehicle can 1700 more GNSS sensor (s) 1758 (eg GPS and / or assisted GPS sensors) to support functions of mapping, perception, occupancy grid creation and / or route planning. In at least one embodiment, any number of GNSS sensor (s) can 1758 including, without limitation, a GPS using a USB connection with an Ethernet-to-serial bridge (e.g. RS-232).

In at least one embodiment, the vehicle can 1700 more RADAR sensor (s) 1760 include. The RADAR sensor (s) 1760 can (can) from the vehicle 1700 can be used for vehicle detection over long distances, even in darkness and / or bad weather conditions. In at least one embodiment, the RADAR functionality security levels can be ASIL B. RADAR sensor (s) 1760 can / can CAN and / or bus 1702 (e.g. for the transmission of data from RADAR sensor (s) 1760 used to control and access object tracking data, with access to ethernet to access raw data in some examples. In at least one embodiment, a variety of types of RADAR sensors can be used. For example and without restriction, RADAR sensor (s) 1760 Be suitable for using RADAR on the front, back and side. In at least one embodiment, one or more RADAR sensor (s) 1760 Pulse Doppler RADAR sensor (s).

In at least one embodiment, the RADAR sensor (s) can 1760 various configurations such as long range narrow field of view, short range wide field of view, short range side coverage, etc. In at least one embodiment, the RADAR can be used over long distances for adaptive cruise control functionality. In at least one exemplary embodiment, RADAR systems with a long range can provide a wide field of view, which is realized by two or more independent scans, for example within a range of 250 m. In at least one embodiment, the RADAR sensor (s) can 1760 help to distinguish between static and moving objects and can (can) from the ADAS system 1738 can be used for emergency brake assist and forward collision warning. Sensors 1760 (s) included in a long-range RADAR system can, without limitation, monostatic multimodal RADAR with several (e.g. six or more) fixed RADAR antennas and one Include high speed CAN and FlexRay interface. In at least one embodiment with six antennas, four central antennas can generate a focused radiation pattern that is designed so that the surroundings of vehicles 1700 is detected at higher speeds with minimal disruption from traffic in adjacent lanes. In at least one exemplary embodiment, the other two antennas can expand the field of view so that vehicles that are in the vehicle's lane 1700 entering or exiting them can be recorded quickly.

For example, in at least one embodiment, medium-range RADAR systems may include a range of up to 160 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 150 degrees (rear). In at least one embodiment, short-range RADAR systems can use any number of RADAR sensor (s) without limitation. 1760 to be installed on both ends of the rear bumper. In at least one embodiment, when a RADAR sensor system is installed at both ends of the rear bumper, it can generate two beams that constantly monitor the blind spot in the rear and adjacent areas of the vehicle. In at least one embodiment, short-range RADAR systems can be used in the ADAS system 1738 can be used to detect a blind spot and / or to support a lane change.

In at least one embodiment, the vehicle can 1700 further one / more ultrasonic sensor (s) 1762 include. The ultrasonic sensor (s) 1762 , the front, rear and / or side of the vehicle 1700 can be attached, can / can be used for parking assistance and / or for creating and updating an occupancy grid. In at least one embodiment, a wide variety of ultrasonic sensor (s) can 1762 used, and various ultrasonic sensor (s) 1762 can be used for different detection areas (e.g. 2.5 m, 4 m). In at least one embodiment, the ultrasonic sensor (s) can 1762 work on the functional safety levels of ASIL B.

In at least one embodiment, the vehicle can 1700 one / more LIDAR sensor (s) 1764 include. The LIDAR sensor (s) 1764 can be used for object and pedestrian detection, emergency braking, collision avoidance and / or other functions. In at least one embodiment, LIDAR sensor (s) can 1764 correspond to the functional safety level ASIL B. In at least one embodiment, vehicle can 1700 several LIDAR sensors 1764 (e.g. two, four, six, etc.) that can be provided using Ethernet (e.g. to provide data for a Gigabit Ethernet switch).

In at least one embodiment, LIDAR sensor (s) can 1764 provide a list of objects and their distances for a 360 degree field of view. In at least one embodiment, commercially available LIDAR sensor (s) 1764 a specified range of approx. 100 m with an accuracy of
2cm - 3cm and with support for a 100 Mbps Ethernet connection, for example. In at least one embodiment, one or more non-protruding lidar sensors 1764 be used. In such an embodiment, the LIDAR sensor (s) 1764 be implemented as a small device that is placed in the front, rear, sides and / or corners of the vehicle 1700 can be embedded. In at least one embodiment, LIDAR sensor (s) 1764 in such an embodiment provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees, with a range of 200 m even for objects with low reflectivity. In at least one embodiment, one or more front-mounted LIDAR sensor (s) can 1764 be configured for a horizontal field of view between 45 degrees and 135 degrees.

In at least one exemplary embodiment, LIDAR technologies, such as, for example, 3D lightning LIDAR, can also be used. 3D Flash LIDAR uses a laser flash as a transmission source to identify the area around the vehicle 1700 to be illuminated up to a distance of approx. 200 m. In at least one exemplary embodiment, a flash LIDAR unit includes, without limitation, a receptor that records the laser pulse transit time and the reflected light on each pixel, which in turn indicates the range of the vehicle 1700 up to corresponds to objects. In at least one exemplary embodiment, with each laser flash high-precision and distortion-free images of the environment can be generated with flash LIDAR. In at least one embodiment, four lightning lidar sensors can be used, one on each side of the vehicle 1700 . In at least one embodiment, 3-D flash lidar systems include, without limitation, a solid-state, 3-D lightning lidar camera having a rigid array with no moving parts other than a fan (eg, a non-scanning lidar device). In at least one embodiment, a 3-D lightning lidar device may have a 5- Use class I nanosecond laser pulses (eye-safe) per image and capture reflected laser light in the form of 3D distance point clouds and co-registered intensity data.

In at least one embodiment, the vehicle can further use IMU sensor (s) 1766 include. In at least one embodiment, the IMU sensor (s) can 1766 in at least one embodiment in the middle of the rear axle of the vehicle 1700 be arranged. In at least one embodiment, the IMU sensor (s) can 1766 include, for example and without limitation, accelerometers, magnetometers, gyroscope (s), magnetic compass (s), and / or other types of sensors. In at least one embodiment, such as in six-axis applications, the IMU sensor (s) can 1766 include, without limitation, accelerometers and gyroscopes. In at least one embodiment, such as. B. in nine-axis applications, the IMU sensor (s) can 1766 include, without limitation, accelerometers, gyroscopes, and magnetometers.

In at least one embodiment, the IMU sensor (s) can 1766 implemented as a miniature version of GPS / INS (GPS / INS = High Performance GPS-Aided Inertial Navigation System), the inertial sensors of micro-electro-mechanical systems (MEMS = Micro-Electro-Mechanical Systems), a highly sensitive GPS receiver and advanced Kalman Combined filter algorithms to estimate position, speed and attitude. In at least one embodiment, the IMU sensor (s) can 1766 the vehicle 1700 be able to estimate course without requiring input from a magnetic sensor by changing speed from GPS to IMU sensor (s) 1766 can be observed and correlated directly. In at least one embodiment, the IMU sensor (s) can 1766 and the GNSS sensor (s) 1758 can be combined in a single integrated unit.

In at least one embodiment, the vehicle can 1700 one or more microphones 1796 include those in and / or around the vehicle 1700 are attached around. In at least one embodiment, the microphone (s) can 1796 can be used, among other things, for the detection and identification of emergency vehicles.

In at least one embodiment, the vehicle can 1700 further include any number of camera types including stereo camera (s) 1768, wide angle camera (s) 1770 , Infrared camera (s) 1772 , Surround camera (s) 1774 , Remote camera (s) 1798 , Medium distance camera (s) 1776 and / or other types of cameras. In at least one embodiment, cameras can be used to capture image data around the entire periphery of the vehicle 1700 used around. In at least one embodiment, the types of cameras used depend on the vehicle 1700 from. In at least one embodiment, any combination of camera types can be used to provide the required coverage around the vehicle 1700 around to provide. In at least one embodiment, the number of cameras used can be different depending on the embodiment. In at least one embodiment, vehicle could 1700 for example, six cameras, seven cameras, ten cameras, twelve cameras, or some other number of cameras. For example, the cameras can support Gigabit Multimedia Serial Link (“GMSL”) and / or Gigabit Ethernet without restriction. In at least one embodiment, each of the cameras will be described in greater detail with respect to FIG 17A and 17B described.

In at least one embodiment, the vehicle can 1700 further one / more vibration sensor (s) 1742 include. The vibration sensor (s) 1742 can / can vibrate components of the vehicle 1700 , such as axis (s), measure. For example, in at least one exemplary embodiment, changes in the vibrations can indicate a change in the road surface. If in at least one embodiment two or more vibration sensors 1742 are used, vibration differences can be used to determine the friction or slip of the road surface (e.g. if there is a vibration difference between a driven axle and a freely rotating axle).

In at least one embodiment, the vehicle can 1700 an ADAS system 1738 include. The ADAS system 1738 may include a SoC in some examples without limitation. In at least one embodiment, the ADAS system 1738 comprise without limitation any number and combination of an autonomous / adaptive / automatic cruise control system (“ACC”), a cooperative adaptive cruise control system (“CACC”), a front impact warning system (“FCW”) and an automatic emergency braking system (“AEB”), a system for warning of leaving the lane ("LDW"), a system to support lane keeping ("LKA"), a system for warning of the blind spot ("BSW"), a system for warning of rearward cross traffic ("RCTW") ), a Collision Warning System (“CW”), a Lane Centering System (“LC”) and / or other systems, features and / or functions.

In at least one embodiment, the ACC system can use RADAR sensor (s) 1760 , LIDAR sensor (s) 1764 and / or use any number of camera (s). In at least one exemplary embodiment, the ACC system can comprise a longitudinal ACC system and / or a lateral ACC system. In at least one exemplary embodiment, the longitudinal ACC system monitors and controls the distance to the vehicle immediately in front of the vehicle 1700 and adjusts the speed of the vehicle 1700 automatically to maintain a safe distance from vehicles in front. In at least one exemplary embodiment, the lateral ACC system maintains the distance and advises the vehicle 1700 to change lanes if necessary. In at least one embodiment, the side ACC is related to other ADAS applications such as LC and CW.

In at least one embodiment, the CACC system uses information from other vehicles via the network interface 1724 and / or the wireless antenna (s) 1726 can be received by other vehicles via a wireless connection or indirectly via a network connection (e.g. via the Internet). In at least one exemplary embodiment, direct connections can be provided by a vehicle-to-vehicle communication connection (“V2V”), while indirect connections can be provided by an infrastructure-to-vehicle communication connection (“I2V”). In general, the V2V communication concept includes information about vehicles immediately ahead (e.g. vehicles immediately in front of and in the same lane as the vehicle 1700 ), while the I2V communication concept provides information about the traffic at a greater distance. In at least one embodiment, the CACC system can comprise one or both of the information sources I2V and V2V. In at least one embodiment, the CACC system can relate to information about vehicles ahead of vehicle 1700 be more reliable and has the potential to improve traffic flow and reduce congestion on the road.

In at least one exemplary embodiment, the FCW system is designed in such a way that it draws the driver's attention to a danger so that he can take corrective action. In at least one embodiment, the FCW system uses a forward facing camera and / or one or more RADAR sensor (s) 1760 coupled to a dedicated processor, DSP, FPGA and / or ASIC that is electrically connected to the driver's feedback, such as a display, speaker and / or vibrating component. In at least one exemplary embodiment, the FCW system can provide a warning, for example in the form of a sound, a visual warning, a vibration and / or a rapid braking pulse.

In at least one exemplary embodiment, the AEB system detects an impending forward collision with another vehicle or another object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. In at least one embodiment, the AEB system can have one or more forward facing cameras and / or one or more RADAR sensors 1760 that are coupled to a dedicated processor, DSP, FPGA and / or ASIC. If the AEB system detects a danger in at least one exemplary embodiment, the AEB system generally first warns the driver to take corrective measures to avoid a collision, and if the driver does not take corrective measures, the AEB system can brake automatically actuate to prevent or at least mitigate the effects of the predicted collision. In at least one exemplary embodiment, the AEB system can include techniques such as dynamic braking assistance and / or braking in the event of an impending collision.

In at least one exemplary embodiment, the LDW system is equipped with visual, acoustic and / or tactile warnings such as steering wheel or seat vibrations in order to warn the driver when the vehicle is in motion 1700 runs over the lane markings. In at least one exemplary embodiment, the LDW system is not activated if the driver indicates an intended departure from the lane by actuating a blinker. In at least one embodiment, the LDW system may use forward facing cameras that are coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to the driver's feedback, such as a display, speaker, and / or or a vibration component. In at least one embodiment, the LKA system is a variant of the LDW system. The LKA system is intended for inputting steering or braking inputs to the vehicle 1700 to correct when the vehicle 1700 begins to leave the lane.

In at least one exemplary embodiment, the BSW system detects and warns the driver of vehicles in the blind spot of an automobile. In at least one embodiment, the BSW system can provide a visual, audible, and / or tactile alarm to indicate that the collision or lane change is unsafe. In at least one embodiment, the BSW system can provide an additional warning when the driver is using a blinker. In at least one embodiment, the BSW system can include rear-facing camera (s) and / or RADAR sensor (s) 1760 that are coupled to a dedicated processor, DSP, FPGA and / or ASIC that is electrically coupled to the driver feedback, such as a display, a loudspeaker and / or a vibration component.

In at least one embodiment, the RCTW system can provide visual, audible, and / or tactile notification when reversing the vehicle 1700 an object is detected outside the range of the reversing camera. In at least one embodiment, the RCTW system includes an AEB system to ensure that the vehicle brakes are applied to avoid an impact. In at least one embodiment, an RCTW system may have one or more rearward-facing RADAR sensor (s) 1760 that is / are coupled to a dedicated processor, DSP, FPGA and / or ASIC that is / are electrically coupled to driver feedback, such as B. a display, a speaker and / or a vibrating component.

In at least one exemplary embodiment, conventional ADAS systems can be prone to false-positive results that can be annoying and distracting for the driver, but are usually not catastrophic since conventional ADAS systems warn the driver and give him the opportunity to admit decide whether a security status actually exists and act accordingly. In at least one exemplary embodiment, the vehicle decides 1700 in the case of contradicting results, whether the result is from a primary or a secondary computer (e.g. first control 1736 or second control 1736 ) should be taken into account. For example, the ADAS system 1738 in at least one embodiment, be a replacement and / or secondary computer that provides perceptual information for a rationality module of a replacement computer. In at least one exemplary embodiment, redundant diverse software can run on hardware components on the rationality monitor of the reserve computer in order to detect perceptual errors and dynamic driving tasks. In at least one embodiment, outputs from the ADAS system 1738 for a higher-level MCU. In at least one embodiment, if outputs from the primary computer and secondary computer conflict, the higher-level MCU determines how the conflict can be resolved to ensure safe operation.

In at least one embodiment, the primary computer can be configured to provide the higher-level MCU with a confidence value that indicates the primary computer's confidence in the selected result. In at least one embodiment, if the confidence exceeds a threshold, the higher-level MCU can follow the direction of the primary computer regardless of whether the secondary computer provides a contradicting or inconsistent result. In at least one embodiment where the confidence does not meet a threshold and where the primary and secondary computers display different results (e.g., a conflict), the parent MCU may mediate between the computers to determine an appropriate result.

In at least one embodiment, the host MCU may be configured to operate a neural network (s) that are trained and configured to operate based at least in part on outputs from the primary computer and the Secondary computer, determines conditions under which the secondary computer provides false alarms. In at least one embodiment, the neural network (s) in the higher-level MCU can learn when and when the output of the secondary computer can and cannot be trusted. For example, if the secondary computer is a RADAR-based FCW system in at least one embodiment, the neural network (s) in the higher-level MCU can learn when the FCW system identifies metallic objects that it is in reality it is not a hazard, such as a drainage grate or a manhole cover that triggers an alarm. In at least one embodiment, if the secondary computer is a camera-based LDW system, a neural network in the higher-level MCU can learn to override the LDW when cyclists or pedestrians are present and leaving the lane is actually the safest maneuver. In at least one embodiment, the higher-level MCU can comprise at least one DLA or GPU unit that is suitable for operating one or more neural networks with associated memory. In at least one embodiment, the control MCU can be a component of SoC (s) 1704 include and / or be included as such.

In at least one embodiment, the ADAS system 1738 include a secondary computer that performs ADAS functionality using traditional rules of computer vision. In at least one embodiment, the secondary computer can operate using classic if-then rules, and the presence of one or more neural networks in the higher-level MCU can improve reliability, safety, and performance . For example, in at least one exemplary embodiment, the overall system becomes more fault-tolerant due to different implementation and intentional non-identity, in particular to errors caused by the functionality of the software (or the software-hardware interface). For example, if in at least one embodiment there is a software bug or error in the software running on the primary computer and non-identical software code on the secondary computer delivers the same overall result, then the higher-level MCU can have greater confidence that the overall result is correct and the error is in the primary computer's software or hardware is not causing a material failure.

In at least one embodiment, the output of the ADAS system 1738 be fed into the perception block of the primary computer and / or into the block of the dynamic driving task of the primary computer. For example, if in at least one embodiment the ADAS system 1738 displays a forward crash warning due to an object immediately ahead, the perception block can use this information when identifying objects using this information. In at least one embodiment, the secondary computer can have its own neural network that is trained and thus reduces the risk of false positive results, as described here.

In at least one embodiment, the vehicle can 1700 further an infotainment SoC 1730 include (e.g. an on-board infotainment system (IVI)). Although shown and described as a SoC, the infotainment system can 1730 in at least one embodiment, it may not be a SoC and may include two or more discrete components without limitation. In at least one exemplary embodiment, the infotainment SoC 1730 include, without limitation, a combination of hardware and software necessary to provide audio (e.g. music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g. television, films, streaming, etc.), telephone (e.g. hands-free calls) , Network connection (e.g. LTE, WiFi etc.) and / or information services (e.g. navigation systems, parking aid, a radio data system, vehicle-related information such as fuel level, total distance traveled, brake fluid level, oil level, door open / closed, air filter information, etc.) to the vehicle 1700 . Infotainment SoC 1730 could for example radios, turntables, navigation systems, video players, USB and Bluetooth connectivity, carputer, in-car entertainment, WiFi, steering wheel audio controls, hands-free voice control, a heads-up display ("HUD"), HMI display 1734 , a telematics device, a control panel (eg for controlling and / or interacting with various components, functions and / or systems) and / or other components. In at least one embodiment, infotainment SoC 1730 can also be used to provide information (for example visually and / or acoustically) for the user (s) of the vehicle, such as information from the ADAS system 1738 , autonomous driving information such as planned vehicle maneuvers, trajectories, information about the surroundings (e.g. intersection information, vehicle information, road information, etc.) and / or other information.

In at least one exemplary embodiment, the infotainment SoC 1730 include any amount and type of GPU functionality. In at least one exemplary embodiment, the infotainment SoC 1730 over the bus 1702 (e.g. CAN bus, Ethernet, etc.) with other devices, systems and / or components of the vehicle 1700 communicate. In at least one exemplary embodiment, the infotainment SoC 1730 be coupled to a higher-level MCU so that the GPU of the infotainment system can perform some self-controlling functions if the primary controller (s) 1736 (e.g. primary and / or backup computer of the vehicle 1700 ) fail. In at least one exemplary embodiment, the infotainment system SoC 1730 the vehicle 1700 Put into a safe stop mode (eng. chaffeur to safe stop mode), as described here.

In at least one embodiment, the vehicle can 1700 further a central display device 1732 include (e.g. a digital dashboard, an electronic instrument cluster, a digital instrument panel, etc.). The central display device 1732 may include, without limitation, a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). In at least one embodiment, the central display device 1732 Any number and combination of instruments such as speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, gear change indicator, seat belt warning light (s), parking brake warning light (s), light (s) for Engine malfunctions, information about additional restraint systems (e.g. airbag), lighting controls, controls for safety systems, navigation information, etc. In some examples, information can be shared between Infotainment SoC 1730 and central display device 1732 displayed and / or shared. In at least one embodiment, the central display device 1732 as part of the infotainment SoC 1730 be included or vice versa.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in the system 17C for inference or prediction of operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

At least one embodiment can be constructed using the techniques described above. In at least one exemplary embodiment, the robot can be an autonomous vehicle, and the networks for evaluation and generation can be implemented in the autonomous vehicle using a computer system.

17D is a diagram of a system 1776 for communication between the cloud-based server (s) and the autonomous vehicle 1700 the 17A , according to at least one embodiment. In at least one embodiment, the system 1776 without restriction server 1778 , Network (s) 1790 and any number and type of vehicles, including vehicle 1700 , include. The server (s) 1778 can / can use a variety of GPUs without limitation 1784 (A) -1784 (H) (collectively referred to herein as GPUs 1784 designated), PCIe switches 1782 (A) -1782 (H) (collectively referred to herein as PCIe switches 1782 designated) and / or CPUs 1780 (A) -1780 (B) (collectively referred to herein as CPUs 1780 designated) include. GPUs 1784 , CPUs 1780 and PCIe switches 1782 can use high-speed connections such as, without limitation, the NVLink interfaces developed by NVIDIA 1788 and / or PCIe links 1786 are interconnected. In at least one embodiment, the GPUs are 1784 via an NVLink and / or NVSwitch SoC and the GPUs 1784 and PCIe switches 1782 connected to each other via PCIe connections. In at least one embodiment, there are eight GPUs 1784 , two CPUs 1780 and four PCIe switches 1782 shown, but this is not intended to represent a limitation. In at least one embodiment, each of the servers 1778 any number of GPUs without limitation 1784 , CPUs 1780 and / or PCIe switches 1782 include in any combination. For example, in at least one embodiment, each server could 1778 eight, sixteen, thirty-two, and / or more GPUs 1784 include.

In at least one embodiment, the server (s) can 1778 via the network (s) 1790 and receive image data from vehicles representing images showing unexpected or changed road conditions, such as recently commenced road works. In at least one embodiment, the server (s) can 1778 via the network (s) 1790 and neural networks to the vehicles 1792 , updated neural networks 1792 and / or map information 1794 including, without limitation, information about traffic and road conditions. In at least one embodiment, updates to the map information 1794 including updates to an HD map 1722 include, for example, information about roadworks, potholes, diversions, floods and / or other obstacles. In at least one embodiment, neural networks 1792 , updated neural networks 1792 and / or map information 1794 result from new training and / or new experiences represented in data received from any number of vehicles in the area and / or based at least in part on training carried out in a data center (e.g. using Server (s) 1778 and / or other servers).

In at least one embodiment, the server (s) can 1778 can be used to train machine learning models (e.g. neural networks) that are at least partially based on training data. Training data can be generated by vehicles and / or in a simulation (eg using the engine). In at least one exemplary embodiment, any number of training data are tagged (for example, if the associated neural network benefits from supervised learning) and / or subjected to other preprocessing. In at least one embodiment, any set of training data is untagged and / or preprocessed (eg, when the associated neural network does not require supervised learning). In at least one embodiment, after training machine learning models, these can be used by vehicles (e.g. via network (s) 1790 broadcast to vehicles and / or machine learning models can be used by server (s) 1778 for remote monitoring of vehicles.

In at least one embodiment, the server (s) can 1778 Receive data from vehicles and apply data to current neural networks for real-time intelligent inference. In at least one embodiment, the server (s) can 1778 Supercomputers for deep learning and / or dedicated AI computers with GPU (s) 1784 include such as the DGX and DGX Station machines developed by NVIDIA. In at least one embodiment, the server (s) can 1778 however, include a deep learning infrastructure using CPU backed data centers.

In at least one embodiment, the deep learning infrastructure of server (s) 1778 can provide fast, real-time inference services and this ability to assess and review the status of processors, software and / or associated hardware in the vehicle 1700 use. For example, in at least one embodiment, the deep learning infrastructure can periodically update the vehicle 1700 received, such as a sequence of images and / or objects that represent the vehicle 1700 localized in this sequence of images (e.g. by computer-aided vision and / or other classification techniques for objects of machine learning). In at least one embodiment, the deep learning infrastructure can operate its own neural network to identify objects and compare them to those of the vehicle 1700 to compare identified objects. If the results do not match and the deep learning infrastructure concludes that the AI is in vehicle 1700 is faulty, then the server (s) can 1778 a signal to the vehicle 1700 send and a fail-safe computer of the vehicle 1700 instruct them to take control, notify passengers, and perform a safe parking maneuver.

In at least one embodiment, the server (s) can 1778 one / more graphics processor (s) 1784 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT 3). In at least one embodiment, the combination of GPU-powered servers and inference acceleration can enable real-time responsiveness. In at least one embodiment, for example when performance is less critical, servers powered by CPUs, FPGAs, and other processors can be used for inference purposes. In at least one embodiment, the hardware structure (s) 1415 used to carry out one or more exemplary embodiments. Details on the hardware structure (x) 1415 are here in connection with 14A and / or 14B provided.

COMPUTER SYSTEMS

18th Figure 8 is a block diagram showing an exemplary computer system, which may be a system of interconnected devices and components, a system-on-a-chip (SOC), or a combination thereof 1800 formed with a processor; which may comprise execution units for executing an instruction, according to at least one embodiment. In at least one embodiment, the computer system 1800 include, without limitation, a component such as a processor 1802 for the use of execution units including logic for executing algorithms for process data according to the present disclosure, such as, for example, in an exemplary embodiment described here. In at least one embodiment, the computer system 1800 Processors such as the PENTIUM® processor family, Xeon ™, Itanium®, XScale ™ and / or StrongARM ™, Intel® Core ™, or Intel® Nervana ™ include microprocessors available from Intel Corporation of Santa Clara, California although other systems (including PCs with other microprocessors, engineering workstations, set-top boxes, and the like) can be used. In at least one embodiment, the computer system 1800 run a version of the WINDOWS operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (e.g., UNIX and Linux), embedded software, and / or graphical user interfaces can be used.

Embodiments can be used in other devices such as handheld devices and embedded applications. Some examples of handheld devices include cell phones, Internet Protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld PCs. In at least one embodiment, embedded applications can include a microcontroller, a digital signal processor ("DSP"), a system on a chip, network computers ("NetPCs"), set-top boxes, network hubs, wide area networks ("WAN") -Switches or any other system that can execute one or more commands, according to at least one embodiment.

In at least one embodiment, the computer system 1800 among other things a processor 1802 include, among other things, one or more execution units 1808 for training and / or inferencing machine learning models according to the techniques described here. In at least one embodiment, the system is 18th a single processor desktop or server system, but in another embodiment, System 18th be a multiprocessor system. In at least one embodiment, the processor 1802 without limitation a microprocessor with a complex instruction set computer (CISC), a microprocessor with a reduced instruction set (RISC), a microprocessor with a very long instruction word (VLIW) , a processor that implements a combination of instruction sets, or any other processing device such as a digital signal processor. In at least one embodiment, the processor 1802 with a processor bus 1810 coupled the data signals between the processor 1802 and other components in the computer system 1800 can transfer.

In at least one embodiment, processor 1802 an internal level without restriction 1 ("L1") Cache memory ("Cache") 1804 include. In at least one embodiment, processor 1802 have a single internal cache or multiple levels of internal cache memory. In at least one embodiment, the cache memory may be external to the processor 1802 are located. Other embodiments may also include a combination of internal and external caches, depending on the particular implementation and need. In at least one embodiment, the register file 1806 store various types of data in various registers including, without limitation, integer registers, floating point registers, status registers, and instruction pointer registers.

The execution unit is located in at least one exemplary embodiment 1808 which includes, among other things, logic for performing integer and floating point operations, also in the processor 1802 . processor 1802 may also include microcode (“ucode”) read only memory (“ROM”) that stores microcode for specific macroinstructions. In at least one embodiment, the execution unit 1808 Logic for handling a packed instruction set 1809 include. In at least one embodiment, the inclusion of the packed instruction set 1809 into the instruction set of a general purpose processor 1802 along with associated circuitry for executing instructional operations used by many multimedia applications using packed data in a general purpose processor 1802 be performed. In one or more embodiments, many multimedia applications can be run faster and more efficiently by using the full width of a processor's data bus to perform operations on packed data, thereby eliminating the need to send smaller units of data over the processor's data bus to perform one or more operations, one data item at a time.

In at least one embodiment, the execution unit 1808 also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits. In at least one embodiment, the computer system 1800 without limitation a memory 1820 include. In at least one embodiment, the memory 1820 be implemented as a dynamic random access memory (“DRAM”) device, static random access memory (“SRAM”) device, flash memory device, or other storage device. Storage 1820 can instruction (s) 1819 and / or data 1821 store represented by data signals received from the processor 1802 can be executed.

In at least one embodiment, the system logic chip can be connected to the processor bus 1810 and the memory 1820 be coupled. In at least one embodiment, the system logic chip may include, without limitation, a memory controller hub (“MCH”) 1816, and the processor 1802 can through the processor bus 1810 with the MCH 1816 communicate. In at least one embodiment, MCH 1816 a high bandwidth storage path 1818 to memory 1820 for command and data storage and for the storage of graphic commands, data and textures. In at least one embodiment, MCH 1816 Data signals between processor 1802 , Storage 1820 and other components in the computer system 1800 route and data signals between processor bus 1810 , Storage 1820 and a system I / O 1822 bypass. In at least one exemplary embodiment, the system logic chip can provide a graphics port for coupling to a graphics controller. In at least one embodiment, MCH 1816 via a high-speed storage path 1818 with memory 1820 and graphics / video card 1812 via an Accelerated Graphics Port (“AGP”) interconnect 1814 with MCH 1816 be coupled.

In at least one embodiment, the computer system 1800 use a System I / O 1822, which is a proprietary hub interface bus, to MCH 1816 to be coupled with the I / O control hub ("ICH") 1830. In at least one embodiment, ICH 1830 Provide direct connections to some I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus may, without limitation, be a high speed I / O bus for connecting peripheral devices to the memory 1820 , the chipset and the processor 1802 include. Examples can, without limitation, be an audio control 1829 , a firmware hub ("Flash BIOS") 1828, a wireless transceiver 1826 , a data store 1824 , an older 1823 I / O controller that contains user input and keyboard interfaces, a serial expansion port 1827 such as Universal Serial Bus ("USB"), and a network controller 1834 include. The data store 1824 may include a hard drive, floppy disk drive, CD-ROM device, flash memory device, or other mass storage device.

According to at least one embodiment 18th a system that includes interconnected hardware devices or "chips" while 18th may show an exemplary system-on-a-chip (“SoC”) in other exemplary embodiments. In at least one embodiment, the in 18th devices shown may be connected with proprietary connections, standardized connections (e.g. PCIe) or a combination thereof. In at least one embodiment, there are one or more components of the system 1800 interconnected by Compute Express Link (CXL) connections.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in the system 18th for inference or prediction of operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

19th Figure 3 shows a block diagram showing an electronic device 1900 for using a processor 1910 shows, according to at least one embodiment. In at least one embodiment, an electronic device 1900 for example and without limitation, a notebook, tower server, rack server, blade server, laptop, desktop, tablet, mobile device, phone, embedded computer, or any other suitable electronic device.

In at least one embodiment, the system 1900 without limitation a processor 1910 that is communicatively coupled to any suitable number or type of components, peripherals, modules, or devices. In at least one embodiment, the processor is 1910 Coupled via a bus or an interface, such as a 1 ° C bus, a system management bus (“SMBus”), a low pin count (LPC) bus, a serial peripheral interface (“SPI”), a high definition Audio ("HDA") bus, a Serial Advance Technology Attachment ("SATA") bus, a Universal Serial Bus ("USB") (versions 1, 2, 3) or a Universal Asynchronous Receiver / Transmitter ("UART" )-Bus. 19th FIG. 12 shows a system that includes interconnected hardware devices or “chips”, according to at least one embodiment 19th may show an exemplary system on-a-chip (“SoC”) in other exemplary embodiments. In at least one embodiment, the in 19th devices shown can be connected to one another by proprietary connections, standardized connections (e.g. PCIe) or a combination thereof. In at least one embodiment, one or more components of 19th interconnected by Compute Express Link (CXL) connections.

In at least one embodiment, can 19th an ad 1924 , a touch screen 1925 , a touchpad 1930 , a unit for near field communication ("NFC") 1945 , a sensor hub 1940 , a thermal sensor 1946 , an express chipset ("EC") 1935 , a Trusted Platform Module ("TPM") 1938 , BIOS / firmware / flash memory ("BIOS", FW Flash ") 1922 , a DSP 1960 , a drive "SSD or HDD") 1920 like a solid state disk ("SSD") or a hard disk drive ("HDD"), a wireless local area network unit ("WLAN") 1950 , a bluetooth unit 1952 , a Wireless Wide Area Network ("WWAN") unit 1956 , a Global Positioning System (GPS) 1955 , a camera ("USB 3. 0 camera") 1954 , e.g. B. a USB 3.0 camera, or a Low Power Double Data Rate ("LPDDR") storage unit ("LPDDR3") 1915 include z. B. is implemented in the LPDDR3 standard. Each of these components can be implemented in any suitable manner.

In at least one embodiment, other components can be communicative to the processor via the components discussed above 1910 be coupled. In at least one Embodiment can be an accelerometer 1941 , an ambient light sensor ("ALS") 1942 , a compass 1943 and a gyroscope 1944 communicative with the sensor hub 1940 be coupled. In at least one embodiment, a temperature sensor 1939 , a fan 1937 , a keyboard 1946 and a touchpad 1930 communicative with the EC 1935 be coupled. In at least one embodiment, the loudspeaker 1963 , a headphone 1964 and a microphone ("Mic") 1965 communicative with an audio unit ("audio codec and class D-amp") 1964 be coupled, which in turn communicatively with the DSP 1960 can be coupled. In at least one embodiment, the audio unit 1964 include, for example and without limitation, an audio encoder / decoder ("codec") and a class D amplifier. In at least one embodiment, the SIM card ("SIM") 1957 communicative with the WWAN unit 1956 be coupled. In at least one exemplary embodiment, components such as the WLAN unit 1950 and the bluetooth unit 1952 as well as the WWAN unit 1956 implemented in a Next Generation Form Factor ("NGFF").

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in the system 19th for inference or prediction of operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

At least one embodiment can be carried out using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

20th shows a computer system 2000 , according to at least one embodiment. In at least one embodiment, the computer system is 2000 configured to implement various methods and methods described in this disclosure.

In at least one embodiment, the computer system comprises 2000 without restriction at least one central processing unit ("CPU") 2002 with a communication bus 2010 connected using any suitable protocol such as PCI ("Peripheral Component Interconnect"), Peripheral Component Interconnect Express ("PCI-Express"), AGP ("Accelerated Graphics Port"), HyperTransport or any other bus or point to-point communication protocol or any other bus or point-to-point communication protocols is implemented. In at least one embodiment, the computer system comprises 2000 a main memory without restriction 2004 and control logic (e.g. implemented as hardware, software, or a combination thereof), and the data is in main memory 2004 which may take the form of random access memory ("RAM"). In at least one embodiment, a network interface subsystem (“network interface”) 2022 is an interface to other computing devices and networks for receiving data from and for transmitting data to other systems from the computer system 2000 provided.

In at least one embodiment, the computing system comprises 2000 , in at least one embodiment, including the input devices 2008 , the parallel processing system 2012 and the display devices 2006 that can be realized with a conventional cathode ray tube (“CRT”), a liquid crystal display (“LCD”), a light emitting diode (“LED”), a plasma display or other suitable display technologies. In at least one embodiment, user inputs are from input devices 2008 such as keyboard, mouse, touchpad, microphone and more. In at least one embodiment, each of the aforementioned modules can be arranged on a single semiconductor platform to form a processing system.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in the system 20th for inference or prediction of operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

At least one embodiment can be carried out using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a computer system as described above.

21 shows a computer system 2100 , according to at least one embodiment. In at least one embodiment, the computer system comprises 2100 , without limitation, a computer 2110 and a USB stick 2120 . In at least one embodiment, computers 2110 include, without limitation, any number and type of processor (s) (not shown) and memory (not shown). In at least one embodiment, includes computers 2110 , without limitation, a server, a cloud instance, a laptop and a desktop computer.

In at least one embodiment, the USB stick comprises 2120 one processing unit without restriction 2130 , a USB interface 2140 and USB interface logic 2150 . In at least one exemplary embodiment, the processing unit 2130 any command execution system, device, or device capable of executing commands. In at least one embodiment, the processing unit 2130 include, without limitation, any number and type of processing cores (not shown). In at least one embodiment, the processing core comprises 2130 an application specific integrated circuit ("ASIC") optimized to perform any number and type of operations associated with machine learning. In at least one embodiment, is the processing core 2130 for example, a Tensor Processing Unit (“TPC”) that is optimized for performing machine learning inference operations. In at least one embodiment, is the processing core 2130 an image processing unit (“VPU”) optimized to perform inference operations related to machine image processing and machine learning.

In at least one embodiment, the USB interface can 2140 be any type of USB plug or USB socket. In at least one embodiment, the USB interface is 2140 for example a USB 3.0 Type-C socket for data and power. In at least one embodiment, the USB interface is 2140 a USB 3.0 connector of type A. In at least one embodiment, the USB interface logic 2150 comprise any amount and type of logic that allows the processing unit 2130 allows via the USB port 2140 an interface with devices (e.g. computer 2110 ) to build.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in the system 21 for inference or prediction of operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

At least one embodiment can be carried out using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a computer system as described above.

22A shows an exemplary architecture in which a variety of GPUs 2210-2213 with a variety of multi-core processors 2205-2206 over high speed connections 2240-2243 is communicatively coupled (e.g. buses, point-to-point connections, etc.). In one embodiment, the links support high speed links 2240-2243 a communication throughput of 4 GB / s, 30 GB / s, 80 GB / s or higher. Various connection protocols can be used including, but not limited to, PCIe 4.0 or 5.0 and NVLink 2.0.

Additionally, and according to one embodiment, there are two or more of the GPUs 2210-2213 over high speed connections 2229-2230 interconnected using the same or different protocols / links than those used for the high-speed links 2240-2243 used can be implemented. Similarly, two or more multi-core processors 2205-2206 over the high speed connection 2228 which can be symmetrical multiprocessor (SMP) buses operating at 20 GB / s, 30 GB / s, 120 GB / s, or higher. Alternatively, the entire Communication between different in 22A The system components shown are made via the same protocols / links (e.g. via a common connection structure).

In one embodiment, each is a multi-core processor 2205-2206 through the storage connections 2226-2227 communicative with a processor memory 2201-2202 coupled, and any graphics processor 2210-2213 is via the GPU memory connections 2250-2253 communicative with the GPU memory 2220-2223 coupled. The storage connections 2226-2227 and 2250-2253 can use the same or different memory access technologies. As an example, without limitation, processor memory 2201-2202 and GPU memory 2220-2223 volatile memory such as dynamic random access memory (DRAM) (including stacked DRAMs), graphics DDR-SDRAM (GDDR) (e.g. GDDR5, GDDR6) or high bandwidth memory (HBM) and / or non-volatile memory such as 3D XPoint or Nano Be ram. In one embodiment, a portion of the processor memory 2201-2202 volatile memory and another part of non-volatile memory (e.g. when using a two-level memory hierarchy (2LM)).

As described here, different processors can 2205-2206 and GPUs 2210-2213 physically to a specific store 2201-2202 or. 2220-2223 can be coupled, but a unified memory architecture can be implemented in which the same virtual system address space (also known as the “effective address space”) is distributed across different physical memories. For example, the processor memory 2201-2202 64 GB each of system memory address space and the GPU memory 2220-2223 each have 32 GB of system memory address space (which in this example results in a total of 256 GB of addressable memory).

22B shows additional details for a connection between a multi-core processor 2207 and a graphics accelerator module 2246 , according to an embodiment. The graphics accelerator 2246 may include one or more GPU chips integrated on a line card that connects over the high-speed link 2240 with the processor 2207 is coupled. Alternatively, the graphics accelerator 2246 in the same package or chip as the processor 2207 be integrated.

In at least one embodiment, the depicted processor comprises 2207 a variety of cores 2260A-2260D , each with a translation lookaside buffer 2261A-2261D and one or more caches 2262A-2262D . In at least one embodiment, the cores 2260A-2260D include various other components for executing instructions and processing data that are not shown. The caches 2262A-2262D may include Level 1 (L1) and Level 2 (L2) caches. In addition, you can have one or more shared caches 2256 in the caches 2262A-2262D be included and from sets of kernels 2260A-2260D shared. For example, one embodiment of the processor includes 2207 24 Cores, each with its own L1 cache, twelve shared L2 caches, and twelve shared L3 caches. In this embodiment, one or more L2 and L3 caches are shared by two adjacent cores. The processor 2207 and the graphics accelerator module 2246 are with the system memory 2214 connected to the processor memory 2201-2202 of 22A may include.

Coherency is used for data and instructions that are in different caches 2262A-2262D , 2256 and in system memory 2214 are stored via communication between the cores via a coherence bus 2264 maintain. For example, cache coherency logic / circuitry can be assigned to each cache to respond to detected reads or writes to specific cache lines via the coherency bus 2264 to communicate. In one implementation, a cache snooping protocol is over the coherence bus 2264 implemented to eavesdrop on cache accesses.

In one embodiment, a proxy circuit couples 2225 the graphics accelerator 2246 communicative to the coherence bus 2264 so that the graphics accelerator 2246 as a peer of the cores 2260A-2260D can participate in a cache coherency protocol. In particular is an interface 2235 for connecting to the proxy circuit 2225 over the high speed connection 2240 (e.g. PCIe bus, NVLink, etc.) and an interface 2237 connects the graphics accelerator module 2246 with the connection 2240 .

In one implementation is an accelerator integration circuit 2236 for a variety of graphics processing engines 2231 , 2232 , N of the graphics accelerator 2246 provided for cache management, memory access, context management and interrupt management services. The engines 2231 , 2232 , N can each comprise a separate graphics processing unit (GPU). Alternatively, the graphics processing engines 2231 , 2232 , N different types of graphics processing engines within a GPU include, such as Graphics execution units, media processing engines (e.g. video encoders / decoders), samplers, and blit engines. In at least one embodiment, the graphics accelerator module 2246 a GPU with a variety of graphics processing engines 2231-2232 , N or graphics processing engines 2231-2232 , N can be individual GPUs that are integrated in a common housing, a line card or a chip.

In one embodiment, the accelerator comprises integration circuitry 2236 a memory management unit (MMU) 2239 to perform various memory management functions such as virtual to physical memory translations (also known as effective to real memory translations) and memory access protocols to access system memory 2214 . MMU 2239 may also include a translation lookaside buffer (TLB) (not shown) for the intermediate storage of translations from virtual / effective to physical / real addresses. In one embodiment, a cache stores 2238 Commands and data for efficient access by the engines 2231-2232 , N. In one embodiment, the 2238 and in the graphics memory 2233-2234 , M data stored with the core caches 2262A-2262D , 2256 and the system memory 2214 kept coherent. As already mentioned, this can be done via the proxy circuit 2225 for cache 2238 and the memory 2233-2234 , M (e.g. sending updates to cache 2238 in connection with changes / accesses to cache lines in the processor caches 2262A-2262D , 2256 and receiving updates from cache 2238 ).

A set of registers 2245 stores context data for threads used by the engines 2231-2232 , N for graphic processing, and a context management circuit 2248 manages thread contexts. The context management circuit 2248 For example, it can perform backup and restore operations to back up and restore the contexts of different threads during context switching (e.g. when a first thread is backed up and a second thread is saved so that a second thread can be executed by the graphics processing engine) . In the event of a context change, for example, the context management circuit 2248 Store current register values in a specific area in the memory (e.g. identified by an identifier). It can then restore register values when returning to a context. In one embodiment, an interrupt management circuit 2247 receives and processes interrupts received from system devices.

In one implementation, virtual / effective addresses are taken from the graphics processing engine 2231 through the MMU 2239 in real / physical addresses in the system memory 2214 translated. An embodiment of the accelerator integration circuit 2236 supports several (e.g. 4, 8, 16) graphics accelerator modules 2246 and / or other accelerator devices. The graphics accelerator module 2246 can for a single, on the processor 2207 running application can be dedicated or shared by multiple applications. In one embodiment, a virtualized graphics execution environment is shown in which the resources of the graphics processing engines 2231-2232 , N can be shared by multiple applications or virtual machines (VMs). In at least one embodiment, resources may be divided into “slices” that are assigned to various VMs and / or applications based on the processing requirements and priorities associated with VMs and / or applications.

In at least one embodiment, the accelerator integration circuit works 2236 as a bridge to a graphics accelerator system 2246 and provides address translation and system memory cache services. In addition, the accelerator integration circuit 2236 Providing virtualization facilities for a host processor in order to virtualize the graphics processing modules 2231-2232 Manage interrupts and memory management.

Since the hardware resources of the graphics processing engines 2231-2232 , N are explicitly assigned to a real address space that is assigned by the host processor 2207 is seen, any host processor can address these resources directly via an effective address value. A functionality of the accelerator integration circuit 2236 , in one embodiment, is the physical separation of the engines 2231-2232 , N, so that they appear as independent units to a system.

In at least one embodiment, there are one or more graphics memories 2233-2234 , M each with the engines 2231-2232 , N coupled. Graphics memory 2233-2234 , M store commands and data issued by the respective graphics processing engine 2231-2232 , N are processed. With the graphics memory 2233-2234 , M can be volatile memories such as DRAMs (including stacked DRAMs), GDDR- Memory (e.g. GDDR5, GDDR6) or HBM and / or non-volatile memory such as 3D XPoint or Nano-Ram.

In one embodiment, traffic will be sent over the link 2240 To reduce biasing techniques used to ensure that the graphics memory 2233-2234 , M stored data is the data that is most commonly used by the graphics processing engines 2231-2232 , N and preferably not from the nuclei 2260A-2260D (at least not often) used. Similarly, a default mechanism tries to be used by the cores (and preferably not the graphics processing engines 2231-2232 , N) required data in the caches 2262A-2262D , 2256 the cores and the system memory 2214 to keep.

22C Fig. 13 shows another exemplary embodiment in which the accelerator integration circuit 2236 in the processor 2207 is integrated. In this exemplary embodiment, the engines communicate 2231-2232 , N directly through the high-speed link 2240 with the accelerator integration circuit 2236 through the interface 2237 and the interface 2235 (which in turn can use any type of bus or interface protocol). The accelerator integration circuit 2236 can perform the same operations as those in 22B but possibly with a higher throughput since it is in close proximity to the coherence bus 2264 and the caches 2262A-2262D , 2256 is located. In one embodiment, various programming models are supported, including a dedicated process programming model (no virtualization of the graphics acceleration module) and shared programming models (with virtualization), which may include programming models implemented by the accelerator integration circuit 2236 can be controlled, as well as programming models that are generated by the graphics acceleration module 2246 being controlled.

In at least one embodiment, the graphics processing engines are 2231-2232 , N for a single application or process running on a single operating system. In at least one embodiment, a single application may have different application needs about the graphics processing engines 2231-2232 , N and thus provide virtualization within a VM / partition.

In at least one embodiment, the graphics processing engines 2231-2232 , N can be shared by multiple VM / application partitions. In at least one embodiment, shared models can use a system hypervisor to virtualize the graphics processing engines 2231-2232 Use, N to allow access by any operating system. For single-partition systems with no hypervisor, the engines are 2231-2232 , N for graphics processing applications part of an operating system. In at least one embodiment, an operating system can run the graphics processing engines 2231-2232 Virtualize, N to allow access to any process or application.

In at least one embodiment, the graphics accelerator module selects 2246 or a single graphics processing engine 2231-2232 , N select a process element using a process handle. In one embodiment, the process elements are in system memory 2214 and are addressable for effective addresses into real addresses using a translation technique described here. In at least one embodiment, a process handle can be an implementation-specific value that a host process has when registering its context with the graphics processing engine 2231-2232 , N is provided (ie calling the system software to add a process item to a process item linked list). In at least one embodiment, the lower 16 bits of a process handle can be an offset of the process element within a process element linked list.

22D shows an exemplary accelerator integration section 2290 . As used herein, a "slice" comprises a specific portion of the processing resources of the accelerator integration circuit 2236 . The effective application address space 2282 within system memory 2214 saves the process elements 2283 . In one embodiment, process elements 2283 in response to GPU calls 2281 of applications 2280 saved that on processor 2207 are executed. A process element 2283 contains the process status for the corresponding application 2280 . A work descriptor (WD) 2284 that is in process element 2283 can be a single job requested by an application, or it can contain a pointer to a queue of jobs. In at least one embodiment, WD is 2284 a pointer to a job request queue in the address space 2282 an application.

The graphics accelerator 2246 and / or the individual graphics processing engines 2231-2232 , N can be shared by all or a subset of processes in a system. In at least one embodiment, an infrastructure for establishing the process state and sending a WD 2284 to a graphics accelerator 2246 to start a job in a virtualized environment.

In at least one embodiment, a programming model for a dedicated process is implementation specific. In this model, a single process owns the graphics accelerator 2246 or a single graphics processing engine 2231 . Since the graphics accelerator 2246 is owned by a single process, a hypervisor initializes the accelerator integration circuit 2236 for a owning partition and an operating system initializes the accelerator integration circuit 2236 for an owning process if the graphics accelerator 2246 is assigned.

A WD pick-up unit picks up during operation 2291 in the accelerator integration section 2290 the next WD 2284 , which is an indication of the performance of one or more graphics processing engines in the graphics accelerator 2246 includes work to be done. Data from WD 2284 can in registers 2245 stored and from the MMU 2239 , the interrupt management circuit 2247 and / or the context management circuit 2248 can be used as shown. For example, one embodiment of MMU 2239 a segment / page scroller for accessing segment / page tables 2286 within the virtual address space 2285 of the operating system. The interrupt management circuit 2247 can interrupt events 2292 process that from the graphics accelerator 2246 be received. When performing graphics operations, it becomes an effective address 2293 by a graphics processing engine 2231-2232 , N is generated by the MMU 2239 translated into a real address.

In one embodiment, the same set of registers is used 2245 for any graphics processing engine 2231-2232 , N and / or graphics accelerator 2246 duplicated and can be initialized by a hypervisor or operating system. Each of these duplicated registers can be included in an accelerator integration section 2290 be included. Exemplary registers that can be initialized by a hypervisor are shown in Table 1. Table 1 - Registers initialized by the hypervisor 1 Register for controlling sections 2 Area pointer for processes planned with real addresses (RA) 3 Register for overwriting the authority mask 4th Offset of an entry in an interrupt vector table 5 Limitation of an entry in an interrupt vector table 6th Status register 7th Logical partition ID 8th Pointer to a hypervisor accelerator usage record with real address (RA) 9 Register for memory description

Exemplary registers that can be initialized by an operating system are shown in Table 2. Table 2 - Registers initialized by the operating system 1 Process and thread identification 2 Pointer for backing up / restoring a context with an effective address (EA) 3 Pointer to usage record of a virtual address (VA) accelerator 4th Pointer for memory segment table with virtual address (VA) 5 Authority mask 6th Work descriptor

In one embodiment, each WD 2284 specific to a particular graphics accelerator 2246 and / or the graphics processing engines 2231-2232 , N. It contains all the information a graphics processing engine needs 2231-2232 , N is required to do its work, or it can be a pointer to a memory location where an application has a command queue for the work to be done.

22E shows additional details for an exemplary embodiment of a common model. This embodiment includes an address space of real addresses 2298 a hypervisor in which a process item list 2299 is stored. The real address space 2298 the hypervisor is through a hypervisor 2296 accessible to the graphics accelerator engines for the operating system 2295 virtualized.

In at least one embodiment, shared programming models enable all or a subset of processes from all or a subset of partitions in a system to use a graphics accelerator module 2246 . There are two models of programming that use the graphics accelerator 2246 is used jointly by several processes and partitions: jointly used for a period of time and jointly used for graphics purposes.

In this model, the system hypervisor contains 2296 the graphics accelerator 2246 and provides its functionality for all operating systems 2295 to disposal. So a graphics accelerator 2246 the virtualization by the system hypervisor 2296 supports the graphics accelerator 2246 meet the following conditions: 1) An application's job request must be autonomous (ie, the state does not need to be maintained between jobs), or the graphics accelerator 2246 must provide a mechanism to back up and restore the context. 2) The graphics accelerator guarantees a job request from an application 2246 that it will complete in a certain amount of time, including any translation errors, or the graphics accelerator 2246 offers the possibility of anticipating the processing of an order. 3) The graphics accelerator module 2246 must guarantee balance between processes when working in a directed shared programming model.

In at least one embodiment, the application is 2280 required to make a system call to the operating system 2295 with a graphics accelerator of the type 2246 , a work descriptor (WD), an AMR value (AMR = Authority Mask Register) and a CSRP value (CSRP = Context Save / Restore Area Pointer). In at least one embodiment, describes the type of graphics accelerator module 2246 a targeted acceleration function for a system call. In at least one embodiment, the type of graphics accelerator module 2246 be a system-specific value. In at least one embodiment, the WD is specific to the graphics accelerator module 2246 formatted and can be in the form of a Graphics Accelerator 2246 command, an effective address pointer to a user-defined structure, an effective address pointer to a command queue, or any other data structure describing the graphics accelerator 2246 work to be carried out is available. In one embodiment, an AMR value is an AMR state to be used for a current process. In at least one embodiment, a value passed to an operating system is similar to an application setting an AMR. When the implementations of the accelerator integration circuit 2236 and the graphics accelerator module 2246 do not support a User Authority Mask Override Register (UAMOR), an operating system can apply a current UAMOR value to an AMR value before an AMR is passed in a hypervisor call. Hypervisor 2296 can optionally apply a current AMOR value (AMOR = Authority Mask Override Register) before an AMR in process element 2283 is passed. In at least one embodiment, CSRP is one of the registers 2245 , which is an effective address of a range in the address space 2282 an application for the graphics accelerator 2246 to save and restore the context state. This pointer is optional if no status needs to be saved between jobs or if a job is brought forward. In at least one embodiment, the context backup / restore area can be hard-wired system memory.

When receiving a system call, the operating system can 2295 check that the application 2280 registered and used for the graphics accelerator 2246 has been authorized. operating system 2295 then calls the hypervisor 2296 with the information listed in Table 3. Table 3 - Call parameters from OS to hypervisor 1 Work Descriptor (WD) 2 Authority Mask Register Value (AMR) (potentially masked) 3 Area pointer for context saving / restoring with effective address (EA) (Context Save / Restore Area Pointer, CSRP) 4th Process ID (PID) and optional thread ID (TID) 5 Pointer for use record of the accelerator with virtual address (VA) (English accelerator utilization record pointer, AURP) 6th Pointer for the stored segment table pointer (SSTP) with a virtual address 7th Service number for logical interruption (LISN)

After receiving a hypervisor call, the hypervisor checks 2296 whether the operating system 2295 registered and authorized to use the graphics accelerator 2246 had received. Hypervisor 2296 then adds process element 2283 into a process element link list for a corresponding type of graphics accelerator 2246 a. A process element can include the information listed in Table 4. Table 4 - Process Element Information 1 Work Descriptor (WD) 2 Authority Mask Register Value (AMR) (potentially masked) 3 Area pointer for context saving / restoring with effective address (EA) (Context Save / Restore Area Pointer, CSRP) 4th Process ID (PID) and optional thread ID (TID) 5 Pointer for use record of the accelerator with virtual address (VA) (English accelerator utilization record pointer, AURP) 6th Pointer for the stored segment table pointer (SSTP) with a virtual address 7th Service number for logical interruption (LISN) 8th Interrupt vector table derived from hypervisor call parameters 9 Status register value (SR) 10 Logical Partition ID (LPID) 11 Pointer for usage record of a hypervisor accelerator with real address (RA) 12th Memory Descriptor Register (SDR)

In at least one embodiment, the hypervisor initializes a plurality of registers 2290 of accelerator integration sections 2290 to 2245 .

As in 22F shown, in at least one embodiment, a uniform memory is used that is addressable via a shared virtual memory address space that is used for access to physical processor memory 2201-2202 and GPU memory 2220-2223 is used. In this implementation, the ones on the GPUs use 2210-2213 operations carried out use a common virtual / effective memory address space to access the processor memory 2201-2202 to access and vice versa, whereby the programmability is simplified. In one embodiment, a first part of a virtual / effective address space becomes processor memory 2201 , a second part to the second processor memory 2202 , a third part is the GPU memory 2220 etc. assigned. In at least one embodiment, this creates an entire virtual / effective memory space (sometimes referred to as an effective address space) on each of the processor memories 2201-2202 and GPU memory 2220-2223 distributed, which allows any processor or GPU to access any physical memory with a virtual address assigned to that memory.

In one embodiment, the circuit arrangement 2294A-2294E for the administration of specifications / coherence within one or more of the MMUs 2239A-2239E the cache coherence between the caches of one or more host processors (e.g. 2205) and the GPUs 2210-2213 secure and implements default techniques that specify physical memories in which certain types of data are to be stored. While several specifications of the circuit arrangement 2294A-2294E to manage the specification / coherence in 22F can be shown, the circuit arrangement for managing the default / coherence within an MMU of one or more host processors 2205 and / or within the accelerator integration circuit 2236 implemented.

In one embodiment, the GPU-attached memory 2220-2223 allocated as part of system memory and accessed using shared virtual memory (SVM) technology, but without the performance degradation associated with full system cache coherence. In at least one embodiment, the option is on GPU-bound memory 2220-2223 as part of the system memory without the hassle of cache coherency overhead, an advantageous operating environment for offloading to GPUs. This arrangement enables the host processor software to do so 2205 Build operands and access calculation results without the overhead of traditional I / O DMA data copies. Such traditional copies include driver calls, interrupts, and memory mapped I / O (MMIO) accesses, all of which are inefficient compared to simple memory accesses. In at least one embodiment, the ability to use cache coherency overhead on GPU-bound memory can be used 2220-2223 access can be decisive for the execution time of an outsourced calculation. For example, in cases with significant streaming write memory traffic, cache coherence overhead can be an effective write bandwidth used by a GPU 2210-2213 is seen, reduce significantly. In at least one exemplary embodiment, the efficiency of the operand structure, the efficiency of the result access and the efficiency of the GPU computation can play a role in determining the effectiveness of an outsourcing to GPUs.

In at least one embodiment, the selection of the GPU default and the host processor default is controlled by a default tracker data structure. A bias table can, for example, be a page-granular structure (ie controlled with the granularity of a memory page) that contains 1 or 2 bits per GPU-connected memory page. In at least one embodiment, a preferences table can be stored in a stolen memory area from one or more GPU-bound memories 2220-2223 with or without default cache in GPU 2210-2213 implemented (e.g. to cache frequently / recently used entries in a default table). Alternatively, an entire specification table can be managed within a graphics processor.

In at least one embodiment, a default table entry is used with each access to GPU-bound memory 2220-2223 is accessed prior to actually accessing a GPU memory, which causes the following operations. First, local requests from the GPU 2210-2213 that find their page in the GPU specification, directly to a corresponding GPU memory 2220-2223 forwarded. Local requests from a graphics processor, which find their page in the specification of the host memory, are sent to the processor 2205 forwarded (e.g. over a high-speed connection, as described above). In one embodiment, requests from the processor include 2205 that find a requested page in the host processor bias, handle a request like a normal memory read. Alternatively, requests directed to a GPU-specified page can be sent to GPU 2210-2213 to get redirected. In at least one embodiment, a GPU can convert a page to a host processor default when it is not currently using a page. In at least one embodiment, the default state of a page can be changed by either a software-based mechanism, a hardware-assisted software-based mechanism, or, for a limited set of cases, a purely hardware-based mechanism.

One mechanism for changing the default state uses an API call (e.g. OpenCL), which in turn calls the device driver of a GPU, which in turn sends a message (or a command description in a queue) to a GPU to instruct it to a default state and perform a cache flush operation on a host for some transitions. In at least one embodiment, the cache flush operation for a transition is from the default of the host processor 2205 used for GPU default, but not used for an opposite transition.

In one embodiment, cache coherency is maintained by using GPU-predetermined pages from the host processor 2205 temporarily not cached. To access these pages, Processor 2205 access from GPU 2210 request that may not grant access immediately. To communication between processor 2205 and GPU 2210 Therefore, it is beneficial to ensure that GPU-mandated pages are the ones that are being used by a GPU but not by the host processor 2205 are needed and vice versa.

The hardware structure (s) 1415 is / are used to carry out one or more exemplary embodiments. Details regarding the hardware structure (s) 1415 are here in connection with 14A and / or 14B provided.

23 FIG. 10 shows exemplary integrated circuits and associated graphics processors that may be fabricated using one or more IP cores, in accordance with various embodiments described herein. In addition to what is shown herein, other logic and circuitry may be included in at least one embodiment, including additional graphics processors / cores, peripheral interface controls, or general purpose processor cores.

23 Figure 3 is a block diagram illustrating an exemplary system on an integrated circuit 2300 shows on a chip that can be fabricated with one or more IP cores, according to at least one embodiment. In at least one embodiment, the integrated circuit comprises 2300 one or more application processor (s) 2305 (e.g. CPUs), at least one graphics processor 2310 and can also use an image processor 2315 and / or a video processor 2320 each of these processors can be a modular IP core. In at least one embodiment, the integrated circuit comprises 2300 Peripheral or bus logic including a USB control 2325 , UART control 2330 , an SPI / SDIO controller 2335 and an I.sup.2S / I.sup.2C controller 2340. In at least one embodiment, the integrated circuit 2300 a display device 2345 that is coupled to one or more of the following controllers: a High-Definition Multimedia Interface (HDMI) controller 2350 and a Mobile Industry Processor Interface (MIPI) display interface 2355 . In at least one embodiment, the memory may be a flash memory subsystem 2360 can be provided comprising a flash memory and a flash memory controller. In at least one embodiment, the memory interface can be via a controller 2365 for access to SDRAM or SRAM storage devices. In at least one embodiment, some integrated circuits additionally include an embedded security engine 2370 .

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in the integrated circuit 2300 for inference or prediction of operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

At least one embodiment can be carried out using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

24A-24B show exemplary integrated circuits and associated graphics processors that may be fabricated with one or more IP cores in accordance with various embodiments. In addition to what is shown herein, other logic and circuitry may be included in at least one embodiment, including additional graphics processors / cores, peripheral interface controls, or general purpose processor cores.

24A-24B show, as block diagrams, exemplary graphics processors for use in a SoC in accordance with the exemplary embodiments described here. 24A shows an exemplary Graphics processor 2410 an integrated circuit on a system-on-a-chip, which can be manufactured with one or more IP cores, according to at least one embodiment. 24B shows an additional exemplary graphics processor 2440 an integrated circuit on a system-on-a-chip that can be fabricated using one or more IP cores, according to at least one embodiment. In at least one embodiment, the graphics processor is 2410 of 24A a graphics processor core with low power consumption. In at least one embodiment, the graphics processor is 2440 of 24B a graphics processor core with higher performance. In at least one embodiment, each of the graphics processors 2410 , 2440 a variant of the graphics processor 2310 out 23 be.

In at least one embodiment, the graphics processor comprises 2410 a vertex processor 2405 and one or more fragment processor (s) 2415A-2415N (e.g. 2415A, 2415B, 2415C, 2415D, through 2415N-1 and 2415N). In at least one embodiment, the graphics processor 2410 run different shading programs using separate logic so that the vertex processor 2405 is optimized to perform operations for vertex shading programs while one or more fragment processor (s) 2415A-2415N Perform fragment shading operations (e.g., pixels) for fragment or pixel shading programs. In at least one embodiment, the vertex processor performs 2405 executes a vertex processing stage of a 3D graphics pipeline and generates primitives and vertex data. In at least one embodiment, the fragment processor (s) uses 2415A-2415N those from the vertex processor 2405 generated primitive and vertex data to generate a frame buffer to be displayed on a display device. In at least one embodiment, the fragment processor (s) is / are 2415A-2415N Optimized for the execution of fragment shader programs as envisaged in an OpenGL API which can be used to perform operations similar to a pixel shader program as envisaged in a direct 3D API .

In at least one embodiment, the graphics processor comprises 2410 additional one or more memory management units (MMUs) 2420A-2420B , Cache (s) 2425A-2425B and circuit connection (s) 2430A-2430B . In at least one embodiment, one or more MMU (s) 2420A-2420B for the assignment of virtual to physical addresses for the graphics processor 2410 provided, including for the vertex processor 2405 and / or the fragment processor (s) 2415A-2415N which, in addition to those in one or more cache (s) 2425A-2425B stored vertex or image / texture data can refer to vertex or image / texture data stored in the memory. In at least one embodiment, one or more MMU (s) 2420A-2420B synchronized with other MMUs within the system, including one or more MMUs connected to one or more application processor (s) 2305 , Image processor (s) 2315 and / or video processor (s) 2320 out 23 are associated so that each processor 2305-2420 can participate in a shared or unified virtual storage system. In at least one embodiment, one or more circuit connection (s) enable 2430A-2430B the graphics processor 2410 the connection with other IP cores within the SoC, either via an internal bus of the SoC or via a direct connection.

In at least one embodiment, the graphics processor comprises 2440 one or more MMU (s) 2420A-2420B , Caches 2425A-2425B , and the circuit connections 2430A-2430B of the graphics processor 2410 out 24A . In at least one embodiment, the graphics processor comprises 2440 one or more shading kernels 2455A-2455N (e.g. 2455A, 2455B, 2455C, 2455D, 2455E, 2455F through 2455N-1, and 2455N) that provide a unified shading core architecture in which a single core or type or core can execute all kinds of programmable shading code, including shading code to implement Vertex -Shades, fragment shades, and / or computation shades. In at least one embodiment, a number of shading kernels can vary. In at least one embodiment, the graphics processor includes 2440 an inter-core task manager 2445 who works as a thread distributor to distribute execution threads to one or more shading cores 2455A-2455N to distribute, and a tiling unit 2458 to speed up tiling operations for tile-based rendering, in which rendering operations for a scene are subdivided in image space, for example to take advantage of the local spatial coherence within a scene or to optimize the use of internal caches.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in the integrated circuit 24A and or 24B used to inference or predict operations that are at least partially are based on weight parameters that have been calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

At least one embodiment can be carried out using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

25A-25B show further exemplary graphics processor logic in accordance with the exemplary embodiments described here. 25A shows a graphics core 2500 , which in at least one embodiment is the graphics processor 2310 of 23 may include, and in at least one embodiment, a unified shading kernel 2455A-2455N as in 24B can represent. 25B Figure 3 shows a highly parallel general graphics processing unit 2530 , which is suitable for use on a multi-chip module, in at least one embodiment.

In at least one embodiment, the graphics core comprises 2500 a shared instruction cache 2502 , a texture unit 2518 and a cache / shared memory 2520 that are the execution resources in the graphics core 2500 are assigned together. In at least one embodiment, the graphics core 2500 several sections 2501A-2501N or include partitions for each core, and a graphics processor can have multiple instances of the graphics core 2500 include. The sections 2501A-2501N can support logic including a local instruction cache 2504A-2504N , a thread scheduler 2506A-2506N , a thread distributor 2508A-2508N and a set of registers 2510A-2510N include. In at least one embodiment, the sections 2501A-2501N a set of additional functional units (AFUs 2512A-2512N ), Floating point units (FPU 2514A-2514N ), integer arithmetic-logical units (ALUs 2516-2516N ), Address calculation units (ACU 2513A-2513N ), Double precision floating point units (DPFPU 2515A-2515N ) and matrix processing units (MPU 2517A-2517N ) include.

In at least one embodiment, the FPUs 2514A-2514N Perform single-precision (32-bit) and half-precision (16-bit) floating point operations while the DPFPUs 2515A-2515N Perform double-precision (64-bit) floating point operations. In at least one embodiment, the ALUs 2516A-2516N Perform variable-precision integer operations with 8-bit, 16-bit, and 32-bit precision and can be configured for mixed precision operations. In at least one embodiment, the MPUs 2517A-2517N can also be configured for mixed precision matrix operations, including floating point and 8-bit integer operations with half precision. In at least one embodiment, the MPUs 2517-2517N Perform a variety of matrix operations to accelerate machine learning application frameworks, including support for accelerated general matrix to matrix multiplication (GEMM). In at least one embodiment, the AFUs 2512A-2512N Perform additional logical operations that are not supported by floating point or integer units, including trigonometric operations (e.g., sine, cosine, etc.).

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in Graphics Core 2500 for inference or prediction of operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

At least one embodiment can be carried out using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

25B shows a general purpose processing unit (GPGPU) 2530 which can be configured so that highly parallel arithmetic operations can be performed by an array of graphics processing units in at least one unit, in at least one Embodiment. In at least one embodiment, the GPGPU 2530 directly with other instances of the GPGPU 2530 can be connected to create a multi-GPU cluster to improve training speed for deep neural networks. In at least one embodiment, GPGPU includes 2530 a host interface 2532 to enable connection to a host processor. In at least one embodiment, is the host interface 2532 a PCI Express interface. In at least one embodiment, the host interface can 2532 be a manufacturer-specific communication interface or communication structure. In at least one embodiment, the GPGPU receives 2530 Commands from a host processor and plans with the help of a global scheduler 2534 the threads of execution associated with these instructions on a set of compute clusters 2536A-2536H . In at least one embodiment, the computing clusters are shared 2536A-2536H a cache memory 2538 . In at least one embodiment, the cache memory 2538 as a higher level cache memory for cache memories within the compute clusters 2536A-2536H serve.

In at least one embodiment, the GPGPU comprises 2530 the memory 2544A-2544B that has a set of memory controls 2542A-2542B with the computing clusters 2536A-2536H is coupled. In at least one embodiment, the memory 2544A-2544B Various types of storage devices include dynamic random access memory (DRAM) or graphics random access memory such as synchronous graphics random access memory (SGRAM), including double data rate graphics memory (SGRAM). graphics double data rate, GDDR).

In at least one embodiment, the compute clusters comprise 2536A-2536H each a set of graphics cores, such as the graphics core 2500 of 25A , which can include several types of integer and floating point logic units that can perform arithmetic operations with a range of accuracies that are also suitable for machine learning. For example, in at least one embodiment, at least a subset of floating point units may be in each of the compute clusters 2536A-2536H can be configured to perform 16-bit or 32-bit floating point operations, while a different subset of floating point units can be configured to perform 64-bit floating point operations.

In at least one embodiment, multiple instances of the GPGPU 2530 configured to work as a compute cluster. In at least one embodiment, the communication provided by the compute clusters varies 2536A-2536H is used for synchronization and data exchange, from embodiment to embodiment. In at least one embodiment, multiple instances of GPGPU are communicating 2530 via the host interface 2532 . In at least one embodiment, the GPGPU includes 2530 an I / O hub 2539 that the GPGPU 2530 with a GPU connection 2540 couples that have a direct connection to other instances of the GPGPU 2530 enables. In at least one embodiment, the GPU is link 2540 coupled with a dedicated GPU-to-GPU bridge that enables communication and synchronization between multiple instances of the GPGPU 2530 enables. In at least one embodiment, the GPU connection is 2540 coupled to a high-speed link to send and receive data to and from other GPGPUs or parallel processors. In at least one embodiment, there are multiple instances of GPGPU 2530 in separate data processing systems and communicate via a network device that uses the host interface 2532 is accessible. In at least one embodiment, the GPU connection can 2540 configured in such a way that they can be used in addition or as an alternative to the host interface 2532 enables connection to a host processor.

In at least one embodiment, the GPGPU 2530 configured to train neural networks. In at least one embodiment, GPGPU 2530 can be used within an inference platform. In at least one embodiment, in the GPGPU 2530 is used for inferencing, GPGPU can use fewer compute clusters 2536A-2536H than when using GPGPU to train a neural network. In at least one embodiment, the memory 2544A-2544B Associated storage technology distinguish between inference and training configurations, with storage technologies with higher bandwidth being used for training configurations. In at least one embodiment, the inference configuration of the GPGPU 2530 support the inferencing of specific commands. In at least one exemplary embodiment, an inference configuration can, for example, offer support for one or more 8-bit integer scalar product instructions that can be used in inference operations for neural networks used.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in Graphics Core 2500 for inference or prediction of operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

At least one embodiment can be carried out using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

26th Figure 3 is a block diagram showing a computer system 2600 shows according to at least one embodiment. In at least one embodiment, the computing system includes 2600 a processing subsystem 2601 with one or more processors 2602 and a system memory 2604 that communicate over a connection path that is a storage hub 2605 may include. In at least one embodiment, the storage hub 2605 be a separate component within a chipset component or in one or more processor (s) 2602 be integrated. In at least one embodiment, the storage hub is 2605 over a connection 2606 with an I / O subsystem 2611 coupled. In at least one embodiment, the I / O subsystem comprises 2611 an I / O hub 2607 who made it to the computing device 2600 can enable input from one or more input device (s) 2608 to recieve. In at least one embodiment, the I / O hub can 2607 a controller for a display running in one or more processors 2602 may be included to enable output to one or more display devices 2610A provide. In at least one embodiment, one or more display device (s) 2610A that / those with the I / O hub 2607 is / are coupled, comprise a local, internal or embedded display device.

In at least one embodiment, the processing subsystem comprises 2601 one or more parallel processor (s) 2612 , who / who via a bus or other connection 2613 with the storage hub 2605 is / are connected. In at least one exemplary embodiment, the communication link can be 2613 Any number of standards-based communication link technologies or protocols, such as, but not limited to PCI Express, or a proprietary communication interface or communication structure. In at least one embodiment, one or more parallel processor (s) 2612 a computational parallel or vector processing system that may include a large number of compute cores and / or processing clusters, such as a many integrated core (MIC) processor. In at least one embodiment, one or more parallel processor (s) 2612 a graphics processing subsystem that sends pixels to one of one or more display devices 2610A that can output via the I / O hub 2607 are coupled. In at least one embodiment, one or more parallel processor (s) 2612 also include a controller and display interface (not shown) for direct connection to one or more display device (s) 2610B to enable.

In at least one embodiment, a system storage unit 2614 with the I / O hub 2607 connected to a storage mechanism for the computer system 2600 provide. In at least one embodiment, an I / O switch 2616 can be used to provide an interface mechanism that controls connections between the I / O hub 2607 and other components, such as a network adapter 2618 and / or a wireless network adapter 2619 that can be integrated into the platform and various other devices that have one or more additional devices 2620 can be added. In at least one embodiment, the network adapter 2618 be an ethernet adapter or other wired network adapter. In at least one embodiment, the wireless network adapter 2619 one or more Wi-Fi, Bluetooth, near field communication (NFC), or other network devices that contain one or more wireless radios.

In at least one embodiment, the computing system can 2600 other components not explicitly shown, including USB or other port connections, optical storage drives, video capture devices, and the like, that also connect to the I / O hub 2607 can be connected. In at least one embodiment, communication paths that various components in 26th interconnect, can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect) based protocols (e.g. PCI-Express) or other bus or point-to-point communication interfaces and / or protocol (s) such as NV-Link high-speed connections or connection protocols.

In at least one embodiment, one or more parallel processor (s) 2612 a circuit arrangement optimized for graphics and video processing, including, for example, a video output circuit, and represents / represent a graphics processing unit (GPU). In at least one exemplary embodiment, one or more parallel processor (s) contain 2612 Circuitry Optimized for General Purpose Processing. In at least one embodiment, components of the computer system 2600 be integrated with one or more other system elements in a single integrated circuit. For example, in at least one embodiment, one or more parallel processor (s) 2612 , Storage hub 2605 , Processor (s) 2602 and I / O hub 2607 be integrated into an integrated circuit of a system on a chip (SoC). In at least one embodiment, components of the computer system 2600 can be integrated into a single housing to form a System in Package (SIP) configuration. In at least one embodiment, at least some of the components of the computer system 2600 can be integrated into a multi-chip module (MCM), which can be connected to other multi-chip modules to form a modular computer system.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in Graphics Core 2500 for inference or prediction of operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

At least one embodiment can be carried out using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a computer system as described above.

PROCESSORS

27A shows a parallel processor 2700 according to at least one embodiment. In at least one embodiment, various components of the parallel processor 2700 can be implemented using one or more integrated circuit devices, such as programmable processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGA). The parallel processor shown in at least one embodiment 2700 is a variant of one or more in 26th shown parallel processor (s) 2612 according to an exemplary embodiment.

In at least one embodiment, the parallel processor comprises 2700 a parallel processing unit 2702 . In at least one embodiment, the parallel processing unit comprises 2702 an I / O base 2704 that enable communication with other devices, including other instances of the parallel processing unit 2702 , made possible. In at least one embodiment, the I / O device 2704 be connected directly to other devices. In at least one embodiment, the I / O unit is 2704 via a hub or switch interface, such as the storage hub 2605 connected to other devices. In at least one embodiment, connections form between the storage hub 2605 and the I / O base 2704 a communication link 2613 . In at least one embodiment, the I / O unit provides 2704 a connection to a host interface 2706 and a memory crossbar 2716 forth, the host interface 2706 Receives commands directed to the performance of processing operations and the memory crossbar 2716 Receives commands directed to performing memory operations.

If, in at least one embodiment, the host interface 2706 via the I / O base 2704 Receiving a command buffer can be the host interface 2706 Work operations to execute these commands on a front end 2708 conduct. In at least one embodiment, the front end is 2708 with a planner 2710 coupled that is configured to send commands or other work items to a processing cluster array 2712 distributed. In at least one embodiment, the planner provides 2710 make sure the processing cluster array 2712 properly configured and in a valid state before starting tasks the processing cluster array 2712 of the processing cluster array 2712 be distributed. In at least one embodiment, the planner is 2710 implemented via firmware logic that runs on a microcontroller. In at least one exemplary embodiment, the is implemented in a microcontroller 2710 configurable to perform complex planning and work-sharing operations in coarse and fine granularity, which enables fast pre-allocation and context switching of threads that are on the processing array 2712 are executed. In at least one embodiment, the host software may have workloads for scheduling on the array 2712 via one of several doorbells for graphics processing. In at least one embodiment, workloads can then be automatically accessed through the processing array 2712 by the logic of the planner 2710 inside a microcontroller that controls the planner 2710 includes, on the processing array 2712 be distributed.

In at least one embodiment, the processing cluster array 2712 include up to "N" processing clusters (e.g. cluster 2714A , Cluster 2714B , up to clusters 2714N) . In at least one embodiment, each cluster can 2714A-2714N of the processing cluster array 2712 running a large number of concurrent threads. In at least one embodiment, the planner can 2710 the clusters 2714A-2714N of the processing cluster array 2712 Assign work using different planning and / or work distribution algorithms that may vary depending on the workload involved in each type of program or calculation. In at least one exemplary embodiment, the planning can be carried out by the planner 2710 be handled dynamically or supported in part by compiler logic during compilation of program logic necessary for execution by the processing cluster arrays 2712 configured. In at least one embodiment, different clusters 2714A-2714N of the processing cluster array 2712 can be assigned to process different types of programs or to perform different types of calculations.

In at least one embodiment, the processing cluster array 2712 can be configured to perform various types of parallel processing operations. In at least one embodiment, the processing cluster is an array 2712 configured to perform general parallel arithmetic operations. In at least one embodiment, the processing cluster array 2712 may include, for example, logic to perform processing tasks including filtering video and / or audio data, performing modeling operations, including physical operations, and performing data transformations.

In at least one embodiment, the processing cluster is an array 2712 configured to perform parallel graphics processing operations. In at least one embodiment, the processing cluster array 2712 additional logic to aid in performing such graphics processing operations including, but not limited to, texture sensing logic to perform texture operations, as well as tessellation logic and other vertex processing logic. In at least one embodiment, the processing cluster array 2712 can be configured to run graphics processing related shading programs such as vertex shading, tessellation shading, geometry shading, and pixel shading. In at least one embodiment, the parallel processing unit 2702 Data from system memory through the I / O base 2704 transferred for processing. In at least one exemplary embodiment, the transferred data can be stored in the on-chip memory (for example in the parallel processor memory) during processing 2722 ) and then written back to system memory.

In at least one embodiment, when using the parallel processing unit 2702 for the execution of a graphic processing of the planner 2710 can be configured to split a processing load into roughly equal tasks to better distribute graphics processing operations across multiple clusters 2714A-2714N of the processing cluster array 2712 to enable. In at least one embodiment, parts of the processing cluster array 2712 can be configured to perform different types of processing. For example, in at least one embodiment, a first part can be configured to perform vertex shading and topology generation, a second part can be configured to perform tessellation and geometry shading, and a third part can be configured to perform Performs pixel shading or other screen space operations to create a rendered image for display. In at least one embodiment, intermediate data received from one or more of the clusters 2714A-2714N generated, are stored in buffers to facilitate the transfer of intermediate data between clusters 2714A-2714N to enable further processing.

In at least one embodiment, the processing cluster array 2712 Processing tasks received through the scheduler 2710 are to be executed, the commands for defining processing tasks from the front end 2708 receives. In at least one embodiment, the processing tasks can include indices of the data to be processed, e.g. surface (patch) data, primitive data, vertex data and / or pixel data, as well as status parameters and commands that define how the data should be processed (e.g. which program is to be executed shall be). In at least one embodiment, the planner can 2710 be configured in such a way that it fetches indices corresponding to the tasks or indices from the front end 2708 receives. In at least one embodiment, the front end 2708 configured to ensure that the processing cluster array 2712 configured to a valid state before initiating any workload specified by incoming command buffers (e.g., batch buffers, push buffers, etc.).

In at least one embodiment, each of one or more instances of the parallel processing unit can 2702 with the parallel processor memory 2722 be coupled. In at least one embodiment, the parallel processor memory 2722 via memory crossbar 2716 Accessible storage requirements from processing cluster array 2712 as well as the I / O base 2704 can receive. In at least one embodiment, via a memory interface 2718 on the parallel processor memory 2722 can be accessed via memory crossbar 2716 . In at least one embodiment, the memory interface 2718 multiple partition units (e.g. partition unit 2720A , Partition unit 2720B , via partition unit 2720N ), each with a part (e.g. memory unit) of the parallel processor memory 2722 can be coupled. In at least one embodiment, a number is partition units 2720A-2720N configured to correspond to a number of storage units such that a first partition unit 2720A a corresponding first storage unit 2724A , a second partition unit 2720B a corresponding storage unit 2724B and an N-th partition unit 2720N a corresponding N-th storage unit 2724N having. In at least one embodiment, a number of partition units 2720A-2720N not equal to a number of storage devices.

In at least one embodiment, the storage units 2724A-2724N Various types of storage devices include dynamic random access memory (DRAM) or graphics random access memory (SGRAM) including graphics double data rate (GDDR) memory. In at least one embodiment, the storage units 2724A-2724N also include 3D stacks, including, but not limited to, high bandwidth memory (HBM). In at least one exemplary embodiment, render targets, such as, for example, frame buffers or texture maps, can be stored via memory units 2724A-2724N be saved away so that the partition units 2720A-2720N Parts of each render target can write in parallel to use the available bandwidth of the parallel processor memory 2722 to use efficiently. In at least one embodiment, a local instance of the parallel processor memory 2722 in favor of a unified memory design that uses system memory in conjunction with local cache memory.

In at least one embodiment, each of the clusters 2714A-2714N of the processing cluster array 2712 Process data stored in one of the storage units 2724A-2724N in parallel processor memory 2722 to be written. In at least one embodiment, the memory crossbar 2716 configured to output from each cluster 2714A-2714N to any partition unit 2720A-2720N or to another cluster 2714A-2714N that can perform additional processing operations on an output. In at least one embodiment, each cluster can 2714A-2714N via the storage matrix 2716 with the memory interface 2718 communicate to read from or write to various external storage devices. In at least one embodiment, the memory crossbar has 2716 a connection to the storage interface 2718 to get to the I / O base 2704 to communicate, as well as a connection to a local instance of the parallel processor memory 2722 , creating processing units within different processing clusters 2714A-2714N communicate with the system memory or another memory that is not local to the parallel processing unit 2702 is. In at least one embodiment, the memory crossbar 2716 Use virtual channels to flow traffic between clusters 2714A-2714N and the partition units 2720A-2720N to separate.

In at least one embodiment, multiple instances of the parallel processing unit 2702 can be provided on a single add-on card, or several add-on cards can be linked together. In at least one embodiment, different instances of the Parallel processing unit 2702 be configured to work together even if the different instances have different numbers of processing cores, different amounts of local parallel processor memory, and / or other configuration differences. For example, in at least one embodiment, some instances of the parallel processing unit 2702 Compared to other instances, include floating point units with higher precision. In at least one embodiment, systems that include one or more instances of the parallel processing unit 2702 or the parallel processor 2700 may be implemented in a variety of configurations and form factors including, but not limited to, desktop, laptop, or handheld personal computers, servers, workstations, game consoles, and / or embedded systems.

27B Figure 3 shows a block diagram of a partition unit 2720 according to at least one embodiment. In at least one embodiment, the partition unit is 2720 an example of one of the partition units 2720A-2720N out 27A . In at least one embodiment, the partition unit comprises 2720 an L2 cache 2721 , a frame buffer interface 2725 and a ROP 2726 (Raster Operations Unit). L2 cache 2721 is a read / write cache configured to perform load and store operations performed by memory crossbar 2716 and ROP 2726 be received. In at least one embodiment, L2 cache read errors and urgent write back requests are detected 2721 for processing at the frame buffer interface 2725 issued. In at least one embodiment, updates can also be made through the frame buffer interface 2725 sent to a frame buffer for processing. In at least one embodiment, is the frame buffer interface 2725 connected to one of the storage units in the parallel processor memory, for example to the storage units 2724A-2724N of 27 (e.g. in the parallel processor memory 2722 ).

In at least one embodiment, it is ROP 2726 a processing unit that performs raster operations such as stencil, z-test, blending, and the like. In at least one embodiment, there is ROP 2726 then processed graphics data that is stored in graphics memory. In at least one embodiment, ROP 2726 compression logic for compressing depth or color data written to memory and decompressing depth or color data read from memory. In at least one embodiment, the compression logic can be lossless compression logic that uses one or more of several compression algorithms. The type of compression that ROP 2726 performed may vary based on statistical properties of the data being compressed. For example, in at least one embodiment, the delta color compression is performed on depth and color data on a per-ktil basis.

In at least one embodiment, it is ROP 2726 in each processing cluster (e.g. cluster 2714A-2714N of 27 ) instead of in the partition unit 2720 contain. In at least one embodiment, read and write requests for pixel data are made via the memory crossbar 2716 transmitted instead of pixel fragment data. In at least one exemplary embodiment, processed graphic data can be displayed on a display device, for example on one or more display devices 2610 out 26th , for further processing by processor (s) 2602 be forwarded or for further processing by one of the processing units within the parallel processor 2700 out 27A to get redirected.

27C Figure 3 shows a block diagram of a processing cluster 2714 within a parallel processing unit, according to at least one embodiment. In at least one embodiment, a processing cluster is an instance of one of the processing clusters 2714A-2714N of 27 . In at least one embodiment, the processing cluster 2714 can be configured to run many threads in parallel, where the term “thread” refers to an instance of a particular program that is executing on a particular set of input data. In at least one embodiment, single instruction (multiple data) instruction issue techniques are used to support parallel execution of a large number of threads without providing multiple independent instruction units. In at least one embodiment, single-instruction multiple-thread (SIMT) techniques are used to support the parallel execution of large numbers of generally synchronized threads using a common instruction unit configured to send instructions to a set of Processing machines within each processing cluster outputs.

In at least one embodiment, the operation of the processing cluster 2714 through a pipeline manager 2732 which distributes the processing tasks to SIMT parallel processors. In at least one embodiment, the pipeline manager receives 2732 Orders from the planner 2710 of 27 and manages the execution of these commands via a graphics multiprocessor 2734 and/ or a texture unit 2736 . In at least one embodiment, the graphics processor is multiprocessor 2734 an exemplary case of a SIMT parallel processor. However, in at least one embodiment, different types of SIMT parallel processors of different architectures can be in processing clusters 2714 be included. In at least one embodiment, one or more instances of the graphics multiprocessor 2734 in a processing cluster 2714 be included. In at least one embodiment, the graphics multiprocessor 2734 Process data, and a data crossbar 2740 can be used to distribute processed data to one of several possible destinations, including other shading units. In at least one embodiment, the pipeline manager 2732 facilitate the distribution of processed data by specifying destinations for processed data that will be passed through the data crossbar 2740 should be distributed.

In at least one embodiment, each graphics multiprocessor can 2734 within the processing cluster 2714 comprise an identical set of functional execution logic (e.g., arithmetic logic units, load storage units). In at least one embodiment, the functional execution logic can be configured in a pipelined manner in which new instructions can be issued before the previous instructions are completed. In at least one embodiment, the functional execution logic supports a variety of operations including integer and floating point arithmetic, comparison operations, Boolean operations, bit shifting, and the computation of various algebraic functions. In at least one embodiment, the same hardware can be used with functional units to perform different operations, and any combination of functional units can be present.

In at least one embodiment, the an form processing clusters 2714 Submitted commands a thread. In at least one embodiment, a set of threads executing on a set of parallel processing engines form a thread group. In at least one embodiment, a thread group forms a program with various input data. In at least one embodiment, each thread within a thread group of a different processing engine within a graphics multiprocessor 2734 be assigned. In at least one embodiment, a thread group can include fewer threads than a number of processing engines within a graphics multiprocessor 2734 . In at least one embodiment, when a thread group includes fewer threads than a number of processing engines, one or more of the processing engines may be idle during the cycles that thread group is being processed. In at least one embodiment, a thread group can also have more threads than a number of processing engines within the graphics multiprocessor 2734 include. If, in at least one embodiment, a thread group has more threads than the number of processing engines within the graphics multiprocessor 2734 includes, the processing can be performed over successive clock cycles in at least one embodiment. In at least one embodiment, multiple thread groups can run concurrently on a graphics multiprocessor 2734 are executed.

In at least one embodiment, the graphics multiprocessor comprises 2734 an internal cache memory for performing load and store operations. In at least one embodiment, the graphics multiprocessor 2734 dispense with an internal cache memory and a cache memory (eg L1 cache 2748) within the processing cluster 2714 use. In at least one embodiment, each graphics processor has a multiprocessor 2734 also access to L2 caches within partition units (e.g. partition units 2720A-2720N of 27 ) by all processing clusters 2714 shared and used to transfer data between threads. In at least one embodiment, the graphics multiprocessor 2734 also access global off-chip memory, which may include one or more local parallel processor memories and / or system memories. In at least one embodiment, each memory can be external to the parallel processing unit 2702 can be used as global storage. In at least one embodiment, the processing cluster comprises 2714 multiple instances of graphics multiprocessor 2734 can share common instructions and data that is in the L1 cache 2748 can be saved.

In at least one embodiment, each processing cluster 2714 an MMU 2745 (Storage Management Unit) configured to map virtual addresses to physical addresses. In at least one embodiment, one or more instances of the MMU 2745 within the storage interface 2718 of 27 are located. In at least one embodiment, MMU 2745 a set of Page Table Entries (PTEs) used to map a virtual address to a physical address of a tile (more about tiling) and optionally a cache line index. In at least one embodiment, MMU 2745 Address Translation lookaside buffers (TLB) or caches reside in a graphics multiprocessor 2734 or L1 cache or processing cluster 2714 can be located. In at least one embodiment, the physical address is processed to distribute the surface data access location to allow efficient interleaving of requests between partition units. In at least one embodiment, the cache line index can be used to determine whether a request for a cache line is a hit or a miss.

In at least one embodiment, a processing cluster 2714 be configured so that any graphics multiprocessor 2734 with a texture unit 2736 is coupled to perform texture mapping operations such as determining texture sample positions, reading texture data, and filtering texture data. In at least one embodiment, texture data is obtained from an internal texture L1 cache (not shown) or from an L1 cache within the graphics multiprocessor 2734 read and fetched from an L2 cache, the local parallel processor memory or the system memory as required. In at least one embodiment, each graphics multiprocessor is 2734 processed tasks to the data crossbar 2740 to transfer processed tasks to another processing cluster 2714 to make available for further processing or to process processed tasks via the memory crossbar 2716 to be stored in an L2 cache, in local parallel processor memory, or in system memory. In at least one embodiment, preROP 2742 (Pre-Raster Operations Unit) configured to receive data from the graphics multiprocessor 2734 receives and forwards data to ROP units which, as described here, can be located at partition units (e.g. partition units 2720A-2720N of 27 ). In at least one embodiment, the unit PreROP 2742 Carry out optimizations for the color mixture, organize pixel color data and carry out address translations.

At least one embodiment can be carried out using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in the system 17C for inference or prediction of operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

27D shows a graphics multiprocessor 2734 according to at least one embodiment. In at least one embodiment, the graphics processor is multiprocessor 2734 with the pipeline manager 2732 of the processing cluster 2714 coupled. In at least one embodiment, the graphics multiprocessor has 2734 via an execution pipeline that has an instruction cache 2752 , a command unit 2754 , an address allocation unit 2756 , a register file 2758 , one or more General Purpose Graphics Processing Unit (GPGPU) cores 2762 and one or more load / store units 2766 includes, but is not limited to. GPGPU cores 2762 and load / storage units 2766 are via a memory and cache connection 2768 with the cache memory 2772 and the shared memory 2770 coupled.

In at least one embodiment, the instruction cache receives 2752 a stream of instructions to be executed from the pipeline manager 2732 . In at least one embodiment, instructions are in the instruction cache 2752 cached and for execution by the command unit 2754 distributed. In at least one embodiment, the command unit 2754 Send commands as thread groups (e.g. warps), with each thread belonging to the thread group of a different execution unit within the GPGPU core 2762 assigned. In at least one embodiment, an instruction can access any local, shared, or global address space by specifying an address within a uniform address space. In at least one embodiment, the address allocation unit 2756 can be used to translate addresses in a uniform address space into a unique memory address to that of the load / storage units 2766 can be accessed.

In at least one embodiment, is in the register file 2758 a set of registers for functional units of the graphics multiprocessor 2734 intended. In at least one embodiment, is in the register file 2758 a temporary memory is provided for operands that are linked to data paths from functional units (e.g. GPGPU cores 2762 , Load / storage units 2766 ) of the graphics multiprocessor 2734 are connected. In at least one embodiment, the register file is 2758 distributed to each of the functional units in such a way that each functional unit has a dedicated part of the register file 2758 assigned. In at least one embodiment, the register file is 2758 split between different warps, which are generated by the graphics multiprocessor 2734 are executed.

In at least one embodiment, GPGPU cores 2762 each include floating point units (FPUs) and / or integer arithmetic logic units (ALUs) which are used to execute commands of the graphics multiprocessor 2734 be used. GPGPU cores 2762 can be similar in architecture or differ in architecture. In at least one embodiment, a first portion comprises the GPGPU cores 2762 a single precision FPU and an integer ALU, while a second part of the GPGPU cores comprises a double precision FPU. In at least one embodiment, FPUs can implement the IEEE 754-2008 standard for floating point arithmetic or enable floating point arithmetic with variable precision. In at least one embodiment, the graphics multiprocessor 2734 additionally comprise one or more fixed function (s) or special functional units in order to carry out certain functions such as copying rectangles or operations for blending pixels. In at least one embodiment, one or more GPGPU cores can also include fixed or special functional logic.

In at least one embodiment, the include GPGPU cores 2762 SIMD logic capable of executing a single instruction on multiple sets of data. In at least one embodiment, GPGPU cores 2762 physically execute SIMD4, SIMD8 and SIMD16 commands and logically execute SIMD1, SIMD2 and SIMD32 commands. In at least one embodiment, SIMD instructions for GPGPU cores may be generated at compile time by a shading compiler or automatically generated when executing programs written and compiled for SPMD (Single Program Multiple Data) or SIMT architectures. In at least one embodiment, multiple threads of a program configured for a SIMT execution model can be executed via a single SIMD instruction. For example, in at least one embodiment, eight SIMT threads executing the same or similar operations can be executed in parallel via a single SIMD8 logic unit.

In at least one embodiment, the memory and cache connection is 2768 an interconnection network that connects each functional unit of the graphics multiprocessor 2734 with the register file 2758 and with the shared memory 2770 connects. In at least one embodiment, the memory and cache connection is 2768 a crossbar connection that connects the load / store unit 2766 enables load and store operations between the shared memory 2770 and the register file 2758 to implement. In at least one embodiment, the register file 2758 at the same frequency as the GPGPU cores 2762 operated so that the data transfer between the GPGPU cores 2762 and the register file 2758 done with very low latency. In at least one embodiment, the shared memory 2770 used to enable inter-thread communication on functional units within the graphics multiprocessor 2734 are executed. In at least one embodiment, the cache memory 2772 can be used as a data cache, for example to store texture data between functional units and the texture unit 2736 buffer. In at least one embodiment, the shared memory 2770 can also be used as a program-managed cache. In at least one embodiment, threads that run on GPGPU cores 2762 run, programmatically store data in shared memory in addition to automatically cached data that is in cache memory 2772 are stored.

In at least one embodiment, a parallel processor or GPGPU as described herein is communicatively coupled to host / processor cores for graphics operations, machine learning, pattern analysis operations, and various general purpose GPU (GPGPU) functions accelerate. In at least one embodiment, the GPU may be communicatively coupled to the host processor / cores via a bus or other connection (e.g., a high speed link such as PCIe or NVLink). In at least one embodiment, the GPU may be integrated in the same package or chip as the cores and communicatively coupled to the cores via an internal processor bus / connection (i.e., internal to the package or chip). In at least one embodiment, regardless of how the GPU is connected, the processor cores can assign work to the GPU in the form of sequences of commands contained in a work descriptor. In at least one embodiment, the GPU then uses dedicated circuitry / logic to efficiently process these commands / instructions.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in the system 17C for inference or prediction of operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

At least one embodiment can be carried out using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

28 shows a multi-GPU computer system 2800 according to at least one embodiment. In at least one embodiment, the multi-GPU computer system 2800 a processor 2802 Include that through a host interface switch 2804 with multiple general purpose graphics processing units (GPGPUs) 2806A-D is coupled. In at least one embodiment, the host is an interface switch 2804 a PCI Express switch device that runs the processor 2802 with a PCI-Express bus through which the processor 2802 with the GPGPUs 2806A-D can communicate. The GPGPUs 2806A-D can have a set of high-speed point-to-point connections between the GPUs 2816 communicate with each other. In at least one embodiment, make GPU-to-GPU connections 2816 to each of the GPGPUs 2806A-D via a dedicated GPU connection. In at least one embodiment, P2P GPU connections 2816 enable direct communication between the individual GPGPUs 2806A-D without any communication over the host interface bus 2804 to which the processor 2802 connected is required. In at least one embodiment where the GPU-to-GPU traffic is routed to the P2P-GPU links 2816, the host interface bus remains 2804 for access to the system memory or for communication with other instances of the multi-GPU computing system 2800 , for example via one or more network devices. While in at least one embodiment the GPGPUs 2806A-D over the switch 2804 the host interface with the processor 2802 are connected, includes the processor 2802 In at least one embodiment, direct support for P2P GPU links 2816 and can be directly with the GPGPUs 2806A-D get connected.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in the system 17C for inference or prediction of operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

At least one embodiment can be carried out using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a GPU as described above.

29 Figure 3 is a block diagram of a graphics processor 2900 , according to at least one embodiment. In at least one embodiment, the graphics processor comprises 2900 a ring interconnect 2902 , a pipeline front end 2904 , the engine 2937 and the graphics cores 2980A-2980N . In at least one embodiment, the ring connection couples 2902 the graphics processor 2900 with other units, including other graphics processors or one or more general processor cores. In at least one embodiment, the graphics processor is 2900 one of many processors integrated into a multi-core processing system.

In at least one embodiment, the graphics processor receives 2900 Batch of commands over the ring connection 2902 . In at least one embodiment, incoming commands are from a command streamer 2903 in the pipeline front end 2904 interpreted. In at least one embodiment, the graphics processor comprises 2900 a scalable execution logic for the implementation of 3D geometry processing and media processing by means of graphics core (s) 2980A-2980N . In at least one embodiment for commands for 3D geometry processing, Command Streamer delivers 2903 Commands to the geometry pipeline 2936 . In at least one embodiment, the command streamer delivers 2903 for at least some media processing commands to a video front end 2934 , the one with the media engine 2937 is coupled. In at least one embodiment, the media engine comprises 2937 a video quality engine (VQE) 2930 for video and image post-processing and a multi-format encoding / decoding engine (MFX) 2933 to provide hardware accelerated encoding and decoding of media data. In at least one embodiment, create the geometry pipeline 2936 and the media engine 2937 each execution thread for thread execution resources used by at least one graphics core 2980A to be provided.

In at least one embodiment, the graphics processor comprises 2900 scalable thread execution resources with modular cores 2980A-2980N (sometimes referred to as core sections), each of which has multiple sub-cores 2950A-550N , 2960A-2960N (sometimes referred to as core subsections). In at least one embodiment, the graphics processor 2900 over any number of graphics cores 2980A to 2980N feature. In at least one embodiment, the graphics processor comprises 2900 a graphics core 2980A with at least one first lower core 2950A and a second sub-core 2960A . In at least one embodiment, the graphics processor is 2900 a low performance processor with a single sub-core (e.g. 2950A). In at least one embodiment, the graphics processor comprises 2900 multiple graphics cores 2980A-2980N , each of which has a set of first sub-ces 2950A-2950N and a set of second sub-ces 2960A-2960N includes. In at least one embodiment, each sub-core includes in the first sub-core 2950A-2950N at least a first set of execution units 2952A-2952N and media / texture scanners 2954A-2954N . In at least one embodiment, each sub-core includes the second sub-core 2960A-2960N at least a second set of execution units 2962A-2962N and scanners 2964A-2964N . In at least one embodiment, each sub-core is shared 2950A-2950N , 2960A-2960N a set of shared resources 2970A-2970N . In at least one embodiment, shared resources include shared cache memory and pixel operation logic.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the inference and / or training logic 1415 in the system 17C for inference or prediction of operations based, at least in part, on weight parameters calculated using training operations with neural networks, functions and / or architectures of neural networks or use cases of neural networks described herein.

At least one embodiment can be carried out using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a GPU as described above.

30th Figure 3 is a block diagram showing a microarchitecture for a processor 3000 shows, which may include logic circuits for executing instructions, according to at least one embodiment. In at least one embodiment, the processor 3000 Execute instructions, including x86 instructions, ARM instructions, specialized instructions for application specific integrated circuits (ASICs), etc. In at least one embodiment, the processor may 3010 Packed data storage registers include, for example, 64-bit wide MMX ™ registers in microprocessors operated with MMX technology from Intel Corporation of Santa Clara, California. In at least one embodiment, MMX registers, available in both integer and floating point form, can operate with packed data elements involving single instructions, multiple data (“SIMD”), and streaming SIMD extensions (“SSE”). In at least one embodiment, 128-bit wide XMM registers that relate to the SSE2, SSE3, SSE4, AVX or beyond (commonly referred to as “SSEx”) technology can contain such packed data operands. In at least one embodiment, the processor 3010 Execute instructions to accelerate machine learning or deep learning algorithms, training or inference.

In at least one embodiment, the processor comprises 3000 an in-order front end (“front end”) 3001 to fetch instructions to be executed and to prepare instructions to be used later in the processor pipeline. In at least one embodiment, the front end 3001 comprise several units. In at least one embodiment, an instruction prefetcher fetches 3026 Instructions from memory and executes instructions to an instruction decoder 3028 to, who in turn decodes or interprets commands. For example, in at least one embodiment, the instruction decoder decodes 3028 a received instruction into one or more operations known as “microinstructions” or “microoperations” (also known as “micro-ops” or “uops”) that can be performed by this machine. In at least one embodiment, the instruction decoder decomposes 3028 an instruction in an opcode and corresponding data and control fields that can be used by the microarchitecture to perform operations according to at least one embodiment. In at least one exemplary embodiment, a trace cache 3030 can store decoded uops in program-ordered sequences or traces in a uop queue 3034 assemble for execution. When the sequence cache 3030 encounters a complex instruction in at least one embodiment, a microcode ROM 3032 the uops required to complete the operation.

In at least one embodiment, some instructions can be converted into a single micro-operation while others require multiple micro-operations to perform a full operation. In at least one embodiment, the instruction decoder can 3028 on the microcode ROM 3032 access to execute an instruction when it takes more than four micro-operations to execute an instruction. In at least one embodiment, an instruction can be processed in the instruction decoder 3028 can be decoded into a small number of micro-ops. In at least one embodiment, an instruction can be in microcode ROM 3032 stored in case a number of micro-ops are required to perform the operation. In at least one embodiment, the trace cache 3030 relates to an entry point programmable logic array (“PLA”) for determining a correct microinstruction pointer for reading microcode sequences to complete one or more instructions from the Microcode ROM 3032 , according to at least one embodiment. In at least one embodiment, the front end 3001 the engine after the microcode ROM 3032 has finished sequencing micro-ops for an instruction, fetching micro-ops from the sequence cache 3030 resume.

In at least one exemplary embodiment, the out-of-order execution engine (“out-of-order engine”, ie execution in a different order than in the program code) 3003 can prepare commands for execution. In at least one embodiment, the execution logic for out-of-order execution has a number of buffers to smooth and reorder the flow of instructions to optimize performance when the pipeline is shut down and execution is scheduled. The out-of-order execution engine 3003 includes, without limitation, an allocator / register renamer 3040 , a storage uop queue 3042, an integer / floating point uop queue 3044 , a storage planner 3046 , a quick planner 3002 , a slow / general floating point planner ("slow / general FP planner") 3004, and a simple floating point planner ("simple FP planner") 3006. In at least one embodiment, the fast planner 3002 , the slow / general floating point planner 3004 and the simple floating point planner 3006 also referred to collectively as “uop planner 3002, 3004, 3006”. The allocator / register renamer 3040 allocates machine buffers and resources that each uop needs to run. In at least one embodiment, the allocator / register renamer names 3040 logical registers to entries in a register file. In at least one embodiment, the allocator / register renamer allocates 3040 also an entry for each uop in one of two uop queues, memory uop queue 3042 for memory operations and integer / floating point op queue 3044 for non-memory operations, prior to the memory scheduler 3046 and uop schedulers 3002, 3004, 3006. In at least one embodiment, uop schedulers 3002, 3004, 3006 determine when based on the readiness of their dependent input register operand sources and the availability of execution resources that uops need to complete their operations a uop is ready to run. In at least one embodiment, the quick planner can 3002 of at least one embodiment, schedule on each half of the main clock cycle while the slow / general floating point scheduler 3004 and the simple floating point planner 3006 can schedule once per main processor clock cycle. In at least one embodiment, uop planners 3002, 3004, and 3006 may arbitrate for distribution ports to schedule uops for execution.

In at least one embodiment, execution block b11 comprises, without limitation, an integer register file / bypass network 3008, a floating point register file / bypass network (“FP register file / bypass network”) 3010, address generation units (“ AGUs ”) 3012 and 3014, fast arithmetic and logical units (ALUs) (“ fast ALUs ”) 3016 and 3018, one slow arithmetic and logical unit (“ slow ALU ”) 3020, one floating point ALU (“ FP ”) 3022 and a floating point ("FP") move unit 3024. In at least one embodiment, the integer register file / bypass network 3008 and the floating point register file / bypass network 3010 also referred to here as “register files 3008, 3010”. In at least one embodiment, the AGUS 3012 and 3014 , the fast ALUs 3016 and 3018 who have favourited slow ALU 3020 who have favourited Floating-point ALU 3022 and the floating point movement unit 3024 Also referred to herein as integer register file / bypass network as "execution units 3012, 3014, 3016, 3018, 3020, 3022, and 3024". In at least one embodiment, execution block b11 can include, without limitation, any number (including zero) and type of register files, bypass networks, address generation units, and execution units in any combination.

In at least one embodiment, the register files 3008 , 3010 between the uop planners 3002, 3004, 3006 and the execution units 3012 , 3014 , 3016 , 3018 , 3020 , 3022 and 3024 to be ordered. In at least one embodiment, the register file / bypass network 3008 integer operations. In at least one embodiment, floating point register file / bypass network performs 3010 Floating point operations. In at least one embodiment, each of the register files 3008 , 3010 include, without limitation, a bypass network that can bypass newly completed results that have not yet been written to the register file or forward them to new dependent uops. In at least one embodiment, the register files 3008 , 3010 Exchange data with each other. In at least one embodiment, the register file / bypass network 3008 include, without limitation, two separate register files, a register file for lower order 32 data bits and a second register file for higher order 32 data bits. In at least one embodiment, the floating point register file / bypass network 3010 without restriction 128 Entries are bit wide, because floating point instructions usually have operands that are 64 to 128 bits wide.

In at least one exemplary embodiment, the execution units 3012 , 3014 , 3016 , 3018 , 3020 , 3022 , 3024 Execute commands. In at least one embodiment, the register files store 3008 , 3010 integer and floating point data operand values that microinstructions must execute. In at least one embodiment, the processor 3000 any number and combination of execution units without restriction 3012 , 3014 , 3016 , 3018 , 3020 , 3022 , 3024 include. In at least one embodiment, the floating point ALU 3022 and the floating point movement unit 3024 Perform floating point, MMX, SIMD, AVX, and SSE or other operations including special machine learning instructions. In at least one embodiment, the floating point ALU 3022 include, without limitation, a 64-bit x64-bit floating point divider for performing division, square root, and remainder micro-operations. In at least one embodiment, instructions containing a floating point value can be processed using floating point hardware. In at least one embodiment, ALU operations can be performed on fast ALUs 3016 , 3018 be handed over. In at least one embodiment, the fast ALUs 3016 , 3018 perform fast operations with an effective latency of half a clock cycle. In at least one embodiment, the most complex integer operations go to the slow ALU 3020 , since the slow ALU 3020 may include, without limitation, integer execution hardware for long latency operations, such as a multiplier, shifts, flag logic, and branch processing. In at least one embodiment, AGUS 3012 , 3014 are executed. In at least one embodiment, the fast ALU 3016 , fast ALU 3018 and slow ALU 3020 Perform integer operations on 64-bit data operands. In at least one embodiment, the fast ALU 3016 , fast ALU 3018 and slow ALU 3020 can be implemented to support a variety of data bit sizes including sixteen, thirty-two, 128, 256, etc. In at least one embodiment, the floating point ALU 3022 and the floating point movement unit 3024 can be implemented to support a range of operands with bits of different widths. In at least one embodiment, the floating point ALU 3022 and the floating point movement unit 3024 work with 128-bit packed data operands in conjunction with SIMD and multimedia commands.

In at least one embodiment, the uop planners 3002, 3004, 3006 schedule dependent operations before the higher-level load has finished executing. Because uops speculatively planned and in processor 3000 can be executed by the processor 3000 also include, in at least one embodiment, logic for handling failed attempts in memory. In at least one embodiment, if there is a failure to load data in the data cache, there may be dependent operations in the pipeline during execution that have exited the scheduler with temporarily bad data. In at least one embodiment, a retry mechanism tracks and re-executes instructions that use incorrect data. In at least one embodiment, it may be necessary to rerun dependent operations and allow independent operations to complete. In at least one embodiment, the scheduler and iteration mechanism of at least one embodiment of a processor can also be designed to capture instruction sequences for text string comparison operations.

In at least one embodiment, the term “register” can refer to memory locations in the processor that can be used as part of instructions to identify operands. In at least one embodiment, registers can be ones that can be used from outside the processor (from a programmer's point of view). In at least one embodiment, registers may not be limited to any particular type of circuit. Rather, in at least one exemplary embodiment, a register can store data, provide data and carry out the functions described here. In at least one embodiment, the registers described herein may be implemented by circuitry within a processor using any number of different techniques, such as dedicated physical registers, dynamically allocated physical registers using register renaming, combinations of dedicated and dynamically allocated physical registers, etc. In At least one embodiment, integer registers store 32-bit integer data. The register file of at least one embodiment also contains eight multimedia SIMD registers for packed data.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, parts of the inference and / or training logic 1415 or all of the inference and / or training logic 1415 in EXE block 3011 and other memories or registers, shown or not shown. For example, in at least one exemplary embodiment, the training and / or inference techniques described here can be one or more of those in the EXE block 3011 Use the ALUs shown. In addition, weighting parameters can be stored in on-chip or off-chip memory and / or in registers (shown or not shown), the ALUs of the EXE block 3011 configure so that they can execute one or more machine learning algorithms, neural network architectures, use cases or training techniques described here.

At least one embodiment can be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a GPU as described above.

31 shows a deep learning application processor 3100 according to at least one embodiment. In at least one embodiment, the deep learning application processor uses 3100 Commands that when sent by the deep learning application processor 3100 running, the deep learning application processor 3100 cause some or all of the processes and techniques described in this disclosure to be carried out. In at least one embodiment, the deep learning is an application processor 3100 an application specific integrated circuit (ASIC). In at least one embodiment, the application processor performs 3100 Matrix multiplication operations that are either "hardwired" into hardware or performed as the result of executing one or more instructions, or both. In at least one embodiment, the deep learning application processor comprises 3100 without restriction processing cluster 3110 (1) -3110 (12), inter-chip connections ("ICLs") 3120 (1) -3120 (12), inter-chip controllers ("ICCs") 3130 (1) -3130 ( 2), second generation high-bandwidth memory ("HBM2") 3140 (1) -3140 (4), memory control ("Mem Ctrlrs") 3142 (1) -3142 (4), a physical layer of high-bandwidth memory (“HBM PHY”) 3144 (1) -3144 (4), a management controller central processing unit (“management control CPU”) 3150, a serial peripheral interface, inter-integrated circuit - and General Purpose Input / Output Block ("SPI, I2C, GPIO") 3160, a peripheral component connection express control and Direct Memory Access Block ("PCIe Controller and DMA") 3170 and a sixteen-lane Peripheral Component Connection Express Port ("PCI Express × 16") 3180.

In at least one embodiment, processing clusters 3110 Perform deep learning operations, including inference or prediction operations based on weight parameters calculated by one or more training techniques, including those described here. In at least one embodiment, each processing cluster 3110 include, without limitation, any number and type of processors. In at least one Embodiment can be the deep learning application processor 3100 any number and type of processing clusters 3100 include. In at least one embodiment, the inter-chip connections are 3120 bidirectional. In at least one embodiment, enable inter-chip connections 3120 and inter-chip controllers 3130 multiple deep learning application processors 3100 the exchange of information, including activation information, resulting from the execution of one or more machine learning algorithms contained in one or more neural networks. In at least one embodiment, the deep learning application processor 3100 any number (including zero) and type of ICLs 3120 and ICCs 3130 include.

In at least one exemplary embodiment, the HBM2s 3140 are provided with a total of 31 gigabytes (GB) of memory. HBM2 3140 (i) is both the controller 3142 (i) as well as the HBM PHY 3144 (i) assigned. In at least one embodiment, any number of HBM2s 3140 can provide any type and total amount of high bandwidth memory, and can work with any number (including zero) and type of memory controllers 3142 and HBM PHYs 3144 be associated. In at least one embodiment, SPI, I2C, GPIO 3160 , PCIe control and DMA 3170 and / or PCIe 3180 can be replaced by any number and type of blocks that enable any number and type of communication standards in any technically suitable manner.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, the deep learning application processor is used to train a machine learning model, such as a neural network, to predict or infer information that is sent to the deep learning application processor 3100 to be provided. In at least one embodiment, the deep learning application processor is 3100 used to infer or predict information based on a trained machine learning model (e.g., a neural network), to be transmitted by another processor or system, or by the deep learning application processor 3100 were trained. In at least one embodiment, the processor 3100 can be used to perform one or more of the neural network use cases described here.

At least one embodiment can be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a GPU as described above.

32 Figure 3 is a block diagram of a neuromorphic processor 3200 , according to at least one embodiment. In at least one embodiment, the neuromorphic processor can 3200 one or more inputs from sources external to the neuromorphic processor 3200 receive. In at least one embodiment, these inputs can be sent to one or more neurons 3202 inside the neuromorphic processor 3200 be transmitted. In at least one embodiment, the neurons can 3202 and their components are implemented using circuitry or logic comprising one or more arithmetic and logic units (ALUs). In at least one embodiment, the neuromorphic processor can 3200 without limitation, thousands or millions of instances of the neurons 3202 but it can include any suitable number of neurons 3202 be used. In at least one embodiment, each instance of the neuron 3202 a neuron input 3204 and a neuron output 3206 include. In at least one embodiment, the neurons can 3202 Generate outputs based on inputs from other instances of the neurons 3202 can be transferred. For example, in at least one embodiment, the neuron inputs 3204 and the neuron outputs 3206 via the synapses 3208 be connected to each other.

In at least one embodiment, the neurons can 3202 and the synapses 3208 so interconnected that the neuromorphic processor 3200 so that it works by the neuromorphic processor 3200 information received is processed or analyzed. In at least one embodiment, the neurons can 3202 send out an output impulse (or "fire" or "spike") when via the neuron input 3204 Input received exceeds a threshold. In at least one embodiment, the neurons can 3202 those at the neuron inputs 3204 sum or integrate received signals. For example, in at least one embodiment, the neurons 3202 implemented as leaky-integrate-and-fire neurons, where when a sum (referred to as a "membrane potential") exceeds a threshold, the neuron 3202 generate an output (or "fire") using a transfer function such as a sigmoid or threshold function. In at least one exemplary embodiment, a leaky integrate and fire neuron can transmit those at the neuron inputs 3204 Add the received signals to a membrane potential and also apply a decay factor (or “leak”) to reduce a membrane potential. In at least one exemplary embodiment, a leaky integrate-and-fire neuron can fire if several input signals are present at the neuron inputs 3204 received fast enough to cross a threshold (i.e. before a membrane potential dies away too low to fire). In at least one embodiment, the neurons can 3202 implemented by circuits or logics that receive inputs, inputs into a Integrate membrane potential and let a membrane potential subside. In at least one embodiment, the inputs can be averaged, or any other suitable transfer function can be used. In addition, the neurons can 3202 in at least one exemplary embodiment comprise, without limitation, comparator circuits or logics which have an output peak at the neuron output 3206 generate when the result of applying a transfer function to the neuron input 3204 exceeds a threshold. In at least one embodiment, the neuron 3202 As soon as it has ignited, it ignores input information received previously, for example by resetting a membrane potential to 0 or another suitable standard value. In at least one embodiment, the neuron 3202 as soon as the membrane potential has been reset to 0, resume normal operation after an appropriate period of time (or refractory period).

In at least one embodiment, the neurons can 3202 via the synapses 3208 be connected to each other. In at least one embodiment, the synapses 3208 function in such a way that they receive signals from an output of a first neuron 3202 to an input of a second neuron 3202 transfer. In at least one embodiment, the neurons can 3202 Information about more than one instance of the synapse 3208 transfer. In at least one embodiment, one or more instances of the neuron output 3206 via an instance of the synapse 3208 with an instance of the neuron input 3204 in the same neuron 3202 be connected. In at least one embodiment, an instance of the neuron 3202 that produces an output that is via an instance of the synapse 3208 should be transmitted in relation to this instance of the synapse 3208 called a “presynaptic neuron”. In at least one embodiment, a neuron 3202 receiving an input via a synapse 3208 is transmitted as a "postsynaptic neuron" in relation to this synapse 3208 are designated. As an instance of the neuron 3202 Input from one or more instances of the synapse 3208 can receive and also outputs via one or more instances of the synapse 3208 can send a single instance of the neuron 3202 hence both a “presynaptic neuron” and a “postsynaptic neuron” in relation to different instances of the synapse 3208 be in at least one embodiment.

In at least one embodiment, the neurons can 3202 be organized in one or more layers. Every instance of a neuron 3202 can have a neuron output 3206 have one or more synapses 3208 to one or more neuron inputs 3204 can fan out. In at least one embodiment, the neuron outputs 3206 of neurons 3202 in a first shift 3210 with the neuron inputs 3204 of neurons 3202 in a second shift 3212 be connected. In at least one embodiment, the layer 3210 referred to as the “feed-forward layer”. In at least one embodiment, each instance of the neuron 3202 in an instance of the first layer 3210 to each instance of the neuron 3202 in the second shift 3212 fan out. In at least one embodiment, the first layer 3210 referred to as the "fully connected feed-forward layer". In at least one embodiment, each instance of a neuron can 3202 in an instance of the second layer 3212 on fewer than all instances of the neuron 3202 in a third shift 3214 fan out. In at least one embodiment, the second layer can 3212 referred to as the "sparsely connected feed-forward layer". In at least one embodiment, the neurons can 3202 in the second shift 3212 on the neurons 3202 fan out into several other layers, including the neurons 3202 in the (same) second shift 3212 . In at least one embodiment, the second layer can 3212 referred to as a "recurring shift". The neuromorphic processor 3200 may include, without limitation, any suitable combination of recurring layers and feed-forward layers, including, without limitation, both sparsely connected feed-forward layers and fully connected feed-forward layers.

In at least one embodiment, the neuromorphic processor can 3200 without limitation a reconfigurable connection architecture or dedicated hardwired connections to connect the synapse 3208 with the neurons 3202 include. In at least one embodiment, the neuromorphic processor can 3200 comprise, without limitation, a circuit arrangement or logic which makes it possible to connect synapses to different neurons as required 3202 based on the topology of the neural network and the neuron fan-in / out. For example, in at least one embodiment, the synapses 3208 with the neurons 3202 can be connected using an interconnect fabric, such as network-on-chip, or with dedicated links. In at least one embodiment, synapse connections and their components can be implemented by circuitry or logic.

At least one embodiment can be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a GPU as described above.

33 Figure 3 is a block diagram of a processing system, in accordance with at least one embodiment. In at least one embodiment, the system includes 3300 one or more processors 3302 and one or more graphics processors 3308 and can be a single processor desktop system, a multiprocessor workstation system, or a server system with a large number of processors 3302 or processor cores 3307 be. In at least one embodiment, is system 3300 a processing platform embedded in a system-on-a-chip (SoC) integrated circuit for use in mobile, portable, or embedded devices.

In at least one embodiment, the system 3300 a game console, including a game and media console, a mobile game console, a handheld game console, or an online game console, or be integrated with a server-based game platform. In at least one embodiment, the system is 3300 a mobile phone, smartphone, tablet computing device or mobile internet device. In at least one embodiment, the processing system 3300 also comprise a wearable device, be coupled to such a device or be integrated into such a device, such as a wearable smart watch device, a smart eyewear device, an augmented reality device or a virtual reality device. In at least one embodiment, the processing system is 3300 a television or set-top box device with one or more processors 3302 and a graphical interface provided by one or more graphics processors 3308 is produced.

In at least one embodiment, include one or more processors 3302 one or more processor cores each 3307 to process commands that, when executed, perform operations for system and user software. In at least one embodiment, each is one or more processor cores 3307 configured to have a specific set of commands 3309 processed. In at least one embodiment, the instruction set 3309 a calculation with a complex instruction set (Complex Instruction Set Computing, CISC), a calculation with a reduced instruction set (Reduced Instruction Set Computing, RISC) or a calculation by means of a very long instruction word (Very Long Instruction Word, VLIW). In at least one embodiment, the processor cores 3307 each have a different instruction set 3309 process, which may include commands to facilitate emulation of other sets of commands. In at least one embodiment, the processor core 3307 also include other processing devices such as a digital signal processor (DSP).

In at least one embodiment, the processor comprises 3302 a cache memory 3304 . In at least one embodiment, processor 3302 have a single internal cache or multiple internal cache levels. In at least one embodiment, the cache memory is used by various components of the processor 3302 shared. In at least one embodiment, the processor uses 3302 also an external cache (e.g. a level 3 cache (L3) or last level cache (LLC)) (not shown) used by the processor cores 3307 shared using known cache coherency techniques. In at least one embodiment, the register file is 3306 additionally in processor 3302 which may include various types of registers for storing various types of data (e.g., integer registers, floating point registers, status registers, and an instruction pointer register). In at least one embodiment, the register file 3306 Include general purpose registers or other registers.

In at least one embodiment, one or more processor (s) 3302 with one or more interface bus (s) 3310 coupled to communication signals such as address, data or control signals between processor 3302 and other components in the system 3300 transferred to. In at least one embodiment, the interface bus can 3310 in one embodiment, be a processor bus, such as a version of a Direct Media Interface (DMI) bus. In at least one embodiment, the interface is 3310 not limited to a DMI bus and may contain one or more Peripheral Component Interconnect buses (e.g. PCI, PCI Express), memory buses, or other types of interface buses. In at least one embodiment, processor (s) 3302 an integrated memory controller 3316 and a platform control hub 3330 . In at least one embodiment, it facilitates memory control 3316 the communication between a memory module and other components of the system 3300 , while the Platform Control Hub (PCH) 3330 Provides connections to I / O modules via a local I / O bus.

In at least one embodiment, the storage device 3320 a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase change memory device, or any other suitable performance storage device that serves as process memory. In at least one embodiment, the device 3320 as system memory for system 3300 work to data 3322 and commands 3321 to save use if one or more processors 3302 run an application or process. In at least one embodiment, the memory controller is 3316 also with an optional external graphics processor 3312 coupled with one or more graphics processors 3308 in processors 3302 can communicate to perform graphics and media operations. In at least one embodiment, a display device 3311 with the processor (s) 3302 get connected. In at least one embodiment, the display device 3311 comprise one or more internal display devices, such as a mobile electronic device or a laptop or an external display device connected via a display interface (e.g. DisplayPort etc.). In at least one embodiment, the display device 3311 a head mounted display (HMD), for example a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

In at least one embodiment, the platform control hub enables 3330 the connection of peripheral devices to the storage controller 3320 and the processor 3302 over a high speed I / O bus. In at least one embodiment, the I / O peripherals include an audio controller, among other things 3346 , a network controller 3334 , a firmware interface 3328 , a wireless transceiver (i.e. transceiver) 3326, touch sensors 3325 , a device for data storage 3324 (e.g. hard drive, flash memory, etc.). In at least one embodiment, the data storage device 3324 can be connected via a storage interface (eg SATA) or via a peripheral bus such as a peripheral component interconnect bus (eg PCI, PCI Express). In at least one embodiment, the touch sensors can 3325 Touchscreen sensors, pressure sensors, or fingerprint sensors include. In at least one embodiment, the wireless transceiver 3326 be a Wi-Fi transceiver, a Bluetooth transceiver, or a transceiver for mobile networks such as a 3G, 4G or Long Term Evolution (LTE) transceiver. In at least one embodiment, the firmware interface enables 3328 communication with the system firmware and can, for example, be a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, the controller 3334 enable a network connection to a wired network. In at least one embodiment, a high performance network controller (not shown) is associated with the interface bus 3310 coupled. In at least one embodiment, the controller is 3346 a multi-channel high definition audio control. In at least one embodiment, the system includes 3300 an optional older 3340 I / O controller for coupling older devices (e.g. personal system 2 (PS / 2)) to the system. In at least one embodiment, the platform control hub 3330 can also be connected with one or more USB controllers (USB = Universal Serial Bus) 3342 to input devices such as keyboard and mouse 3343 combinations, a camera 3344 or other USB input devices.

In at least one embodiment, an instance of the memory controller 3316 and the platform control hub 3330 into a discrete external graphics processor, such as the external graphics processor 3312 , to get integrated. In at least one embodiment, the platform control hub 3330 and / or the memory controller 3316 external to one or more processor (s) 3302 be. In at least one embodiment, System 3300 for example an external memory controller 3316 and a platform control hub 3330 that can be configured as a memory control hub and a peripheral control hub within a system chipset that is compatible with processor (s) 3302 is in communication.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, parts or all of the inference and / or training logic 1415 into the graphics processor 3300 be integrated. For example, in at least one embodiment, the training and / or inferencing techniques described herein may use one or more ALUs contained in the 3D pipeline 3312. In addition, in at least one exemplary embodiment, the inferencing and / or training operations described here can be performed with a different logic than that in FIG 14A or 15B logic shown. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown), the ALUs of the graphics processor 3300 configure them to have one or more here described algorithms for machine learning, neural network architectures, use cases or training techniques.

At least one embodiment can be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

34 Figure 3 is a block diagram of a processor 3400 with one or more processor cores 3402A-3402N , an integrated memory controller 3414 and an integrated graphics processor 3408 , according to at least one embodiment. In at least one embodiment, the processor 3400 include additional cores up to and including the additional core 3402N represented by dashed boxes. In at least one embodiment, each includes the processor cores 3402A-3402N one or more internal cache units 3404A-3404N . In at least one embodiment, each processor core also has access to one or more shared cache units 3406 .

In at least one embodiment, the represent internal cache units 3404A-3404N and the shared memory 3406 a cache memory hierarchy within the processor 3400 . In at least one embodiment, the cache storage units 3404A-3404N include at least one level of instruction and data cache within each processor core and one or more levels of shared mid-level cache, such as, e.g. B. Level-2 (L2), Level-3 (L3), Level-4 (L4) or other cache levels, with the highest cache level in front of the external memory being classified as LLC. In at least one embodiment, the cache coherency logic maintains coherency between different cache units 3406 and 3404A-3404N upright.

In at least one embodiment, the processor 3400 also a set of one or more bus control units 3416 and a system service core 3410 include. In at least one embodiment, one or more bus control units manage 3416 a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, it is the system service core 3410 provided for the functionality of managing various processor components. In at least one embodiment, the system service core comprises 3410 one or more integrated memory controllers 3414 for managing access to various external storage devices (not shown).

In at least one embodiment, include one or more of the processor cores 3402A-3402N Simultaneous multithreading support. In at least one embodiment, the core comprises the system service 3410 Components for coordinating and operating the cores 3402A-3402N during multi-threaded processing. In at least one embodiment, the system service core 3410 additionally comprise a power control unit (PCU), the logic and components for regulating one or more power states of the processor cores 3402A-3402N and the graphics processor 3408 contains.

In at least one embodiment, the processor comprises 3400 additionally the graphics processor 3408 for performing graphics processing operations. In at least one embodiment, the graphics processor is 3408 with shared cache units 3406 and the system service core 3410 coupled to the one or more integrated memory controllers 3414 includes. In at least one embodiment, the system service core comprises 3410 also an ad control 3411 to control the output of the graphics processor on one or more coupled displays. In at least one embodiment, the display controller 3411 also be a separate module that has at least one connection to the graphics processor 3408 is coupled, or it can be in the graphics processor 3408 be integrated.

In at least one embodiment, a ring-based connection unit 3412 used to internal components of the processor 3400 to pair. In at least one embodiment, an alternative connection unit can be used, such as e.g. B. a point-to-point connection, a switch connection or other techniques. In at least one embodiment, the graphics processor is 3408 with the ring connection 3412 coupled via an I / O connection 3413.

In at least one embodiment, the I / O connection 3413 represents at least one of several types of I / O connections, including an I / O connection in the package, the communication between various processor components and an embedded high-performance memory module 3418 such as B. an eDRAM module allows. In at least one embodiment, each of the processor cores use 3402A-3402N and the graphics processor 3408 embedded memory modules 3418 as a shared last-level cache.

In at least one embodiment, the processor cores are 3402A-3402N homogeneous cores that execute a common instruction set architecture. In at least one embodiment, the processor cores are 3402A-3402N heterogeneous in terms of instruction set architecture (ISA), with one or more processor cores 3402A-3402N execute a common set of instructions while one or more other cores of the processor cores 3402A-34-02N execute a subset of a common instruction set or another instruction set. In at least one embodiment, the processor cores are 3402A-3402N heterogeneous in terms of microarchitecture, with one or more cores with a relatively higher power consumption being coupled with one or more power cores with a lower power consumption. In at least one embodiment, the processor 3400 be implemented on one or more chips or as an integrated SoC circuit.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, parts or all of the inference and / or training logic 1415 into the graphics processor 3410 be integrated. For example, in at least one embodiment, the techniques described herein for training and / or inferencing may use one or more of the ALUs that are in the 3D pipeline 3312, the graphics core (s) 3415A , the shared functional logic 3416 , the graphics core (s) 3415B , the shared functional logic 3420 or some other logic in 34 are included. In addition, in at least one embodiment, the inferencing and / or training operations described here can be performed with a different from that in 14A or 14B logic shown. In at least one embodiment, weighting parameters can be stored in an on-chip or off-chip memory and / or registers (shown or not shown), the ALUs of the graphics processor 3410 configure to perform one or more machine learning algorithms, architectures, use cases, or neural network training techniques described herein.

At least one embodiment can be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

35 Figure 3 is a block diagram of a graphics processor 3500 , which can be a discrete graphics processing unit, or a graphics processor integrated into a variety of processing cores. In at least one embodiment, the graphics processor is communicating 3500 via an I / O interface assigned to the memory with registers of the graphics processor 3500 and with commands entered into memory. In at least one embodiment, the graphics processor comprises 3500 a memory interface 3514 for memory access. In at least one embodiment, the memory interface is 3514 an interface to local memory, to one or more internal caches, to one or more shared external caches and / or to system memory.

In at least one embodiment, the graphics processor comprises 3500 also an ad control 3502 for controlling the output of display data to a display device 3520 . In at least one embodiment, the display control includes 3502 Hardware for one or more overlay levels for the device 3520 and a composite of multiple layers of video or user interface elements. In at least one embodiment, the display device 3520 be an internal or external display device. In at least one embodiment, the display device is 3520 a head-mounted display device such as a VR (Virtual Reality) display or an AR (Augmented Reality) display. In at least one embodiment, the graphics processor comprises 3500 a video codec engine 3506 for encoding, decoding, or transcoding media to, from, or between one or more media encoding formats including, but not limited to, Moving Picture Experts Group (MPEG) formats such as MPEG-2, Advanced Video Coding (AVC) formats such as H.264 IMPEG-4 AVC, as well as the Society of Motion Picture & Television Engineers (SMPTE) 421M / VC-1 and the Joint Photographic Experts Group (JPEG) formats such as JPEG and Motion JPEG (MJPEG) formats.

In at least one embodiment, the graphics processor comprises 3500 one engine 3504 for block image transfer (BLIT) in order to perform two-dimensional (2D) rasterization operations, including, for example, bit-boundary block transfers. In at least one embodiment, however, the 2D graphics operations are performed using one or more components of the graphics processing engine (GPE). 3510 executed. In at least one embodiment, the GPE is 3510 a computational engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

In at least one embodiment, the GPE comprises 3510 a 3D pipeline 3512 for performing 3D operations, such as rendering three-dimensional images and scenes with processing functions that work with 3D primitive shapes (e.g. rectangle, triangle, etc.). The 3D pipeline 3512 includes programmable and fixed functional elements that perform various tasks and / or threads of execution on a 3D / media subsystem 3515 produce. While the 3D pipeline 3512 can be used to perform media operations in at least one embodiment includes the GPE 3510 also a media pipeline 3516 used for performing media operations such as B. video post-processing and image enhancement is used.

In at least one embodiment, the media engine comprises 3516 Fixed functional units or programmable logical units for performing one or more specialized media operations, such as accelerating video decoding, video de-interlacing and accelerating video coding instead of or for the video codec engine 3506 . In at least one embodiment, the media pipeline comprises 3516 additionally a thread generation unit for generating threads for execution on the 3D / media subsystem 3515 . In at least one embodiment, the generated threads perform computations for media operations on one or more graphics execution units that are in the 3D / media subsystem 3515 are included.

In at least one embodiment, the 3D / media subsystem comprises 3515 Logic to execute threads by the 3D pipeline 3512 and the media pipeline 3516 be generated. In at least one embodiment, send the 3D pipeline 3512 and the media pipeline 3516 Thread execution requirements on the 3D / Media subsystem 3515 which includes thread distribution logic to arbitrate and distribute various requests for available thread execution resources. In at least one embodiment, the execution resources include an array of graphics execution units for processing 3D and media threads. In at least one embodiment, the 3D / media subsystem comprises 3515 one or more internal caches for thread instructions and data. In at least one embodiment, the subsystem comprises 3515 also shared memory, including registers and addressable memory, to share data between threads and to store output data.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, parts or all of the inference and / or training logic 1415 into the graphics processor 3500 to get integrated. For example, in at least one embodiment, the training and / or inferencing techniques described herein can use one or more ALUs that are in the 3D pipeline 3512 are included. In addition, in at least one exemplary embodiment, the inferencing and / or training operations described here can be performed with a different logic than that in FIG 14A or 15B logic shown. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memories and / or registers (shown or not shown), the ALUs of the graphics processor 3500 Configure so that they can execute one or more of the machine learning algorithms, network architectures, use cases, or training techniques for neural networks described here.

At least one embodiment can be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

36 Figure 3 is a block diagram of a graphics processing engine 3610 a graphics processor, according to at least one embodiment. In at least one embodiment, the graphics processing engine (GPE) is 3610 a version of the in 35 shown GPE 3510 . In at least one embodiment is the media pipeline 3516 optional and cannot be explicitly used in GPE 3610 be included. In at least one embodiment, a separate media and / or image processor is attached to GPE 3610 coupled.

In at least one embodiment, the GPE is 3610 with a command streamer 3603 coupled or comprised of an instruction stream for the 3D pipeline 3512 and / or the media pipelines 3516 provides. In at least one embodiment, the command streamer is 3603 coupled to memory, which may be system memory or one or more internal caches and shared caches. In at least one embodiment, the command streamer receives 3603 Commands from memory and sends commands to the 3D pipeline 3512 and / or the media pipeline 3516 . In at least one embodiment, the instructions are instructions, primitives or micro-operations that are fetched from a ring buffer in the instructions for the 3D pipeline 3512 and the media pipeline 3516 are stored. In at least one embodiment, a ring buffer can additionally comprise batch command buffers in which batches of a plurality of commands are stored. In at least one embodiment, instructions for the 3D pipeline 3512 also include references to data stored in memory such as, but not limited to, vertex and geometry data for the 3-D pipeline 3512 and / or image data and storage objects for the media pipeline 3516 . In at least one embodiment, process the 3D pipeline 3512 and the media pipeline 3516 Commands and data by performing operations or one or more threads of execution on a graphics core array 3614 send. In at least one embodiment, graphics core includes array 3614 one or more blocks of graphics cores (e.g. graphics core (s) 3615A , Graphics core (s) 3615B) , each block comprising one or more graphics cores. In at least one embodiment, each graphics core comprises a set of graphics execution resources, which includes general and graphics-specific execution logic for performing graphics and arithmetic operations as well as texture processing with fixed functionality and / or machine learning and acceleration logic with artificial intelligence, including inference and / or training logic 1415 in 14A and 14B .

In at least one embodiment, the 3D pipeline comprises 3512 Logic with fixed functionality and programmable logic to create one or more shading programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders or other shading programs, by processing commands and sending execution threads to graphics core array 3614 to process. In at least one embodiment, graphics core represents array 3614 provides a unified block of execution resources for use in processing shading programs. In at least one embodiment, the general execution logic (e.g. execution units) within the graphics core (s) 3615A-3715B of the graphics core array 3614 Support for various 3D API shading languages and can run multiple simultaneous threads of execution associated with multiple shaders.

In at least one embodiment, the graphics core comprises an array 3614 also execution logic for executing media functions such as video and / or image processing. In at least one embodiment, the execution units include, in addition to the graphics processing operations, general purpose logic that is programmable to perform parallel general arithmetic operations.

In at least one embodiment, output data generated by threads running on the graphics core array 3614 executed, data in memory in a Unified Return Buffer (URB) 3618 output. The URB 3618 can store data for multiple threads. In at least one embodiment, the URB 3618 used to send data between different threads running on graphics core array 3614 are executed. In at least one embodiment, URB 3618 in addition to the synchronization between threads on the graphics core array 3614 and fixed functional logic within the common functional logic 3620 be used.

In at least one embodiment, the graphics core is an array 3614 so scalable that the graphics core array 3614 comprises a variable number of graphics cores, each of which has a variable number of execution units based on a target and performance level of GPE 3610 having. In at least one embodiment, the execution resources are dynamically scalable so that the execution resources can be activated or deactivated as required.

In at least one embodiment, the graphics core is an array 3614 with the shared functional logic 3620 coupled, which includes multiple resources, used by the graphics cores in graphics core array 3614 shared. In at least one embodiment, shared functions are those of the shared functional logic 3620 executed in hardware logic Units embodies the special additional functions to the graphics core array 3614 provide. In at least one embodiment, the shared functional logic comprises 3620 the logic of scanner 3621 , Mathematics 3622 and inter-thread communication (ITC) 3623 but is not limited to this. In at least one embodiment, one or more cache (s) are 3625 in the shared functional logic 3620 included or coupled with it.

In at least one embodiment, a shared function is used when there is a demand for a specialized function for inclusion in graphics core arrays 3614 not enough. In at least one embodiment, a single instantiation of a specialized function is used in the shared function logic 3620 used and by other execution resources within the graphics core array 3614 shared. In at least one embodiment, specific shared functions can be used within the shared function logic 3620 that in large part of graphics core array 3614 used in the shared functional logic 3616 within graphics core array 3614 be included. In at least one embodiment, the shared functional logic 3616 within the graphics core array 3614 some or all of the logic within the shared functional logic 3620 include. In at least one embodiment, all logic elements within the shared functional logic 3620 in the shared functional logic 3616 of the graphics core array 3614 be duplicated. In at least one embodiment, the function logic is shared 3620 in favor of the jointly used functional logic 3616 within the graphics core array 3614 locked out.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, parts of the inference and / or training logic 1415 or all of the inference and / or training logic 1415 into the graphics processor 3610 to get integrated. For example, in at least one of the exemplary embodiments, training and / or inference techniques described here, one or more of the ALUs that are in the 3D pipeline can be used 3612 , the graphics core (s) 3615A , the common functional logic 3616 , the graphic core (s) 3615B , the common functional logic 3620 or some other logic in 36 are included. Furthermore, in at least one exemplary embodiment, the inferencing and / or training operations described here can be performed with a different logic than that in 14A or 15B logic shown. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown), the ALUs of the graphics processor 3610 configure them to execute one or more of the algorithms described here for machine learning, network architectures, use cases or training techniques of neural networks.

At least one embodiment can be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

37 Figure 4 is a block diagram of the hardware logic of a graphics processor core 3700 , according to at least one embodiment that is described here. In at least one embodiment, the graphics processing core is 3700 contained in a graphics core array. In at least one embodiment, the graphics processor core 3700 , sometimes referred to as a core section, can be one or more graphics cores within a modular graphics processor. In at least one embodiment, the graphics processing core is 3700 exemplary of a graphics core section, and a graphics processor as described herein, may comprise multiple graphics core sections that are based on target power and performance envelopes. In at least one embodiment, each graphics core can 3700 a fixed function block 3730 include, the one with several sub-kems 3701A-3701F is coupled, which are also referred to as subsections and comprise modular blocks with general and fixed functional logic.

In at least one embodiment, the block includes fixed functionality 3730 a geometry pipeline with fixed functionality 3736 by all sub-kemas in the graphics processor 3700 can be shared, for example in graphics processor implementations with lower performance and / or lower power consumption. In at least one embodiment, the geometry pipeline includes fixed functionality 3736 a pipeline with fixed 3D functions, a video front-end unit, a thread producer and thread distributor as well as a standardized return buffer manager that manages standardized return buffers.

In at least one embodiment, the block includes fixed functionality 3730 also a graphics SoC interface 3737 , a graphics microcontroller 3738 and a media pipeline 3739 . The graphics SoC interface 3737 is an interface between the graphics core 3700 and other processor cores within a system on an integrated circuit on a chip. In at least one embodiment, the graphics microcontroller is 3738 a programmable subprocessor that is configurable to perform various functions of the graphics processor 3700 Manage, including thread dispatcher, scheduler and pre-emption. In at least one embodiment, the media pipeline comprises 3739 Logic to facilitate the decoding, encoding, preprocessing and / or post-processing of multimedia data, including image and video data. In at least one embodiment, implements the media pipeline 3739 Media operations via requests for computation or sampling logic within the sub-cores 3701-3701F .

In at least one embodiment, the SoC interface enables 3737 the graphics core 3700 communication with general application processor cores (e.g. CPUs) and / or other components within a SoC, including memory hierarchy elements such as a shared top-level cache, system RAM, and / or embedded on-chip or on-package DRAM . In at least one embodiment, the SoC interface 3737 communication with devices with fixed functions within a SoC, such as camera image processing pipelines, enables and enables the use and / or implementation of global memory modules (atomics) from the graphics core 3700 and CPUs can be shared within a SoC. In at least one embodiment, the SoC interface 3737 also power management controls for the graphics core 3700 and implement an interface between a clock domain of the graphics core 3700 and other clock domains within a SoC. In at least one embodiment, the SoC interface enables 3737 receiving command buffers from a command streamer and global thread dispatcher configured to provide commands and commands to each of one or more graphics cores within a graphics processor. In at least one embodiment, commands and instructions can be sent to the media pipeline 3739 sent when media operations are to be performed, or to a fixed-function geometry pipeline (e.g., fixed-function geometry pipeline 3736 , Geometry pipeline with fixed function 3714 ) when graphics processing operations are to be performed.

In at least one embodiment, the graphics microcontroller can 3738 can be configured to perform various planning and management tasks for the graphics core 3700 to execute. In at least one embodiment, the graphics microcontroller can 3738 Graphics and / or workload planning for various parallel graphics engines within the arrays 3702A-3702F and 3704A-3704F of the execution unit (EU) within the sub-core 3701A-3701F carry out. In at least one embodiment, host software may run on a CPU core of a SoC including the graphics core 3700 is running, workloads send one of several graphics processor doorbells, which invokes a scheduling operation on an appropriate graphics engine. In at least one embodiment, the planning operations include determining the workload to be executed next, broadcasting a workload to a command streamer, anticipating existing workloads running on an engine, monitoring progress of a workload, and notifying the host software when a workload is completed. In at least one embodiment, the graphics microcontroller can 3738 also low-power or idle states for the graphics core 3700 enable by having the graphics core 3700 offers the option of registering within the graphics kernel 3700 save and restore on a system independently of an operating system and / or graphics driver software during low-power state transitions.

In at least one embodiment, the graphics core 3700 more or less than the sub-ces shown 3701A-3701F have up to N modular sub-ces. In at least one embodiment, the graphics core 3700 also shared functional logic for each set of N sub-cores 3710 , shared and / or cache memory 3712 , a geometry pipeline with fixed functionality 3714 as well as additional fixed functional logic 3716 to speed up various graphics and arithmetic operations. In at least one embodiment, the shared functional logic 3710 Logic units (e.g., scanner, math, and / or inter-thread communication logic) include those of each of the N sub-kernels within the graphics kernel 3700 can be used together. The shared and / or cache memory 3712 can have a last-level cache for N subemes 3701A-3701F within the graphics core 3700 and can also serve as a shared memory that can be accessed by multiple subemes. In at least one embodiment, the geometry Fixed functionality pipeline 3714 instead of the geometry pipeline with fixed functionality 3736 within the fixed function block 3730 be included and can comprise the same or similar logical units.

In at least one embodiment, the graphics core comprises 3700 an additional logic with fixed functionality 3716 , the various acceleration logic with fixed functionality for use by the graphics core 3700 may include. In at least one embodiment, the additional logic includes fixed functionality 3716 an additional geometry pipeline for use in position shading only. In position-only shading, at least two geometry pipelines exist, while in a complete geometry pipeline within the geometry pipeline with fixed functionality 3716 , 3736 and a cull pipeline, which is an additional geometry pipeline, which in the additional logic with fixed functionality 3716 may be included. In at least one embodiment, the fade out pipeline is a stripped-down version of a full geometry pipeline. In at least one embodiment, a full pipeline and a hide pipeline can run different instances of an application, with each instance having a separate context. In at least one embodiment, only positional shading can suppress long fade out passes for discarded triangles, which in some cases can complete the shading earlier. For example, in at least one embodiment, the gating pipeline logic may be within the additional fixed functional logic 3716 Perform positional shading in parallel with a main application and generally produce critical results faster than a full pipeline because the fade pipeline retrieves and shades the positional attribute of vertices without rasterizing and rendering pixels to a frame buffer. In at least one embodiment, the hide pipeline can use generated critical results to compute visibility information for all triangles, regardless of whether those triangles are hidden. In at least one embodiment, a full pipeline (which in this case can be referred to as a rendering pipeline) can use the visibility information to skip hidden triangles and only shade the visible triangles that will eventually be passed on to a rasterization phase.

In at least one embodiment, the additional logic can be fixed function 3716 also contain an acceleration logic for machine learning, such as a multiplication logic with a fixed function matrix, for implementations that contain optimizations for machine learning or inferencing.

In at least one embodiment, each includes graphics sub-kernels within 3701A-3701F a set of execution resources that can be used to perform graphics, media, and computational operations in response to requests from the graphics pipeline, media pipeline, or shading programs. In at least one embodiment, the graphics sub-cores include 3701A-3701F multiple EU arrays 3702A-3702F , 3704A-3704F , Thread distribution and inter-thread communication logic (TD / IC) 3703A-3703F , a 3D (e.g. texture) scanner 3705A-3705F , a media scanner 3706A-3706F , a shading processor 3707A-3707F and shared local memory (SLM) 3708A-3708F . The EU arrays 3702A-3702F , 3704A-3704F each comprise multiple execution units, which are general-purpose graphics processing units capable of performing floating point and integer / fixed point logic operations to support graphics, media, or arithmetic operations, including graphics, media, or arithmetic shading programs . Compute Shader Program). In at least one embodiment, the TD / IC logic performs 3703A-3703F local thread distribution and control operations for execution units within a sub-core and facilitates communication between threads running on execution units of a sub-core. In at least one embodiment, the 3-D scanner can 3705A-3705F Read texture or other 3D graphics data into memory. In at least one embodiment, the 3-D scanner may read texture data differently based on a configured scan state and texture format associated with a given texture. In at least one embodiment, the media scanner 3706A-3706F perform similar operations based on a type and format associated with media data. In at least one embodiment, each graphics sub-kernel 3701A-3701F alternately comprise a unified 3D and media scanner. In at least one embodiment, threads running on execution units within each sub-core 3701A-3701F running on the shared local storage 3708A-3708F within each sub-core so that threads running within a thread group can run using a shared pool of on-chip memory.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least In one embodiment, some or all of the inference and / or training logic can be used 1415 into the graphics processor 3710 to get integrated. For example, in at least one embodiment, the training and / or inferencing techniques described herein may use one or more ALUs that are in the 3D pipeline 3710, the graphics microcontroller 3738 , the geometry pipeline with fixed functionality 3714 and 3736 or other logic in 35 are included. In addition, in at least one exemplary embodiment, the inferencing and / or training operations described here can be performed with a method other than that in 14A or 15B logic shown. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown), the ALUs of the graphics processor 3700 configure them to execute one or more of the machine learning algorithms, architectures, use cases, or training techniques of neural networks described here.

At least one embodiment can be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

38A-38B show the thread execution logic 3800 , comprising an array of processing elements of a graphics processor core, according to at least one embodiment. 38A Figure 3 shows at least one embodiment in which the thread execution logic 3800 is used. 38B shows exemplary internal details of an execution unit in accordance with at least one exemplary embodiment.

As in 38A shown includes the thread execution logic 3800 in at least one embodiment, a shading processor 3802 , a thread distributor 3804 , the command cache 3806 , a scalable array of execution units that includes a variety of execution units 3808A-3808N , a scanner 3810 , a data cache 3812 and a data port 3814 includes. In at least one embodiment, a scalable array of execution units can be dynamically scaled by adding one or more execution units (e.g., any of the execution units 3808A , 3808B , 3808C , 3808D up to 3808N-1 and 3808N) can be enabled or disabled based on the computational requirements of a workload. In at least one exemplary embodiment, scalable execution units are connected to one another via a connection structure which establishes a connection to each of the execution units. In at least one embodiment, the thread includes execution logic 3800 one or more connections to the memory, e.g. To system memory or cache memory, via one or more of instruction caches 3806 , Data port 3814 , Scanner 3810 and execution units 3808A-3808N . In at least one embodiment, each execution unit (eg 3808A) is an independent programmable general processing unit that is able to execute several simultaneous hardware threads and process several data elements in parallel for each thread. In at least one embodiment, the array is the execution units 3808A-3808N scalable to include any number of individual execution units.

In at least one embodiment, the execution units 3808A-3808N primarily used to run shading programs. In at least one embodiment, the shading processor 3802 various shading programs process and threads of execution associated with shading programs through a thread dispatcher 3804 to distribute. In at least one embodiment, the thread dispatcher comprises 3804 Logic for arbitrating thread initiation requests from graphics and media pipelines and for instantiating requested threads on one or more execution units in the execution units 3808A-3808N . For example, in at least one embodiment, a geometry pipeline may distribute vertex, tessellation, or geometry shaders to the thread execution logic for processing. In at least one embodiment, the thread dispatcher 3804 also process thread creation requests from executing shading programs at runtime.

In at least one embodiment, the execution units support 3808A-3808N an instruction set that includes native support for many standard 3D graphics shading commands so that shading programs from graphics libraries (e.g. Direct 3D and OpenGL) can be executed with a minimal translation. In at least one embodiment, the execution units support vertex and geometry processing (e.g., vertex programs, geometry programs, vertex shaders), pixel processing (e.g., pixel shaders, fragment shaders), and general processing (e.g., arithmetic) - and media shaders). In at least one embodiment, each of the execution units is 3808A-3808N , which include one or more arithmetic-logic units (ALUs), capable of executing SIMD (Single Instruction Multiple Data) execution with several instructions, and of multi- Despite higher latency memory accesses, thread operation enables an efficient execution environment. In at least one embodiment, each hardware thread within each execution unit has a dedicated high bandwidth register file and an associated independent thread state. In at least one embodiment, execution occurs multiple times per clock cycle on pipelines that can execute integer, single- and double-precision floating point operations, SIMD branches, logical operations, transcendent operations, and other various operations. In at least one embodiment, the dependency logic initiates within the execution units 3808A-3808N while waiting for data from the memory or one of the shared functions, a waiting thread to pause (sleep) until the requested data has been returned. In at least one embodiment, it can happen that while a waiting thread is in the idle state, hardware resources are used for the processing of other threads. For example, in at least one embodiment, during a delay associated with a vertex shading operation, an execution unit may perform operations on a pixel shader, fragment shader, or other type of shading program, including another vertex shader.

In at least one embodiment, each execution unit operates in the execution units 3808A-3808N with arrays of data elements. In at least one embodiment, a number of data items is the "execution size" or the number of channels for an instruction. In at least one embodiment, an execution channel is a logical unit of execution for data element access, masking, and flow control within instructions. In at least one embodiment, a number of channels may be independent of a number of physical arithmetic logic units (ALUs) or floating point units (FPUs) for a particular graphics processor. In at least one embodiment, the execution units support 3808A-3808N Integer and floating point data types.

In at least one exemplary embodiment, an instruction set of an execution unit comprises SIMD instructions. In at least one embodiment, different data elements can be stored as a packed data type in a register, and the execution unit processes different elements based on the data size of the elements. For example, in at least one embodiment, when operating with a 256-bit wide vector 256 Bits of a vector are stored in a register, and an execution unit operates on a vector as four separate 64-bit packed data elements (data elements of quad-word size (QW)), eight separate 32-bit packed data elements (data elements of double word size (DW)), sixteen separate 16-bit packed data elements (data elements of size Word (W)) or thirty-two separate 8-bit data elements (data elements of size byte (B)). In at least one embodiment, however, different vector widths and register sizes are possible.

In at least one exemplary embodiment, one or more execution units can form a merged execution unit 3809A-3809N with a thread control logic ( 3807A-3807N ), which is common to all merged EUs. In at least one exemplary embodiment, several EUs can be merged into one EU group. In at least one embodiment, each ES in a merged ES group can be configured to run a separate SIMD hardware thread. The number of EUs in a merged EU group can vary according to different exemplary embodiments. In at least one embodiment, different SIMD widths can be implemented per EU, including, but not limited to, SIMD8, SIMD16, and SIMD32. In at least one embodiment, each comprises fused graphics execution unit 3809A-3809N at least two execution units. For example, the merged execution unit comprises 3809A in at least one embodiment, a first ES 3808A , a second EU 3808B and control of the thread 3807A that the first EU 3808A and the second EU 3808B is common. In at least one embodiment, the thread control logic controls 3807A those on the merged graphics execution unit 3809A executed threads so that each EU is within the merged execution units 3809A-3809N can be executed using a common instruction pointer register.

In at least one embodiment, there are thread execution logic 3800 contain one or more internal instruction caches (e.g., 3806) to cache thread instructions for execution units. In at least one embodiment, one or more data caches (e.g., 3812) are included to cache thread data during thread execution. In at least one embodiment, a sampler 3810 is included to provide texture sampling for 3D operations and media sampling for media operations. In at least one embodiment, includes scanners 3810 specialized texture or media sampling functionality to process texture or media data during the sampling process before providing sampled data to an execution unit.

During execution, in at least one embodiment, graphics and media pipelines send thread initiation requests to thread execution logic through thread creation and dispatch logic 3800 . In at least one embodiment, once a group of geometric objects has been processed and rasterized into pixel data, pixel processor logic (e.g., pixel shading logic, fragment shading logic, etc.) is implemented within the shading processor 3802 called to calculate further output information and to cause the results to be written on output surfaces (e.g. color buffer, depth buffer, template buffer, etc.). In at least one embodiment, a pixel shader or fragment shader computes values of various vertex attributes to be interpolated over a rasterized object. In at least one embodiment, the pixel processor logic performs within the shading processor 3802 then run a pixel or fragment shading program provided by an application programming interface (API). In at least one embodiment, the shading processor distributes 3802 to execute a thread shading program through the thread dispatcher 3804 to an execution unit (e.g. 3808A). In at least one embodiment, the shading processor uses 3802 the texture sampling logic in sampler 3810 to access texture data in texture maps allocated in memory. In at least one embodiment, arithmetic operations on texture data and input geometry data are used to calculate pixel color data for each geometrical fragment or to exclude one or more pixels from further processing.

In at least one embodiment, the data port 3814 a memory access mechanism for the thread execution logic 3800 ready to output processed data to memory for further processing on a graphics processor output pipeline. In at least one embodiment, the data port comprises 3814 one or more cache memories (e.g. data cache 3812 ) or is coupled to these in order to temporarily store data for memory access via a data port.

As in 38B shown, a graphics execution unit 3808 in at least one embodiment, an instruction fetch unit 3837 , a general register file array (GRF) 3824 , an Architecture Register File Array (ARF) 3826 , a thread arbiter 3822 , a transmitter unit 3830 , a branch unit 3832 , a set of SIMD floating point units (FPUs) 3834 and, in at least one embodiment, a set of dedicated integer SIMD ALUs 3835 include. In at least one embodiment, the register files include GRF 3824 and ARF 3826 a set of general register files and architectural register files associated with each simultaneous hardware thread running on the graphics execution unit 3808 can be active. In at least one embodiment, the architecture state per thread is in ARF 3826 while the data used during thread execution is stored in GRF 3824 get saved. In at least one embodiment, the execution state of each thread, including the instruction pointers for each thread, can be stored in thread-specific registers in ARF 3826 being held.

In at least one exemplary embodiment, the graphics execution unit 3808 an architecture that is a combination of Simultaneous Multi-Threading (SMT) and fine-grained Interleaved Multi-Threading (IMT). In at least one embodiment, the architecture has a modular configuration that can be fine-tuned at design time based on a target number of simultaneous threads and a number of registers per execution unit, dividing the execution unit's resources among the logic required to execute multiple simultaneous threads is used.

In at least one embodiment, the graphics execution unit 3808 Issue several commands, each of which can be different, together. In at least one embodiment, the thread arbiter 3822 of the graphics execution unit thread 3808 Commands to be executed on one of the transmitter units 3830 , Branching unit 3842 or send SIMD-FPU (s) 3834. In at least one embodiment, each thread of execution can access 128 general registers in the GRF 3824 access, each register 32 Can store bytes that can be accessed as a SIMD 8-element vector of 32-bit data elements. In at least one embodiment, each thread of the execution unit has access to 4K bytes within the GRF 3824 although the execution units are not as limited and more or fewer register resources may be provided in other embodiments. In at least one embodiment, up to seven threads can be executed simultaneously, although the number of threads per execution unit can also vary depending on the embodiment. In at least one embodiment in which seven threads can access 4K bytes, the GRF 3824 save a total of 27 KB. In at least one embodiment, flexible addressing modes can enable registers to be addressed together in order to effectively build wider registers or to represent dashed rectangular block data structures.

In at least one exemplary embodiment, memory operations, scanner operations and other system communications with a longer latency period are sent via “send” commands that are sent by the message-transferring sending unit 3830 are executed. In at least one embodiment, branch instructions are sent to a dedicated branch unit 3832 sent to facilitate SIMD divergence and finally convergence.

In at least one embodiment, the graphics execution unit comprises 3808 one or more SIMD floating point units (FPU (s)) 3834 for performing floating point operations. In at least one embodiment, the FPU (s) 3834 also support integer computation. In at least one embodiment, FPU (s) 3834 SIMD can perform up to M 32-bit floating point (or integer) operations, or SIMD can perform up to 2M 16-bit integer or 16-bit floating point operations. In at least one embodiment, at least one of the FPU (s) is equipped with advanced math functions to support high-throughput transcendent math functions and double-precision 64-bit floating point operations. In at least one embodiment, there is also a set of 8-bit integer SIMD ALUs 3835 that can be specifically optimized to perform operations associated with machine learning computations.

In at least one embodiment, arrays can consist of multiple instances of graphics core array 3808 instantiated in a graphics sub-core grouping (e.g. a subsection). In at least one unit of the execution unit 3808 In at least one embodiment, instructions can be executed over a variety of execution channels. In at least one embodiment, each thread running on the graphics execution unit 3808 is running, running in a different channel.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, parts or all of the inference and / or training logic 1415 into the execution logic 3800 to get integrated. Furthermore, in at least one embodiment, inferencing and / or training operations described herein may use different logics than that in FIG 14A or 15B logic shown can be used. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown), the ALUs of the execution logic 3800 Configure so that they can execute one or more of the machine learning algorithms, architectures, use cases, or training techniques for neural networks described here.

At least one embodiment can be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

39 FIG. 3 shows a parallel processing unit (“PPU”) 3900 according to at least one exemplary embodiment. In at least one embodiment, the PPU is 3900 configured with machine readable code which when sent from the PPU 3900 is running the PPU 3900 causes some or all of the processes and techniques described in this disclosure to be performed. In at least one embodiment, it is PPU 3900 a multithreaded processor implemented on one or more integrated circuit devices that uses multithreading as a latency reduction technique designed to process computer-readable instructions (also known as machine-readable instructions or just instructions) on multiple threads in parallel. In at least one embodiment, a thread refers to an execution thread and is an instantiation of an instruction set that is configured to be received from PPU 3900 is performed. In at least one embodiment, it is PPU 3900 a graphics processing unit ("GPU") configured to implement a graphics rendering pipeline for processing three-dimensional ("3D") graphics data to produce two-dimensional ("2D") image data for display on a display device such as a a liquid crystal display device ("LCD"). In at least one embodiment, PPU 3900 used to perform calculations such as linear algebra operations and machine learning. 39 Illustrates an example of a parallel processor for illustrative purposes only and is to be understood as a non-limiting example of processor architectures that may be considered within the scope of this disclosure, and any suitable processor may be used to supplement and / or replace it.

In at least one embodiment, there are one or more PPUs 3900 configured to accommodate high performance computing ("HPC"), data centers and machine learning applications accelerate. In at least one embodiment, the PPU is 3900 configured to accelerate systems and applications for deep learning, including the following non-limiting examples: autonomous vehicle platforms, deep learning, high-precision speech, image, text recognition systems, intelligent video analysis, molecular simulations, drug discovery, disease diagnosis, weather forecasting, large-scale analysis Datasets, astronomy, molecular dynamics simulation, financial modeling, robotics, factory automation, real-time voice translation, online search optimizations and personalized user recommendations, and much more.

In at least one embodiment, the PPU comprises 3900 , without restriction, an input / output unit ("I / O") 3906, a front-end unit 3910 , a planning unit 3912 , a labor distribution unit 3914 , a hub 3916 , a crossbar (“Xbar”) 3920, one or more general processing clusters (“GPCs”) 3918, and one or more partition units (“memory partition units”) 3922. In at least one embodiment, the PPU 3900 with a host processor or other PPUs 3900 Connected via one or more high-speed GPU connections (“GPU Connections”) 3908. In at least one embodiment, it is PPU 3900 connected via an interconnect 3902 to a host processor or other peripheral devices. In at least one embodiment, it is PPU 3900 connected to local storage comprising one or more storage devices ("Storage") 3904. In at least one exemplary embodiment, the memory chips comprise 3904 one or more Dynamic Random Access Memory (“DRAM”) modules without restriction. In at least one exemplary embodiment, one or more DRAM modules are configured and / or configurable as memory subsystems with high bandwidth (“HBM”), with a plurality of DRAM chips being stacked in each device.

In at least one embodiment, the GPU can be high-speed connection 3908 refer to a wired, multi-lane communication link used by systems to scale and incorporate one or more PPUs 3900 in combination with one or more central processing units ("CPUs"), the cache coherence between PPUs 3900 and CPUs as well as CPU mastering supported. In at least one embodiment, data and / or commands are passed through the high speed GPU interconnect unit 3908 over the hub 3916 to / from other units of the PPU 3900 such as one or more copy engines, video encoders, video decoders, power management units and other components that are included in 39 cannot be shown explicitly.

In at least one embodiment, the I / O unit 3906 is configured to receive communications (e.g., commands, data) from a host processor (not in 39 shown) via the system bus 3902 sends and receives. In at least one embodiment, the I / O unit 3906 communicates with the host processor directly over the system bus 3902 or via one or more intermediate devices such as a memory bridge. In at least one embodiment, the I / O unit 3906 may be connected to one or more other processors, such as one or more PPUs 3900 , via the system bus 3902 communicate. In at least one embodiment, the I / O unit 3906 implements a Peripheral Component Interconnect Express (“PCIe”) interface for communication over a PCIe bus. In at least one embodiment, the I / O unit 3906 implements interfaces for communication with external devices.

In at least one embodiment, the I / O unit 3906 decodes over the system bus 3902 received packets. In at least one embodiment, at least some packets represent commands configured to use the PPU 3900 cause various operations to be performed. In at least one embodiment, the I / O unit 3906 transmits decoded commands to various other units of the PPU 3900 as specified by commands. In at least one embodiment, commands are sent to the front-end unit 3910 and / or to the hub 3916 or other units of the PPU 3900 such as one or more copy engines, a video encoder, a video decoder, a power management unit, etc. (not explicitly in 39 shown). In at least one embodiment, the I / O unit 3906 is configured to facilitate communication between and between different logical units of PPU 3900 forwards.

In at least one embodiment, a program executed by the host processor encodes an instruction stream in a buffer, the PPU 3900 Provision workloads for processing. In at least one embodiment, a workload includes instructions and data to be processed by those instructions. In at least one embodiment, the buffer is an area in memory that both the host processor and the PPU 3900 can access (e.g. read / write) - a host interface unit can be configured to handle memory requests made over the system bus 3902 from the I / O base 3906 via the system bus 3902 are transferred to the buffer in one with the system bus 3902 connected system memory. In at least one embodiment, the host processor writes an instruction stream to the buffer and then sends a pointer to the beginning of the instruction stream to PPU 3900 so that the front end unit 3910 Receives pointer to one or more instruction streams and manages one or more instruction streams, reads instructions from instruction streams and sends instructions to various units of PPU 3900 forwards.

In at least one exemplary embodiment, the front-end unit is 3910 with the planner unit 3912 coupled to the different GPCs 3918 configured to process tasks defined by one or more instruction streams. In at least one embodiment, the planning unit is 3912 configured to provide status information to various from the scheduler unit 3912 managed tasks, with status information indicating which of the GPCs 3918 a task is assigned, whether the task is active or inactive, which priority level is associated with the task, etc. In at least one exemplary embodiment, the planning unit plans 3912 the execution of a variety of tasks on one or more GPCs 3918 .

In at least one embodiment, the planning unit is 3912 with the work distribution unit 3914 coupled that is configured to run tasks on GPCs 3918 distributed. In at least one embodiment, the work distribution unit tracks 3914 a number of scheduled tasks performed by the scheduler unit 3912 are received, and the work distribution unit 3914 maintains a pool of outstanding tasks and a pool of active tasks for each of the GPCs 3918 . In at least one embodiment, the pending task pool includes a number of slots (eg, 32 slots) that contain tasks that are to be processed by a particular GPC 3918 are assigned; the active task pool may include a number of slots (e.g. 4 slots) for tasks to be performed by the GPCs 3918 actively processed so that if any of the GPCs 3918 completes the execution of a task, that task from the Active Task Pool for GPC 3918 is displaced and one of the other tasks is selected from the pool of outstanding tasks and executed on GPC 3918 is planned. If in at least one embodiment there is an active task on GPC 3918 is inactive, e.g. while waiting for a data dependency to be resolved, the active task becomes GPC 3918 displaced and returned to the pending task pool, while another task is selected from the pending task pool and ready to run on GPC 3918 is planned.

In at least one embodiment, the work distribution unit communicates 3914 with one or more GPCs 3918 via the XBar 3920. In at least one embodiment, the XBar 3920 is an interconnection network that includes many units of the PPU 3900 with other units of the PPU 3900 couples and can be configured so that the work distribution unit 3914 with a specific GPC 3918 is coupled. In at least one embodiment, one or more other units of PPU 3900 over the hub 3916 connected to the XBar 3920.

In at least one exemplary embodiment, the tasks are carried out by the planning unit 3912 and managed by the Labor Distribution Unit 3914 to one of the GPCs 3918 cleverly. GPC 3918 is configured to process tasks and produce results. In at least one embodiment, the results of other tasks within GPC 3918 used, via XBar 3920 to another GPC 3918 forwarded or in memory 3904 get saved. In at least one embodiment, results can be across partition units 3922 into memory 3904 which is a memory interface for reading and writing data to / from memory 3904 to implement. In at least one embodiment, results can be sent over the high-speed GPU link 3908 to another PPU 3904 or CPU. In at least one embodiment, the PPU comprises 3900 without restriction a number U of partition units 3922 that is the number of those using the PPU 3900 coupled separate and distinct storage devices 3904 corresponds to. In at least one embodiment, the partition unit 3922 combined with 41 described in more detail herein.

In at least one embodiment, a host processor executes a driver kernel that implements an application programming interface ("API") that enables one or more applications running on the host processor to operate for execution on the PPU 3900 to plan. In at least one embodiment, multiple computing applications are run concurrently by PPU 3900 executed, and the PPU 3900 provides isolation, quality of service ("QoS"), and independent address spaces for multiple computing applications. In at least one embodiment, an application generates commands (e.g., in the form of API calls) that cause the driver kernel to perform one or more tasks for execution by the PPU 3900 to generate, and the driver kernel outputs tasks to one or more data streams, those from the PPU 3900 are processed. In at least one embodiment, each task includes one or more groups of related threads, which can be referred to as a warp. In at least one embodiment, a warp comprises a plurality of related threads (eg 32 threads) that can be executed in parallel. In at least one embodiment, cooperating threads can refer to a plurality of threads that contain instructions for performing tasks and exchange data via a shared memory. In at least one embodiment, threads and cooperating threads are used in accordance with at least one embodiment in conjunction with 41 described in more detail.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, a deep learning application processor is used to train a machine learning model, such as a neural network, to use that of the PPU 3900 Predict or infer the information provided. In at least one embodiment, the deep leaming application processor 3900 used to be based on a trained machine learning model (e.g. a neural network) generated by another processor or system or by the PPU 3900 has been trained to infer or predict information. In at least one embodiment, PPU 3900 can be used to perform one or more of the neural network use cases described here.

At least one embodiment can be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

40 FIG. 10 shows a general processing cluster (“GPC”) 4000 in accordance with at least one embodiment. In at least one embodiment, GPC is 4000 the GPC 3918 out 39 . In at least one embodiment, each includes GPC 4000 without limitation, a number of hardware units for processing tasks, and each GPC 4000 includes, without limitation, a pipeline manager 4002 , a Pre-Raster Operations Unit (“PROP”) 4004, a raster engine 4008 , a work distribution crossbar ("WDX") 4016, a memory management unit ("MMU") 4018, one or more data processing clusters ("DPCs") 4006 as well as any suitable combination of elements.

In at least one embodiment, GPC 4000 from the pipeline manager 4002 controlled. In at least one embodiment, the pipeline manager manages 4002 the configuration of one or more DPCs 4006 for processing the GPC 4000 assigned tasks. In at least one embodiment, the pipeline manager configures 4002 at least one of one or more DPCs 4006 to implement at least part of a graphics rendering pipeline. In at least one embodiment, DPC is 4006 configured to run a shading program on a 4014 Streaming Programmable Multiprocessor ("SM"). In at least one of the embodiments, the pipeline manager is 4002 configured to send packets received from a work distribution unit to appropriate logical units within GPC 4000 forwards, and some packets can be sent to hardware units with fixed functionality in PROP 4004 and / or the raster engine 4008 while other packets are forwarded to DPCs 4006 for processing by a primitive engine 4012 or SM 4014 can be forwarded. In at least one embodiment, the pipeline manager configures 4002 at least one of the DPCs 4006 to implement a model of a neural network and / or a computation pipeline.

In at least one embodiment, the PROP unit 4004 configured to use the engine 4008 and the DPCs 4006 generated data to a Raster Operations ("ROP") unit in the partition unit 3922 forwards that above in connection with 39 is described in more detail. In at least one embodiment, the PROP unit 4004 configured to perform color mixing optimizations, organize pixel data, perform address translations, and much more. In at least one embodiment, the raster engine comprises 4008 without limitation, a number of fixed functionality hardware units configured to perform various raster operations and the raster engine 4008 includes, without limitation, a setup engine, a coarse grid engine, a culling engine, a clipping engine, a fine grid engine, a tile coalescing engine, and any suitable combination thereof. In at least one embodiment, the setup engine receives transformed vertices and generates plane equations using a geometric primitive defined by vertices are associated; Plane equations are passed to the coarse raster engine to generate coverage information (e.g., an x, y coverage mask for a tile) for the primitive; the output of the coarse raster engine is passed to the culling engine, where fragments associated with the primitive that fail a z-test are hidden, and passed to a clipping engine where fragments that are outside a Cones of vision lie, are cut off. In at least one embodiment, the fragments that survive clipping and culling are passed to the fine grid engine to generate attributes for pixel fragments based on the plane equations generated by the setup engine. In at least one embodiment, the output includes the engine 4008 Fragments processed by any suitable entity, for example one in the DPC 4006 implemented fragment shader.

Everyone in GPC 4000 included DPC 4006 consists in at least one embodiment of, without limitation, an M-pipe controller (“MPC”) 4010; the primitive engine 4012 ; one or more SMs 4014 and any suitable combination thereof. In at least one embodiment, MPC controls 4010 the operation of DPC 4006 and directs those from the pipeline manager 4002 received packets to appropriate units in DPC 4006 further. In at least one embodiment, packets associated with a vertex are sent to the primitive engine 4012 forwarded configured to fetch vertex attributes associated with the vertex from memory; in contrast, packets associated with a shading program can be sent to SM 4014 to get redirected.

In at least one embodiment, SM 4014 without limitation, a programmable streaming processor configured to process tasks represented by a number of threads. In at least one embodiment, the SM 4014 multiple threads (multi-threaded) and is configured in such a way that it executes the multitude of threads (eg 32 threads) from a certain group of threads at the same time and implements a SIMD architecture (“single instruction, multiple data”) where each thread in a group of threads (e.g. a warp) is configured to process a different data set based on the same instruction set. In at least one embodiment, all of the threads in a group of threads execute the same instructions. In at least one embodiment, SM 4014 a SIMT ("Single-Instruction, Multiple Thread") architecture, where each thread in a group of threads is configured to process a different data set based on the same instruction set, but with individual threads in a group of threads during the Execution may diverge. In at least one embodiment, a program counter, call stack, and execution state are maintained for each warp, thereby enabling concurrency between warps and serial execution within warps when threads diverge within a warp. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, thereby enabling equal concurrency between all threads within and between warps. In at least one embodiment, the execution state is maintained for each individual thread, and threads executing the same instructions can be converged and executed in parallel for better efficiency. At least one embodiment of SM 4014 is described in more detail herein.

In at least one embodiment, the MMU 4018 an interface between GPC 4000 and the storage partition unit (e.g. partition unit 3922 of 39 ) and provides the MMU 4018 virtual address to physical address translation, memory protection, and memory request arbitration. In at least one embodiment, the MMU 4018 one or more translation lookaside buffers ("TLBs") for translating virtual addresses into physical addresses in memory.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, a deep learning application processor is used to train a machine learning model, such as a neural network, to predict or derive information, the GPC 4000 to provide. In at least one embodiment, the GPC 4000 used to be based on a trained machine learning model (e.g. a neural network) generated by another processor or system or by GPC 4000 has been trained to infer or predict information. In at least one embodiment, GPC 4000 can be used to perform one or more of the neural network use cases described here.

At least one embodiment can be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

41 Figure 13 shows a memory partition unit 4100 a parallel processing unit (“PPU”), according to at least one embodiment. In at least one embodiment, the memory partition unit comprises 4100 without restriction a raster operations unit ("ROP") 4102, an L2 cache (level two, "L2") 4104, a memory interface 4106 and any suitable combination thereof. The storage interface 4106 is coupled to the memory. The storage interface 4106 can implement 32-, 64-, 128-, 1024-bit data buses or the like for high-speed data transfer. In at least one embodiment, the PPU U contains memory interfaces 4106 , a memory interface 4106 per pair of partition units 4100 , with each pair of partition units 4100 is connected to a corresponding device. In at least one embodiment, the PPU can be connected, for example, to up to Y storage devices, such as storage stacks with high bandwidth or memories with double graphics data rate, version 5, synchronous dynamic random access memory (“GDDR5 SDRAM”).

In at least one embodiment, implements the memory interface 4106 a second generation memory interface (“HBM2”) with high bandwidth, where Y is half of U. In at least one exemplary embodiment, the HBM2 memory stacks are located in the same housing as the PPU, as a result of which considerable power and space savings are made available compared to conventional GDDR5 SDRAM systems. In at least one exemplary embodiment, each HBM2 stack comprises four memory modules without restriction and Y is equal to 4, with each HBM2 stack comprising two 128-bit channels per module for a total of 8 channels and a data bus width of 1024 bits. In at least one exemplary embodiment, the memory supports the Single Error Correcting Double Error Detecting (“SECDED”) Error Correction Code (“ECC”) to protect the data. ECC provides higher reliability for computing applications that are sensitive to data corruption.

In at least one embodiment, the PPU implements a multi-level memory hierarchy. In at least one embodiment, the memory partition device supports 4100 unified memory to provide a single unified virtual address space for the central processing unit (“CPU”) and PPU memory, thereby enabling data to be shared between virtual storage systems. In at least one embodiment, the frequency of accesses by a PPU to memory located on other processors is tracked to ensure that pages of memory are moved into physical memory of the PPU that is accessing pages more frequently. In at least one embodiment, the GPU supports high speed connectivity 3908 Address translation services which enable the PPU to access the page tables of a CPU directly and which enable the PPU to have full access to the CPU memory.

In at least one embodiment, the copy engines transfer data between multiple PPUs or between PPUs and CPUs. In at least one embodiment, copy engines can generate page faults for addresses that are not allocated in page tables and the memory partition unit 4100 then handles page faults by mapping addresses in page tables, after which the copy engine carries out the transfer. In at least one embodiment, the memory for multiple operations of the copy engines is pinned between multiple processors (ie, cannot be swapped out), which significantly reduces the available memory. In at least one embodiment with hardware page faults, addresses can be passed to the copy engines regardless of whether memory pages are resident, and the copying process is transparent.

Data from memory 3904 of 39 or some other system memory is used by the memory partition unit 4100 is retrieved and stored in L2 cache 4104, which is on-chip and shared by various GPCs in accordance with at least one embodiment. Any storage partition unit 4100 In at least one exemplary embodiment, comprises at least a part of the L2 cache that is assigned to a corresponding storage device, without restriction. In at least one embodiment, lower level caches are implemented in different units within GPCs. In at least one embodiment, each of the SMs 4014 implement a level 1 ("L1") cache, where the L1 cache is private storage reserved for a specific SM 4014 and data is fetched from the L2 cache 4104 and stored in each of the LI caches for processing in functional units of the SMs 4014 get saved. In at least one embodiment, the L2 cache 4104 is with the memory interface 4106 and the XBar 3920 coupled.

The ROP unit 4102 performs graphical rasterization operations on pixel colors, such as color compression, pixel blending, and more, in at least one embodiment. The ROP unit 4102 leads in at least one embodiment in connection with the raster engine 4008 performs a depth check and receives a depth for a sample location, which is a pixel fragment from the screening engine of the rasterization engine 4008 assigned. In at least one embodiment, the depth is tested against a corresponding depth in a depth buffer for a sample site associated with the fragment. In at least one embodiment, if the fragment passes the sample position depth test, the ROP unit updates 4102 the depth buffer and transmits the result of the depth test to the raster engine 4008 . It should be noted that the number of partition units 4100 can differ from the number of GPCs and therefore each ROP unit 4102 may be coupled to each of the GPCs in at least one embodiment. In at least one embodiment, the ROP unit tracks 4102 Packets that are received by different GPCs and defines which packets are sent to by the ROP unit 4102 generated result is forwarded via XBar 3920.

42 FIG. 3 shows a streaming multiprocessor (“SM”) 4200, in accordance with at least one exemplary embodiment. In at least one embodiment, SM 4200 the SM from 40 . In at least one embodiment, SM 4200 an instruction cache without restriction 4202 ; one or more planning units 4204 ; a register file 4208 ; one or more processing cores ("Cores") 4210; one or more special function units ("SFUs") 4212; one or more load storage units ("LSUs") 4214; an interconnection network 4216 ; a shared memory / level 1 ("L1") cache 4218 ; and any suitable combination thereof. In at least one embodiment, a work distribution unit distributes tasks for execution to general processing clusters (“GPCs”) of parallel processing units (“PPUs”), each task being associated with a specific data processing cluster (“DPC”) within a GPC and, if the task is associated with a shading program , one of the SMs 4200 is assigned. In at least one embodiment, the scheduler unit receives 4204 Tasks from the work distribution unit and manages the command scheduling for one or more thread blocks, the SM 4200 assigned. In at least one exemplary embodiment, the planning unit plans 4204 Thread blocks for execution as warps of parallel threads, with at least one warp being assigned to each thread block. In at least one embodiment, each warp threads. In at least one exemplary embodiment, the planning unit manages 4204 a multitude of different thread blocks, assigns the warps to different thread blocks and then sends commands from a multitude of different cooperative groups to different functional units (e.g. processing cores) during each clock cycle 4210 , SFUs 4212 and LSUs 4214 ).

In at least one embodiment, cooperative groups can refer to a programming model for organizing groups of communicating threads that allows developers to specify the granularity with which the threads communicate, which leads to the design of richer, more efficient parallel decompositions enables. In at least one embodiment, cooperative startup APIs support synchronization between thread blocks to execute parallel algorithms. In at least one embodiment, applications of conventional programming models are provided with a single, simple construct for synchronizing cooperating threads: a barrier over all threads of a thread block (e.g. the syncthreads () function). In at least one embodiment, however, programmers can define groups of threads with a granularity that is smaller than that of a thread block and synchronize within the defined groups in order to achieve greater performance, design flexibility and software reuse in the form of collective group-wide functional interfaces enable. In at least one embodiment, cooperative groups allow programmers to explicitly define groups of threads on sub-block (i.e., as small as a single thread) and multi-block granularities and to perform collective operations such as synchronizing threads in a cooperative group. The programming model supports a clean composition across software boundaries so that libraries and utility functions can be safely synchronized within their local context without having to make assumptions about convergence. In at least one embodiment, the cooperative group primitives enable new patterns of cooperative parallelism, including, without limitation, manufacturer-to-consumer parallelism, opportunistic parallelism, and global synchronization across a grid of thread blocks.

In at least one embodiment, there is a distribution unit 4206 configured to transmit instructions to one or more functional units and the scheduler unit 4204 includes, without limitation, two distribution units 4206 which make it possible to schedule two different instructions from the same chain during each clock cycle. In at least one embodiment, each scheduler comprises a unit 4204 a single distribution unit 4206 or additional distribution units 4206 .

In at least one embodiment, each includes SM 4200 in at least one embodiment, a register file 4208 , which is a set of registers for functional units of the SM 4200 provides. In at least one embodiment, the register file is 4208 distributed to each of the functional units so that each functional unit has a specific part of the register file 4208 assigned. In at least one embodiment, the register file is 4208 divided into different warps, those of SM 4200 and the register file 4208 is intended as a temporary memory for operands that are connected to data paths of functional units. In at least one embodiment, each includes SM 4200 without limitation, a variety of L processing cores 4210 . In at least one embodiment, SM 4200 A large number (eg 128 or more) of different processing cores, without limitation 4210 . In at least one embodiment, each includes processing core 4210 in at least one embodiment, without limitation, a full pipeline, single precision, double precision, and / or mixed precision processing unit that includes, without limitation, a floating point arithmetic and logic unit and an integer arithmetic and logic unit. In at least one embodiment, floating point arithmetic logic units implement the IEEE 754-2008 standard for floating point arithmetic. In at least one embodiment, the processing cores comprise 4210 without restriction 64 Single-precision floating-point cores (32-bit), 64 integer cores, 32 double-precision floating-point cores (64-bit), and 8 tensor cores.

Tensor cores are configured to perform matrix operations, according to at least one embodiment. In at least one embodiment, one or more tensor cores are in the processing cores 4210 contain. In at least one embodiment, the tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for training and inferencing neural networks. In at least one embodiment, each tensor kernel operates on a 4x4 matrix and performs a matrix multiply and accumulate operation D = AXB + C, where A, B, C, and D are 4x4 matrices.

In at least one embodiment, the inputs A and B to matrix multiplication are 16-bit floating point matrices and the accumulation matrices C and D are 16-bit floating point or 32-bit floating point matrices. In at least one embodiment, the tensor cores operate on 16-bit floating point input data with 32-bit floating point accumulation. In at least one embodiment, 16-bit floating point multiplication uses 64 operations and results in a full precision product which is then accumulated using 32-bit floating point addition with other intermediate products for 4x4x4 matrix multiplication. Tensor kernels are used to perform much larger two-dimensional or higher-dimensional matrix operations built from these smaller elements in at least one embodiment. In at least one embodiment, an API such as the CUDA 9 C ++ API provides special matrix load, matrix multiply and accumulate, and matrix store operations to efficiently use tensor cores from a CUDA C ++ program. In at least one embodiment, the warp-level interface at the CUDA level accepts 16x16 matrices that span all 32 threads of the warp.

In at least one embodiment, each includes SM 4200 without restriction M SFUs 4212 that fulfill special functions (e.g. attribute evaluation, reciprocal square root, etc.). In at least one embodiment, the include SFUs 4212 , without limitation, a tree traversal unit configured to traverse a hierarchical tree data structure. In at least one embodiment, the include SFUs 4212 without limitation, a texture engine configured to perform texture map filtering operations. In at least one embodiment, the texture units are configured to load texture maps (e.g., a 2D array of texels) from memory and sample texture maps to generate sampled texture values for use in shading programs developed by SM 4200 are executed. In at least one embodiment, the texture maps are in shared memory / L1 cache 4218 saved. In at least one exemplary embodiment, the texture units implement texture operations such as, for example, filter operations using mip maps (for example texture maps with different degrees of detail), in accordance with at least one exemplary embodiment. In at least one embodiment, each includes SM 4200 , without limitation, two texture units.

Every SM 4200 includes, without limitation, N LSUs 4214 , the load and store operations between shared memory / L1 cache 4218 and the register file 4208 implement, according to at least one embodiment. Every SM 4200 includes, without limitation, the interconnection network 4216 that each of the functional units with the register file 4208 and the LSU 4214 with the register file 4208 connects, and the shared memory / L1 cache 4218 , according to at least one embodiment. In at least one embodiment, the interconnection network is 4216 a crossbar, which can be configured to use any of the functional units with any of the registers in the register file 4208 and the LSUs 4214 with the register file 4208 and the locations in the shared memory / L1 cache 4218 connects.

In at least one embodiment, the shared memory is / L1 cache 4218 an array of on-chip memory that handles data storage and communication between the SM 4200 and the primitive engine as well as between threads in the SM 4200 allows, according to at least one embodiment. In at least one embodiment, the shared memory includes / L1 cache 4218 without restriction 128 KB storage capacity and is located in the path of the SM 4200 to the partition unit. In at least one embodiment, the shared memory is / L1 cache 4218 used in at least one embodiment for caching read and write operations. In at least one embodiment, one or more are shared memory / L1 cache 4218 , L2 cache and additional memory storage.

The combination of the functionalities of data cache and shared memory in a single memory block provides improved performance for both types of memory access, in accordance with at least one embodiment. In at least one embodiment, a capacity of programs that do not use shared memory is used as a cache or can be used as such, e.g. if the shared memory is configured so that it uses half the capacity, texture and loading / Storage operations use remaining capacity. Integration into a shared memory / L1 cache 4218 enables shared memory / L1 cache 4218 to act as a line with high throughput for a data stream and at the same time provide access with high bandwidth and low latency to frequently reused data, according to at least one embodiment. In at least one embodiment, when configured for general parallel computations, a simpler configuration compared to graphics processing can be used. In at least one exemplary embodiment, graphics processing units with fixed functionality are bypassed, as a result of which a much simpler programming model is created. In a general configuration for parallel computations, the work distribution unit in at least one exemplary embodiment assigns blocks of threads directly to the DPCs and distributes them. In at least one step in at least one embodiment, threads in a block execute the same program, using a unique thread ID in the computation to ensure that each thread produces unique results, the SM 4200 for executing programs and performing calculations, the shared memory / L1 cache 4218 for communication between threads, and and the LSU 4214 for reading and writing the global memory via the shared memory / Ll cache 4218 and the memory partition unit can be used. In at least one embodiment, the SM writes 4200 , when configured for general parallel calculations, commands the scheduler unit 4204 to start new work on DPCs.

In at least one embodiment, the PPU is in a desktop computer, a laptop computer, a tablet computer, servers, supercomputers, a smartphone (e.g. a wireless, portable device), a personal digital assistant ("PDA"), a digital camera , a vehicle, a head-mounted display, a portable electronic device, and more. In at least one embodiment, the PPU is formed on a single semiconductor substrate. In at least one embodiment, the PPU is in a system-on-a-chip (“SoC”) together with one or more other devices such as additional PPUs, memory, a CPU with reduced instruction set (“RISC”), a memory management unit (“MMU "), A digital-to-analog converter (" DAC ") and the like.

In at least one embodiment, the PPU can be included on a graphics card that includes one or more storage devices. The graphics card can be configured to connect to a PCIe slot on the motherboard of a desktop computer. In at least one embodiment, the PPU can be an integrated graphics processing unit (“iGPU”) that is included in the chipset of the motherboard.

The inference and / or training logic 1415 is used to perform inference and / or training operations associated with one or more embodiments. Details on the inference and / or training logic 1415 are here in connection with 14A and / or 14B provided. In at least one embodiment, a deep leaming application processor is used to train a machine learning model, such as a neural network, to work for SM 4200 Predict or infer any information provided. In at least one embodiment, SM 4200 used, on the basis of a trained machine learning model (e.g. a neural network) that is generated by another processor or system or by SM 4200 has been trained to infer or predict information. In at least one embodiment, the SM 4200 can be used to perform one or more of the neural network use cases described here.

At least one embodiment can be implemented using the techniques described above. In at least one embodiment, the networks for evaluating and generating a robot control system can be implemented using a processor as described above.

In at least one embodiment, a single semiconductor platform may refer to a single unitary integrated circuit or chip on a semiconductor basis. In at least one exemplary embodiment, multi-chip modules with increased connectivity can be used, which simulate on-chip operation and have significant improvements over the use of a conventional central processing unit (“CPU”) and a bus implementation. In at least one exemplary embodiment, different modules can also be arranged separately or in different combinations of semiconductor platforms, as desired by the user.

In at least one embodiment, computer programs are in the form of machine-readable executable code or algorithms of the computer control logic in the main memory 2004 and / or secondary storage. Computer programs, when executed by one or more processors, enable the system 2000 to perform various functions according to at least one embodiment. Storage 2004 , Memory, and / or any other storage are possible examples of computer readable media. In at least one embodiment, secondary storage may refer to any suitable storage device or system, such as a hard disk drive and / or a removable storage drive that includes a floppy disk drive, magnetic tape drive, compact disk drive, DVD drive, recording device, USB Flash memory (Universal Serial Bus), etc. In at least one exemplary embodiment, the architecture and / or functionality are various previous figures in connection with CPU 2002 , the parallel processing system 2012 , an integrated circuit that provides at least some of the capabilities of both CPUs 2002 can cover, the parallel processing system 2012 , a chipset (e.g., a group of integrated circuits designed and sold as a unit to perform related functions, etc.), and any suitable combination of integrated circuits.

In at least one embodiment, the architecture and / or functionality of various previous figures are implemented in the context of a general computer system, a circuit board system, a game console system for entertainment purposes, an application specific system, and more. In at least one embodiment, the computing device 2000 the form of a desktop computer, a laptop, a tablet computer, servers, supercomputers, a smartphone (e.g. a wireless, hand-held device), a personal digital assistant ("PDA"), a digital camera, a vehicle, one on the head attached display, handheld electronic device, cell phone, television, workstation, game consoles, embedded system, and / or any other type of logic.

In at least one embodiment, the parallel processing system comprises 2012 a variety of Parallel Processing Units ("PPUs") 2014 and associated memories, without limitation 2016 . In at least one embodiment, the are PPUs 2014 via an interconnect 2018 and a switch 2020 or multiplexers connected to a host processor or other peripheral devices. In at least one embodiment, the parallel processing system distributes 2012 Arithmetic tasks on the PPUs 2014 that can be parallelized, for example as part of the distribution of computing tasks over several thread blocks of the GPUs (Graphics Processing Unit). In at least one embodiment, memory (e.g., for read and / or write access) is shared across some or all of the PPUs 2014 shared and made accessible, although such shared memory has a performance penalty in terms of the use of local memory and registers contained in a PPU 2014 are resident. In at least one embodiment, the operation of the PPUs 2014 by using a command like _syncthreads (), all threads are synchronized in a block (e.g. across multiple PPUs 2014 across) to reach a specific point of code execution before proceeding.

Other variations are within the spirit of the present disclosure. Thus, while the disclosed techniques are susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof have been shown in the drawings and described in detail above been. It should be understood, however, that it is not intended to limit the disclosure to the particular form (s) disclosed, but on the contrary is intended to cover all modifications, alternative constructions, and equivalents that come within the spirit and scope of the disclosure, as defined in the appended claims.

The use of the terms “a” and “the” and similar references in connection with the description of apparent exemplary embodiments (in particular in connection with the following claims) is to be interpreted in such a way that they encompass both singular and plural, if used herein not otherwise stated or clearly contradicted by context, and not as a definition of a term. The terms "comprise", "have", "include" and "contain" are to be construed as unlimited terms (i.e., "including, but not limited to") unless otherwise specified. The term “connected” when it refers to physical connections and remains unchanged is to be construed as being included, in whole or in part, added to or connected to one another, even with something in between. The mention of value ranges in this document serves as a short process of individual reference to each individual value that falls within the range, unless otherwise stated herein and each individual value is included in the description as if it were individually recited would. The use of the term “sentence” (e.g. “a set of objects”) or “sub-clause” is to be understood as a non-empty set comprising one or more elements, unless otherwise indicated or indicated to the contrary in the context. Furthermore, the term “subset” of a corresponding sentence, unless otherwise stated or contradicting the context, does not necessarily denote a true subset of the corresponding set, but the subset and the corresponding set can be the same.

Conjunctive language, such as expressions of the form "at least one of A, B and C" or "at least one of A, B and C", is otherwise understood, unless expressly stated otherwise or otherwise clearly contradicted by the context, as is generally used in context to represent that item, term, etc., can be either A or B or C or any non-empty subset of theorem of A and B and C. For example, in an illustrative example of a three-member sentence, subjunctive clauses "at least one of A, B and C" and "at least one of A, B and C" refer to any of the following sentences: {A}, {B} , {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Such a subjunctive language is therefore generally not intended to imply that at least one of A, at least one of B and at least one of C must be present for certain exemplary embodiments. Unless otherwise indicated or indicated to the contrary by context, the term “plurality” also denotes a state of plurality (e.g. “a plurality of elements” denotes several elements). The number of elements in a plurality is at least two, but can be more if either explicitly or by context. Unless otherwise stated or otherwise clear from the context, the phrase “based on” further means “based at least in part on” and not “based solely on”.

Operation of the processes described herein can be performed in any suitable order, unless otherwise indicated herein or clearly described to the contrary in context. In at least one embodiment, a process such as the processes described herein (or variants and / or combinations thereof) is executed under the control of one or more computer systems configured with executable instructions and as code (e.g., executable instructions, one or more computer programs, or one or more applications) that runs collectively on one or more processors, by hardware, or combinations thereof. In at least one exemplary embodiment, the code is stored on a computer-readable storage medium, for example in the form of a computer program that comprises a multiplicity of instructions that can be executed by one or more processors. In at least one embodiment, a computer-readable storage medium is a non-volatile computer-readable storage medium that excludes volatile signals (e.g. propagating transient electrical or electromagnetic transmission), but non-volatile circuitry for data storage (e.g. buffers, cache and queues) within transceivers for volatile Includes signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media that stores executable instructions (or other storage for storing executable instructions) that when executed (i.e., as a result their execution) by one or more processors of a computer system cause the computer system to carry out the operations described here. Set of non-transitory computer-readable storage media comprises, in at least one embodiment, multiple non-transitory computer-readable storage media, and one or more individual non-volatile storage media of multiple non-transitory computer-readable storage media lacks all code, while multiple non-transitory computer-readable storage media collectively store all code. In at least In one embodiment, executable instructions are executed such that different instructions are executed by different processors - e.g. a non-transitory computer-readable storage medium stores instructions and a central processing unit ("CPU") executes some of the instructions, while a graphics processing unit ("GPU") executes other instructions . In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

Accordingly, in at least one embodiment, computer systems are configured to implement one or more services that individually or collectively perform operations of the processes described herein, and such computer systems are configured with suitable hardware and / or software that enable operations to be performed. Further, a computer system that implements at least one embodiment of the present disclosure is a single device, and in another embodiment is a distributed computer system that includes multiple devices that operate differently such that the distributed computer system performs the operations described herein and a single device is not performs all operations.

The use of examples or exemplary formulations (e.g. “such as”) merely serves to better illustrate exemplary embodiments of the disclosure and does not restrict the scope of the disclosure unless otherwise stated. Nothing in the specification should be construed as indicating any unclaimed element as essential to the practice of the disclosure.

All references cited herein, including publications, patent applications, and patents, are hereby incorporated by reference to the same extent as if each reference were individually and specifically given as incorporated by reference and given in their entirety.

In the description and in the claims, the terms “coupled” and “connected” can be used together with their derivatives. It should be understood that these terms cannot be used as synonyms for each other. Rather, the terms “connected” or “coupled” may be used in individual examples to indicate that two or more elements are in direct or indirect physical or electrical contact with one another. “Coupled” can also mean that two or more elements are not in direct contact with one another, but can still cooperate or interact with one another.

Unless expressly stated otherwise, it is to be understood that, throughout the description, terms such as “processing”, “calculating”, “calculating”, “determining” or the like refer to actions and / or processes of a computer or computer system or a similar electronic computing device relate to the data that are represented as physical, e.g. electronic, quantities in the registers and / or memories of the computing system, manipulate and / or transform into other data that are similar to physical quantities in the memories, registers or other such storage devices , Transmission or display of information.

Similarly, the term “processor” can refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that is stored in registers and / or memory can be. As a non-limiting example, “processor” can be a CPU or a GPU. A “computing platform” can include one or more processors. As used herein, “software” processes can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent processing programs. In addition, each process can refer to multiple processes to execute commands in sequence or in parallel, continuously or intermittently. The terms “system” and “process” are used interchangeably here insofar as a system can embody one or more processes and processes can be viewed as a system.

Reference may be made in this document to the procurement, acquisition, reception or input of analog or digital data into a subsystem, computer system or computer-implemented machine. The process of obtaining, capturing, receiving or entering analog and digital data can be carried out in various ways, for example by receiving data as parameters of a function call or a call to an application programming interface. In some implementations, the process of obtaining, capturing, receiving, or entering analog or digital data can be done by transmitting data over a serial or parallel interface. In a In another implementation, the process of obtaining, capturing, receiving or entering analog or digital data can be performed by the transmission of data over a computer network from the providing entity to the receiving entity. Reference can also be made to the provision, output, transmission, transmission or display of analog or digital data. In various examples, the process of providing, outputting, transmitting, sending or displaying analog or digital data can be achieved by transmitting data as input or output parameters of a function call, as parameters of an application programming interface or an interprocess communication mechanism.

Although exemplary implementations of the techniques described are listed in the discussion above, other architectures may be used to implement the functionality described, which are intended to fall within the scope of this disclosure. In addition, different roles and responsibilities may be allocated and divided in different ways depending on the circumstances, although specific descriptions of responsibilities have been defined above for the purposes of discussion.

Although subject matter has been described in language relating to structural features and / or methodological acts, it should be understood that subject matter claimed in the appended claims is not necessarily limited to the particular features or acts that are described. Rather, specific features and acts are disclosed as exemplary forms of implementing the claims.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Non-patent literature cited

• Standard No. J3016-201806 published June 15, 2018, Standard No. J3016-201609 published September 30, 2016 [0108]

Claims (25)

A computer implemented method comprising: Obtaining touch sensor information from a robot limb manipulating an object in the real world; Generating a plurality of simulations of the robot limb manipulating the object, the individual simulations having different poses for the object; Determining a plurality of costs, wherein each cost of the plurality of costs corresponds to a respective one of the plurality of simulations, and each cost of the plurality of costs is based at least in part on differences between the touch sensor information and simulated touch sensor information generated by the respective simulation of the plurality of simulations is produced; Identifying a single simulation of the plurality of simulations based at least in part on the cost; Determining a pose of the object in the real world based at least in part on a pose of the object in the identified individual simulation; and Providing the pose of the object to a robotic control system that controls a robot to perform a task based at least in part on the pose of the object. Computer-implemented method according to Claim 1 , further comprising updating one or more physical parameters of the plurality of simulations to reduce a difference between a simulation and an observation in the real world. Computer-implemented method according to Claim 1 or 2 , wherein the plurality of simulations are implemented using a GPU accelerated physics simulator. The computer-implemented method of any preceding claim, wherein the single simulation is identified by identifying a single simulation that generates simulated touch-sensor information that is most similar to the touch-sensor information. A computer-implemented method according to any one of the preceding claims, further comprising: Obtaining an initial pose estimate of the object; and Generating a plurality of possible poses to be used in the plurality of simulations, the plurality of possible poses being generated by modifying the initial pose estimate. Computer-implemented method according to Claim 5 wherein the initial pose estimate of the object is determined based on an image of the object obtained before the object is grasped by the robot limb. The computer-implemented method of any preceding claim, wherein the single simulation is identified by identifying a single simulation having the lowest associated cost. The computer-implemented method of any preceding claim, wherein the touch sensor information comprises a two-dimensional array of force values for each finger of the robot limb. System comprising: one or more processors; and computer readable memory that stores executable instructions that, when executed by the one or more processors, cause the system to: Obtaining data describing forces on a robot limb grasping an object; Generating simulations of the robot limb grasping the object, individual simulations of the simulations using different poses for the object; Determining a plurality of values, wherein each value of the plurality of values corresponds to a respective simulation of the plurality of simulations and each value of the plurality of values is based at least in part on differences between the forces and simulated-tactile simulated forces generated by the respective simulation of the plurality generated by simulations; Identifying a single simulation of the simulations based at least in part on the value; and Determining a pose of the object in the real world based at least in part on a pose of the object in the identified individual simulation. System according to Claim 9 wherein the executable instructions cause the system to further update one or more physical parameters of the simulations to reduce a difference between a state of a simulation and an observed state in the real world. System according to Claim 9 or 10 wherein the one or more processors comprise a graphics processing unit. System according to one of the Claims 9 to 11 wherein the single simulation is identified by identifying a single simulation that produces simulated data that most closely matches the data. System according to one of the Claims 9 to 12th wherein the executable instructions further cause the system to: obtain an initial pose of the object; and generating a plurality of poses to be used on objects in the simulations, the plurality of possible poses being generated by changing the initial pose. System according to Claim 13 wherein the initial object pose is determined using an image of the object obtained before the object is grasped by the robot limb. System according to one of the Claims 9 to 14th , where: the value is a measure of a difference between the forces and the simulated forces; and identifying the single simulation by identifying a single simulation with a lowest associated value. System according to one of the Claims 9 to 15th wherein the data is touch sensor information generated by a touch force sensor on each finger of the robot limb. Computer-readable medium that stores instructions which, when executed by one or more processors of a computer system, cause the computer system to: Obtaining data describing forces on a robot limb grasping an object; Performing simulations of the robot limb grasping the object, individual simulations of the simulations having different poses for the object; Determining a plurality of values, wherein each value of the plurality of values corresponds to a respective simulation of the plurality of simulations and each value of the plurality of values is based at least in part on differences between the forces and simulated-tactile simulated forces generated by the respective simulation of the plurality generated by simulations; Identifying a single simulation of the simulations based at least in part on the value; and Determining a pose of the object in the real world based at least in part on a pose of the object in the identified individual simulation. Computer readable medium according to Claim 17 wherein the instructions further cause the computer system to update one or more parameters of the simulations to reduce a difference between a state of a simulation and an observed state in the real world. Computer readable medium according to Claim 17 or 18th wherein the one or more processors comprise a multi-core graphics processing unit. Computer-readable medium according to one of the Claims 17 to 19th where the simulations are performed in parallel using a plurality of processors. Computer-readable medium according to one of the Claims 17 to 20th wherein the instructions further cause the computer system to: obtain an initial pose of the object; and generating a plurality of poses to be applied to objects in the simulations, the plurality of possible poses being generated by changing the initial pose. Computer readable medium according to Claim 21 wherein the initial pose of the object is determined using an image of the object obtained with a depth camera. Computer-readable medium according to one of the Claims 17 to 22nd , in which: the value is a measure of the difference between the forces and the simulated forces; and identifying the single simulation by identifying a single simulation with a lowest associated value. Computer-readable medium according to one of the Claims 17 to 23 wherein the data is touch sensor information generated by a touch force sensor on the robot limb. A robot comprising: an arm comprising one or more movable members connected by one or more servo motors; a robot limb connected to the arm, the robot limb having one or more touch sensors; one or more processors; and the computer readable medium of any of Claims 17 to 24 associated with the one or more processors.

Images