JP2021197151A - Object-to-robot pose estimation from single rgb image - Google Patents

Abstract

To provide methods and systems of performing object-to-object pose estimation from a single image.SOLUTION: A method comprises: identifying an image of a first object and a target object, the image captured by a camera external to the first object and the target object; processing the image, using a first neural network, to estimate a first pose of the target object with respect to the camera; processing the image, using a second neural network, to estimate a second pose of the first object with respect to the camera; and calculating a third pose of the first object with respect to the target object, using the first pose and the second pose.SELECTED DRAWING: Figure 1

Description

This application is a US patent application No. 16 / 405,662 (Reference: 741851 / 18-RE-0161) entitled "DETECTING AND ESTIMATING THE POS OF AN OBJECT USING A NEURAL NETWORK MODEL" filed on May 7, 2019. -US02) is a partial continuation application, and the US provisional patent application No. 62 / 672,767 (Reference: 18) entitled "DECTION AND POSESTITION OF HOUSEHOLD OBJECTS FOR HUMAN-ROBOT INTERACTION" filed on May 17, 2018. -RE-0161US01) claims priority, which are incorporated herein by reference in their entirety.

This application is part of US Patent Application No. 16 / 657,220 (Reference: 1R2674.06901 / 19-SE-0341US01) entitled "POSE DETERMINATION USING ONE OR MORE NEURAL NETWORKS" filed on October 18, 2019. It is a continuation application and is incorporated herein by reference in its entirety.

The present disclosure relates to a posture estimation system and method.

Posture estimation generally refers to computer vision techniques that determine the Euclidean position and orientation of an object, usually with respect to a particular camera. Posture estimation has many uses, but is particularly useful in the context of robotic manipulation systems. Historically, robotic manipulation systems have required a camera installed on the robot itself to capture the image of the object (ie, the camera at hand) and / or an external camera of the robot to capture the image of the object. I have been. In both cases, the camera must then be calibrated to the robot in order to estimate the attitude of the captured object to the robot.

Calibration of the camera at hand only needs to be done once to determine the position of the camera with respect to the robot, but because the camera is firmly placed on the robot, this camera has a limited field of view and therefore looks at the surrounding context. This also prevents it from being easily adjustable when the object moves from the camera's field of view. The external camera can have a wide field of view and can optionally assist the camera at hand, but the external camera needs to be calibrated every time it is moved. However, calibration is typically a tedious, delicate, and error-prone offline process. These problems are described specifically for robotic manipulation systems, but the same restrictions can behave to estimate the attitude of one object to another (ie, may or may not be moving). Note that this applies to any attitude estimation system.

There is a need to address these and / or other issues associated with the prior art.

Methods, computer-readable media, and systems for object-to-object attitude estimation from images are disclosed. In use, images of the first object and the target object are identified and the images are captured by a camera outside the first object and the target object. In addition, the image is processed using a first neural network to estimate the first attitude of the target object with respect to the camera. In addition, the image is processed using a second neural network to estimate the second orientation of the first object with respect to the camera. Furthermore, the third posture of the first object with respect to the target object is calculated using the first and second postures.

It is a figure of the method for estimating the posture from the object to the object from the image by one Example. It is a figure of the system for the posture estimation from the object to the object from the image by one Example. FIG. 2 is a block diagram associated with the first neural network of FIG. 2, according to an embodiment. FIG. 2 is a block diagram associated with a second neural network of FIG. 2, according to an embodiment. FIG. 3 is a diagram of a robotic gripping system controlled using a robot-to-object posture estimation system according to an embodiment. It is a figure which shows the reasoning and / or the training logic by at least one Example. It is a figure which shows the reasoning and / or the training logic by at least one Example. It is a figure which shows the training and development of a neural network by at least one Example. FIG. 3 illustrates an exemplary data center system according to at least one embodiment.

FIG. 1 illustrates a method 100 for estimating an object-to-object posture from an image according to one embodiment. Method 100 can be performed by any computing system, the computing system including one or more computing devices, one or more computer processors, non-temporary memory, circuits, and the like. Can be done. In one embodiment, the non-temporary memory may store instructions that can be executed by one or more computing devices and / or one or more computer processors to perform method 100. can. In another embodiment, the circuit may be configured to perform method 100.

As shown in operation 102, images of the first object and the target object are identified, and the images are captured by a camera outside the first object and the target object. In the context of this description, the target object and the first object are physically separate objects (eg, somewhat in close proximity to each other). In one embodiment, the first object can be a robotic gripping system with a robotic arm for gripping the object. In facilitating this embodiment, the target object can be a known object gripped by a robotic gripping system. In another embodiment, the first object can be another autonomous object, such as an autonomous vehicle, that interacts with (or makes related decisions about) a target object, such as another vehicle. However, of course, the first object and the target object may be any object whose posture between the two is desirable (eg, for computer vision applications).

As mentioned above, cameras outside the first and second objects capture images of the first and target objects. For this description, the camera is located outside the first object and the target object by being placed independently of the first object and the target object. For example, the camera may not be installed on either the first object or the target object. In one embodiment, the camera may be mounted on a tripod or other rigid surface to capture images of a first object and a target object.

The image of the target object may be a red-green-blue (RGB) image in one embodiment, or may be a grayscale image in another embodiment. The image may or may not include depth. In another embodiment, the image may be a single image. In a further embodiment, a combination of images obtained from various types of sensors such as red, green, blue, infrared, ultraviolet, and 6-channel images that can include radar. May be. Furthermore, the camera may be a monocular RGB camera. However, of course, the camera can capture any number of channels of light of any wavelength (visible or invisible to humans) and even non-light wavelengths. For example, cameras can utilize infrared, ultraviolet, microwave, radar, sonar, or other techniques to capture images.

In addition, as shown in motion 104, the image is processed using a first neural network to estimate the first pose of the target object with respect to the camera. The first neural network can be trained to output a two-dimensional (2D) image location (x, y coordinates) of the key points of the target object. These 2D image locations can then be used to estimate the attitude of the target object with respect to the camera along the 3D coordinates of the target object model. In one embodiment, the PnP algorithm can be used to calculate the attitude of the target object with respect to the camera. In another embodiment, the first orientation of the target object with respect to the camera can include three-dimensional (3D) rotation and translation of the target object with respect to the camera.

As an example, the first neural network is U.S. Patent Application No. 16 / 405,662 entitled "DECTING AND ESTIMATING THE POSE OF AN OBJECT USING A NEURAL NETWORK MODEL" filed May 7, 2019 (Reference: 741851 / 18-RE-0161-US02), which is incorporated herein by reference in its entirety. Further details relating to the first neural network embodiment are described below with reference to FIG.

Further, as shown in motion 106, the image is processed using a second neural network to estimate the second attitude of the first object with respect to the camera. The second neural network can be trained to output a two-dimensional (2D) image location (x, y coordinates) of the key points of the first object. These 2D image locations can then be used to estimate the orientation of the first object with respect to the camera along the 3D coordinates of the first object model. In one embodiment, the perceptive-n-point (PnP) algorithm can be used to calculate the attitude of the first object with respect to the camera. In another embodiment, the second orientation of the first object with respect to the camera can include 3D rotation and translation of the first object with respect to the camera.

Alternatively, a second neural network can perform online calibration of the camera. This online calibration allows the camera to move during runtime, including during operation of, for example, a robotic gripping system or other autonomous system.

As an example, the second neural network is U.S. Patent Application No. 16 / 657,220 entitled "POSE DETERMINATION USING ONE OR MORE NEURAL NETWORKS" filed on October 18, 2019 (Reference: 1R2674.0006901 / 19-). It may be disclosed in SE-0341US01), which is incorporated herein by reference in its entirety. Further details relating to the second neural network embodiment are described below with reference to FIG.

Furthermore, as shown in motion 108, the third posture of the first object with respect to the target object is calculated using the first and second postures. Therefore, the attitude estimation from the first object to the target object can be calculated. In one embodiment, the third posture can be the posture of the target object with respect to the coordinate frame of the first object. In another embodiment, the third posture is the reciprocal of one output of the neural network multiplied by another output of the neural network, eg, the second posture. , Or it can be calculated by multiplying the reciprocal of the second posture by the first posture.

In the context where the first object is a robotic gripping system, the robotic gripping system can be made to grip the target object (ie, a known object) using a further third posture. For example, a third posture is given to the robotic gripping system, which can allow the robotic gripping system to locate and grip the target object. Of course, in other embodiments, the third attitude is used for other purposes, including, for example, controlling the autonomous vehicle or having the autonomous vehicle make decisions based on the relative location of the target object. be able to.

As an option, the first and / or second posture can be refined prior to calculating the third posture. For example, the first pose can be refined by iteratively matching the image with the composite projection of the model by the first pose and then adjusting the parameters of the first pose based on the result of the iterative match. can. Similarly, the second pose is refined by iteratively matching the image with the composite projection of the model in the second pose and then adjusting the parameters of the second pose based on the result of the iterative match. Can be done.

For this purpose, the attitude between the first object and the target object can be estimated using two neural networks. In particular, one neural network can estimate the first attitude of the first object with respect to the camera, and the second neural network can estimate the second attitude of the first object with respect to the camera. Can be done. The output of the neural network (ie, the first and second poses) can then be used as the basis for determining the pose between the first object and the target object.

Further descriptive information will then be described in relation to various optional architectures and features that can implement the framework described above at the user's discretion. It should be noted in particular that the following information is provided for illustrative purposes and should not be construed as limiting in any way. Any of the following features may optionally be incorporated with or without exclusion of the other features described.

FIG. 2 illustrates a system 200 for object-to-object attitude estimation from an image according to one embodiment. For example, the system 200 can implement the method 100 of FIG. The system 200 may, in one embodiment, be located in the cloud and thus remotely with respect to the object. In another embodiment, the system 200 may be located on a computing device that is local to the object.

As shown, the system 200 includes a first module 201 including a first neural network 202, a second module 203 including a second neural network 204, and a processor 206. The image is input to the first neural network 202 and the second neural network 204. The image is of a first object and a target object and is captured by cameras outside both the first object and the target object. The camera may be outside the system 200. However, the image may be given to the system 200 by a camera, over a network, through shared memory, or in any other way.

The first neural network 202 receives the image as input and processes the image to estimate the first attitude of the target object with respect to the camera (ie, the attitude from the target object to the camera). The first neural network 202 may be a deep neural network that estimates the 6DoF orientation (eg, 3D rotation and translation) of a known object with respect to the camera. The network 202 may consist of a multi-stage convolutional network that converts the input RGB image into a set of belief maps, one for each key point. In one embodiment, the n = 9 key point is used to represent the vertices of the bounding rectangular parallelepiped, along with the center of gravity. In addition to the belief map, network 202 can output one n-1 affinity map for each key point that is not the center of gravity. Each map is a 2D field of unit vectors pointing to the closest object center of gravity. Maps can be used to individualize objects through post-processing steps, allowing the system to handle multiple instances of each object. The posture can be determined by a first module 201 that applies the PnP algorithm to the key points detected as peaks in the belief map.

In one embodiment, the input to network 202 may consist of 533x400 images processed by a VGG-based feature extractor, resulting in a 50x50x512 feature block. These features can be processed by a series of 6 stages (each containing 7 convolutional layers) that outputs and refines the belief map described above.

The second neural network 204 receives the image as input and processes the image to estimate the second attitude of the first object with respect to the camera (ie, the attitude from the first object to the camera). .. In one embodiment, the second neural network 204 may be a deep neural network that estimates the robot's 6DoF posture with respect to the camera. The network 204 may consist of an encoder-decoder that converts the input RGB image into a set of belief maps, one for each key point. Since there is only one robot in a scene, an affinity field may not be needed.

In one embodiment, key points placed on the joints of the robot may be defined to achieve postural stability when the arm is largely out of the camera's field of view, which allows the camera to view the scene from close range. Occurs when looking. In one embodiment, the input to network 204 is a 400x400 image, downsampled from the original 640x480 and center cropped. The layers of network 204 may be as follows: the encoder follows the same layer structure as the VGG, but the decoder is constructed from 4 upsampling layers followed by 2 convolution layers, with a key point belief map. give. The second module 203 applies the PnP algorithm to the key points detected as peaks in the belief map.

As an option, the first and / or second poses can be refined to reduce errors in estimation due to limited training data and / or network capacity. This refinement can be performed by adjusting the pose parameters by iteratively matching the input image with the composite projection of the model with the current pose.

Since the first neural network 202 and the second neural network 204 do not need each other's outputs, the first neural network 202 and the second neural network 204 operate in parallel if desired. It doesn't have to work. The output of each of the first neural network 202 and the second neural network 204 is given to the processor 206. Processor 206 uses the first and second poses to calculate the third pose of the first object with respect to the target object. The third posture can be calculated using Equality 1 as described in accordance with one embodiment.

は、ロボット・フレームにおける物体の姿勢であり、

は、カメラ・フレームにおけるロボットの姿勢であり（第２のネットワーク２０４によって計算される）、

は、カメラ・フレームにおける物体の姿勢である（第１のネットワーク２０２によって計算される）。 here,

Is the posture of the object in the robot frame,

Is the posture of the robot in the camera frame (calculated by the second network 204).

Is the pose of the object in the camera frame (calculated by the first network 202).

In an exemplary embodiment where the first object is a robotic gripping system and the target object is a known object, the processor 206 here uses a third posture to grip the target object into the robotic gripping system. Can be made to. Processor 206 is via a network such as the Internet (eg, if system 200 is located remotely to an object) or Ethernet® (eg, if system 200 is located locally to an object). Can communicate with the robotic gripping system.

Since the first neural network 202 only estimates the attitude of the target object with respect to the camera, only the first neural network 202 (not necessarily the second neural network 204) can be trained on the target object. In this way, the target object can be "known" to the first neural network 202, or in other words, a "known object". Of course, the first neural network 202 may also be trained on target objects of other categories or types.

Similarly, since the second neural network 204 only estimates the attitude of the first object with respect to the camera, only the second neural network 204 (not necessarily the first neural network 202) is the first. Can be trained on objects. In other words, the first object can be "known" to the second neural network 204. However, it should be noted that the second neural network 204 can also be trained on other objects (eg, other robots, or other autonomous objects).

To this end, in contrast to end-to-end learning, the modular approach presented in the system 200 allows the system to be reused without the need to retrain all networks. For example, in order to apply the system 200 to a new robot, only the second network 204 needs to be retrained. Similarly, in order to apply the system 200 to a new object, only the first network 202 needs to be trained on this object, independent of the robot and other objects. Moreover, this modular approach can facilitate testing and refinement of individual components to ensure accuracy and reliability.

FIG. 3 is a block diagram associated with the first neural network 202 of FIG. 2 according to an embodiment. Of course, the block diagram is described as one possible embodiment associated with the first neural network 202 of FIG. This example is U.S. Patent Application No. 16 / 405,662 (Reference: 741851 / 18-RE-) entitled "DECTING AND ESTIMATING THE POSE OF AN OBJECT USING A NEURAL NETWORK MODEL" filed on May 7, 2019. It is described in more detail in 0161-US02) and is incorporated herein by reference in its entirety.

In the embodiments shown, the attitude estimation system 300 includes a key point module 310, a set of multi-stage modules 305, and an attitude unit 320. The attitude estimation system 300 is described in the context of the processing unit, where one or more of the key point module 310, the set of multi-stage modules 305, and the attitude unit 320 may be a program, a custom circuit, or a custom circuit and program. Can be performed by a combination of. For example, the key point module 310 can be implemented by a GPU, a central processing unit (CPU), or any processor capable of processing an image to generate key point data.

The attitude estimation system 300 receives an image captured by a single camera. The image may contain one or more objects for detection. In one embodiment, in an image that does not have any depth data, the image contains pixel-by-pixel color data. The attitude estimation system 300 first detects the key points associated with the object and then estimates the 2D projection of the vertices that define the bounding volume surrounding the object associated with the key points. Key points can include the center of gravity of the object and the vertices of the bounding volume surrounding the object. The key points are not explicitly visible in the image, but are instead inferred by the attitude estimation system 300. In other words, the object of interest is visible in the image except for the portion of the object that can be occluded, and the key points associated with the object of interest are not explicitly visible in the image. The 2D location of the key points is estimated by the attitude estimation system 300 using only the image data. The attitude unit 320 restores the 3D orientation of the object using the estimated 2D location, camera-specific parameters, and object dimensions.

The key point module 310 receives an image containing an object and outputs an image feature. In one embodiment, the key point module 310 includes a multi-layered convolutional neural network (ie, a first network 202). In one embodiment, the key point module 310 is the first 10 layers of a Visual Geomery Group (VGG-19) neural network pre-trained using the ImageNet training database, followed by feature dimensions from 512 to 256. It also contains two 3x3 convolution layers to reduce from 256 to 128. The key point module 310 outputs one feature of the three channels for each channel (for example, RGB).

Image features are input to the set of multi-stage modules 305. In one embodiment, the set of multi-stage modules 305 is a first multi-stage module 305 configured to detect the center of gravity of the object, and an additional configured to detect the vertices of the bounding volume surrounding the object. Multi-stage modules 305 are included in parallel. In one embodiment, a set of multi-stage modules 305 includes a single multi-stage module 305 used to process image features in multiple paths to detect the centroids and vertices of sequential bounding volumes. In one embodiment, the multi-stage module 305 is configured to detect vertices without detecting the center of gravity.

Each multi-stage module 305 includes a T-stage belief map unit 315. In one embodiment, the number of stages is equal to 6 (eg, T = 6). The belief map unit 315-1 is the first stage, the belief map unit 315-2 is the second stage, and so on. The image features extracted by the key point module 310 are passed to each of the belief map units 315 in the multistage module 305. In one embodiment, the keypoint module 310 and the multistage module 305 receive an RGB image of size wxhx3 as input and a feedforward neural network that branches to produce two different outputs, eg, belief maps. That is, it includes a first neural network 202). In one embodiment, w = 640 and h = 480. The stages of the belief map unit 315 operate sequentially, and at this time, each stage (belief map unit 315) considers not only the image features but also the output of the immediately preceding stage.

The stages of the belief map unit 315 in each multi-stage module 305 generate a belief map for estimating a single 2D location associated with an object in the image. The first belief map contains the probability values for the center of gravity of the object, and the additional belief map contains the probability values for the vertices of the bounding volume surrounding the object.

In one embodiment, the 2D locations of the detected vertices are the 2D coordinates of the 3D bounding vertices, each surrounding the object and projected into the image space of the scene. Representing each object with a 3D bounding box defines an abstract representation of each object that is sufficient for postural estimation but is still irrelevant to the details of the object's shape. If the bounding volume is a 3D bounding box, nine multi-stage modules 305 can be used to generate belief maps for the center of gravity and eight vertices in parallel. The attitude unit 325 estimates the 2D coordinates of the 3D bounding box vertices projected into image space, and then uses either a conventional computer vision algorithm or another neural network to persective-n-. The object location and orientation in 3D space are inferred from the point (PnP). PnP estimates the pose of an object using a set of n-locations in 3D space and a projection of n-locations in image space. In one embodiment, the posture estimation system 300 estimates the 3D posture of a known object in the clutter from a single RGB image in real time.

In one embodiment, the stages of the belief map unit 315 are each stage of a convolutional neural network (CNN). If each stage is a CNN, each stage utilizes an increasingly increased receptive field as the data passes through the neural network. This property allows the stage of the belief map unit 315 to resolve the ambiguity by incorporating an increasing amount of context in the rear stage.

In one embodiment, the stage of the belief map unit 315 receives the 128-dimensional features extracted by the key point module 310. In one embodiment, the belief map unit 315-1 comprises three 3x3x128 layers and one 1x1x512 layer. In one embodiment, the belief map unit 315-2 has 1 × 1 × 9 layers. In one embodiment, the belief map units 315-3 to 315-T each receive a 153 dimensional input (128 + 16 + 9 = 153) before the 1x1x128 layer or the 1x1x16 layer. It is the same as the first stage except that it includes five 7 × 7 × 128 layers and one 1 × 1 × 128 layer. In one embodiment, each of the belief map units 315 is _{of size w / 8} and _{h / 8} , and the normalized linear unit (ReLU) activation functions are totally alternated.

FIG. 4 is a block diagram associated with the use of the second neural network 204 of FIG. 2 according to one embodiment. Of course, the block diagram is described as one possible embodiment associated with the second neural network 204 of FIG. This example relating to the second neural network 204 is a US patent application 16 / 657,220 entitled "POSE DETERMINATION USING ONE OR MORE NEURAL NETWORKS" filed on October 18, 2019 (Reference: 1R2674.7006901 /). 19-SE-0341US01), which is incorporated herein by reference in its entirety.

As shown, the captured image 402 of the robot is given to the trained neural network 204 as input. Although robots are described in the context of this embodiment, it should be noted that the block diagrams described herein are equally applicable to other objects (eg, other autonomous systems). Alternatively, some pre-processing or enhancement of this image may be performed prior to processing, such as to adjust resolution, color depth, or contrast. In at least one embodiment, the network 204 can be trained specifically for the type of robot 204, as different robots can have different shapes, sizes, configurations, kinematics, and features.

In use, the neural network 204 can analyze the input image 402 and output a set of belief maps 406 as a set of inferences. As an option, other dimensional inferences can be generated to locate feature points. For example, the neural network 204 can infer one belief map 406 for each robot feature identified.

In at least one embodiment, the robot model used for training can identify specific features to be tracked. These features can be learned through the training process. In another embodiment, the features may be located on various moving parts or components of the robot so that the posture of the robot can be determined from these features. In a further embodiment, the features are such that each posture of the robot corresponds to one and only one feature setting, and each feature setting corresponds to one and only one robot pose. Can be selected. This uniqueness can allow the camera-to-robot attitude to be determined based on the unique orientation of the features as represented in the captured image data.

With respect to network 204, the automatic encoder network can detect key points. In at least one embodiment, the neural network 204 receives as input an RGB image of size wxhx3 and outputs n belief maps 406 with form wxhxn. In at least one embodiment, RGBD or stereoscopic images can be similarly received as input. Arbitrarily, w = 640 and h = 480. In at least one embodiment, the output per key point is a 2D belief map, where the pixel value represents the likelihood that the key point will be projected onto that pixel.

In one embodiment, the encoder of network 204 includes a convolutional layer of VGG-19 pre-trained with ImageNet. In another embodiment, a ResNet-based encoder can be used. In a further embodiment, the decoder or upsampling component of the network 204 is composed of four 2D transposed convolutional layers, each layer followed by a normal 3x3 convolutional layer and a ReLU activation function. Furthermore, the output head may be composed of three convolutional layers (3 × 3, stride = 1, padding = 1) with ReLU activation on 64, 32, and n channels, respectively. In at least one embodiment, there may be no active layer after the last convolutional layer. In another embodiment, the encoder network can be trained using an L2 loss function that compares the output belief map to the ground truth belief map, where the correct belief map is Generated using σ = 2 pixels to generate the peak. In at least one embodiment, the use of stereoscopic image pairs can allow fusion of poses estimated from these images, or point clouds can be calculated, such as Procrustes analyze or iterative close point (ICP). Posture determined using processing.

In at least one embodiment, the belief map 406 may, as an input, be given to a peak extraction component 408, or service, capable of determining a set of coordinates in two dimensions representing the location of relevant robot features. .. As an option, the keypoint coordinates are the weighted average of the values near the peak, which is the threshold, after first applying Gaussian smoothing to these belief maps to reduce the noise effect in each belief map. Can be calculated as. This weighted average can allow for subpixel accuracy. In at least one embodiment, these two-dimensional coordinates (or pixel locations) can be provided as input to an attitude determination module such as the perceptive-n-point (PnP) module 414. This orientation module is camera-specific data 410, such as camera calibration information that can be used as input to account for image artifacts due to lens asymmetry, focal length, principal point, or other such factors. Can be accepted.

In at least one embodiment, the posture determination module can also receive, as an input, information about the forward kinematics 412 of this type of robot for determining possible postures. Kinematics can be used to narrow the search space where only certain feature locations are possible due to the physical configuration or limitations of this type of robot. This information can be analyzed using the PnP algorithm to output the determined camera-to-robot pose. In at least one embodiment, the perceptive-n-point is used to restore external factors of the camera, assuming that the joint configuration of this robot manipulator is known. This attitude information can be used to determine the relative distance and orientation between the camera and the robot, as the base coordinates or other features of the robot can be accurately identified in the camera space or camera coordinate system. can. To this end, the neural network 204 can be trained to infer the location of these features, such as by inferring a set of belief maps 406.

In at least one embodiment, relative position and orientation information is used to ensure that the camera coordinate space from the camera's point of view is aligned with the robot coordinate space of the robot, both in dimension and alignment. be able to. In at least one embodiment, these coordinates may be correct from the camera coordinate system but not in the robot coordinate system, so the robot 504 performs some action due to the incorrect orientation or position of the camera 502 with respect to the robot 504. Wrong coordinates may be given. For this purpose, the relative position and orientation of the camera 502 can be determined relative to the robot 504. In at least one embodiment, the relative position of the camera 502 may be sufficient, while the orientation information may be useful depending on factors such as camera specificity, in which case asymmetric image properties. Can affect accuracy if not properly considered. This relative position / orientation can be calibrated online (eg, during the robot's runtime).

FIG. 5 illustrates a robotic gripping system 500 controlled using a robot-to-object posture estimation system according to an embodiment. The robotic gripping system 500 can be implemented in the context of the embodiments described above.

As shown, the camera 502 can be used to capture an image, potentially in the form of a moving image frame of an autonomous object such as a robot 504. The camera 502 allows the robot 504 to be within the field of view 510 of the camera 502 and to allow the camera 502 to capture images that may include at least a partial representation of the robot 504 if it is not a complete view representation. , Can be positioned, or can be mounted externally. The captured images can be used to assist the robot 504 in commanding it to perform a particular task.

In at least one embodiment, the captured image can be analyzed to determine the location of the object 512 with respect to the robot 504, using that location for the robot 504 to lift or modify the object 512 in some way. And so on. In particular, the system 200 of FIG. 2 can be used to determine the attitude from the robot to the object for the purpose of giving accurate commands to the robot 504 or the control system for the robot 504. The robot-to-object posture can also be used for other purposes, such as assisting in navigating the robot 504 or providing current information about the state of the robot 504. In at least one embodiment, accurate position and orientation data between the robot 504 and the object 512 is such that the robot 504 grips and manipulates the object, interacts with the robot, and collides in an unstructured dynamic environment. Allows robotic operation while performing tasks such as detection and avoidance. Therefore, it is desirable to determine at least one of the position or orientation of the robot 504 with respect to the camera 502 and the position or orientation of the object 512 with respect to the camera 502, and then determine the posture from the robot to the object.

As mentioned above, an image showing the current orientation of the robot 504 with respect to the camera 502 and the current orientation of the object 512 with respect to the camera 502 can be captured by the camera 502. The robot 504 can have various articulated limbs 508 or components so that the robot 504 has different configurations or "postures" with respect to the orientation of the robot 504. In at least one embodiment, different postures of the robot 504 can result in different representations in the image captured by the camera 502. In at least one embodiment, a single image captured by the camera 502 can be analyzed to determine the features of the robot 504 that can be used to determine the posture of the robot 504. Features can correspond to joints or locations that the robot can move or adjust its position or orientation. Furthermore, since the dimensions and kinematics of the robot 504 are known, determining the attitude of the robot 504 from the viewpoint of the camera 502 makes it possible to accurately determine the distance and orientation from the camera to the robot. There is.

Developed with machine learning processors, Deep Neural Networks (DNN), which includes deep learning of models, is used for everything from self-driving cars to rapid drug discovery, from automatic image captioning in online image databases to video chat. -It is used in various use cases, including smart real-time language translation in applications. Deep learning is a technique that models the neural learning process of the human brain, continuously learning, continuously becoming smarter, and providing more accurate results in faster time. Children are initially taught by adults to correctly identify and classify various shapes and eventually to be able to identify shapes without any coaching. Similarly, deep learning or neural learning systems assign object context to objects while recognizing and classifying objects in order to identify basic objects, occluded objects, etc. smarter and more efficiently. Need to be trained about.

At the simplest level, neurons in the human brain look at the various inputs they receive, assign a level of importance to each of these inputs, and the output is passed to other neurons to act. Artificial neurons or perceptrons are the most basic models of neural networks. In one embodiment, the perceptron can receive one or more inputs that represent various features of the object that the perceptron is trained to recognize and classify, each of which features the shape of the object. In defining, a certain weight is assigned based on the importance of the feature.

Deep Neural Networks (DNN) models are of many connected nodes that can be trained with vast amounts of input data (eg, perceptrons, Boltzmann machines, radius-based functions, convolutional layers, etc.). It includes multiple layers and solves complex problems quickly with high accuracy. In one example, the first layer of the DNN model decomposes the input image of the car into various sections, looking for basic patterns such as lines and angles. The second layer assembles the lines and looks for higher order patterns such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the last few layers generate labels for the input image and identify the specific car brand model.

Once the DNN is trained, the DNN can be expanded and used to identify and classify objects or patterns in a process known as inference. Examples of inference (the process by which DNN extracts useful information from a given input) are identifying the handwritten numbers on a check deposited on an ATM machine, identifying the image of a friend in a photo, 50 million. Providing movie recommendations to these users, identifying and classifying various types of vehicles, pedestrians, and road hazards in unmanned vehicles, or translating human speech in real time.

During training, the data passes through the DNN in the forward propagation phase until a prediction indicating the label corresponding to the input is generated. If the neural network does not label the input correctly, the error between the correct label and the predicted label is analyzed and DNN takes that input and other inputs in the training data set during the backpropagation phase. The weights are adjusted for each feature until they are labeled correctly. Training complex neural networks requires a large amount of parallel computing performance, including floating-point multiplication and addition. Inferring is less computationally intensive than training, and what trained neural networks have seen before for image classification, speech translation, and general inference of new information. It is a latency-sensitive process that applies to no new inputs.

Reasoning and Training Logic As mentioned above, deep learning or neural learning systems need to be trained to generate inferences from input data. Details regarding inference and / or training logic 615 for deep learning or neural learning systems are given below, along with FIGS. 6A and / or 6B.

In at least one embodiment, the inference and / or training logic 615 corresponds to the neurons or layers of the neural network trained and / or used to infer in one or more embodiments. / Or output weights and / or data storage 601 for storing input / output data may be included without limitation. In at least one embodiment, the data storage 601 is one or more while forward propagating input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. Store weight parameters and / or input / output data for each layer of the neural network to be trained or used in conjunction with a plurality of embodiments. In at least one embodiment, any portion of the data storage 601 is included with the L1, L2, or L3 cache of the processor, or other on-chip or off-chip data storage including system memory. May be good.

In at least one embodiment, any portion of the data storage 601 may be inside or outside one or more processors, or other hardware logic devices or circuits. In at least one embodiment, the data storage 601 includes a cache memory, a dynamic random addressable memory (“DRAM”: dynamic random addressable memory), and a static random addressable memory (“SRAM”: static random addressable memory). ), Non-volatile memory (eg, flash memory), or other storage. In at least one embodiment, the choice of whether the data storage 601 is, for example, inside or outside the processor, or whether it consists of DRAM, SRAM, flash, or any other type of storage. Storage available on-chip vs. off-chip, latency requirements for training and / or inference functions performed, batch size of data used in inference and / or training in neural networks, or factors thereof. It may be decided according to the combination of the above.

In at least one embodiment, the inference and / or training logic 615 corresponds to the neurons or layers of the neural network trained and / or used to infer in one or more embodiments. / Or output weights and / or data storage 605 for storing input / output data may be included without limitation. In at least one embodiment, the data storage 605 is one or more while backpropagating input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. Store weight parameters and / or input / output data for each layer of the neural network to be trained or used in conjunction with a plurality of embodiments. In at least one embodiment, any portion of the data storage 605 is included with the L1, L2, or L3 cache of the processor, or other on-chip or off-chip data storage including system memory. May be good. In at least one embodiment, any part of the data storage 605 may be inside or outside of one or more processors, or other hardware logic devices or circuits. In at least one embodiment, the data storage 605 may be cache memory, DRAM, SRAM, non-volatile memory (eg, flash memory), or other storage. In at least one embodiment, the choice of whether the data storage 605 is, for example, inside or outside the processor, or whether it consists of DRAM, SRAM, flash, or any other type of storage. Storage available on-chip vs. off-chip, latency requirements for training and / or inference functions performed, batch size of data used in inference and / or training in neural networks, or factors thereof. It may be decided according to the combination of the above.

In at least one embodiment, the data storage 601 and the data storage 605 may have separate storage structures. In at least one embodiment, the data storage 601 and the data storage 605 may have the same storage structure. In at least one embodiment, the data storage 601 and the data storage 605 may have a partially the same storage structure and a partially separate storage structure. In at least one embodiment, any portion of the data storage 601 and the data storage 605 is a cache of L1, L2, or L3 of the processor, or other on-chip or off-chip data including system memory. -May be included with storage.

In at least one embodiment, the inference and / or training logic 615 is one or more for performing logical and / or arithmetic operations that are at least partially based on or indicated by the training and / or inference code. The arithmetic logic unit (“ALU”) 610 may be included without limitation, and the result is stored in the activation storage 620 for activation (eg, output value from a layer or neuron in a neural network). These are functions of the input / output and / or weight parameter data stored in the data storage 601 and / or the data storage 605. In at least one embodiment, the activation stored in the activation storage 620 is generated according to the linear algebraic and / or matrix-based calculations performed by the ALU610 in response to the execution of an instruction or other code. Here, the weight value stored in the data storage 605 and / or the data 601 is used as an operand together with other values such as bias value, gradient information, momentum value, or other parameters or hyperparameters, and any of these. Or all may be stored in data storage 605 or data storage 601 or another storage on-chip or off-chip. In at least one embodiment, the ALU610 is contained within one or more processors, or other hardware logic devices or circuits, whereas in another embodiment, the ALU610 is a processor or other hardware that uses them. It may be outside the logic device or circuit (eg, a coprocessor). In at least one embodiment, the ALU610 may be contained within an execution unit of a processor, or may be within the same processor or with different types of different processors (eg, central processing unit, graphics processing unit, fixed function unit). Etc.), which may otherwise be contained within an ALU bank accessible by the execution unit of the processor, which is either distributed among. In at least one embodiment, the data storage 601, data storage 605, and activation storage 620 may be in the same processor or other hardware logical device or circuit, in another embodiment they are different processors. Or it may be in another hardware logic device or circuit, or it may be in some combination of the same processor or other hardware logic device or circuit with a different processor or other hardware logic device or circuit. In at least one embodiment, any portion of the activated storage 620 is included with the L1, L2, or L3 cache of the processor, or other on-chip or off-chip data storage including system memory. May be good. In addition, inference and / or training code may be stored with the processor or other code accessible to other hardware logic or circuits, including processor fetching, decoding, scheduling, execution, retirement, and / or other. It may be fetched and / or processed using logic circuits.

In at least one embodiment, the activation storage 620 may be cache memory, DRAM, SRAM, non-volatile memory (eg, flash memory), or other storage. In at least one embodiment, the activation storage 620 may be completely or partially inside or outside one or more processors or other logic circuits. In at least one embodiment, the choice of whether the activated storage 620 is, for example, inside or outside the processor, or whether it consists of DRAM, SRAM, flash, or any other type of storage. Available storage of on-chip vs. off-chip, latency requirements for training and / or inference functions performed, batch size of data used in inference and / or training in neural networks, or these factors. It may be decided according to the combination from any. In at least one embodiment, the inference and / or training logic 615 shown in FIG. 6A is a Tensorflow® processing unit from Google, an inference processing unit (IPU) from a Graphcore®, or an Intel Corp. It may be used in conjunction with an application specific integrated circuit (“ASIC”) such as a Nervana® (eg, “Lake Crest”) processor from. In at least one embodiment, the reasoning and / or training logic 615 shown in FIG. 6A is a central processing unit (“CPU”: central processing unit) hardware, graphics processing unit (“GPU”: graphics processing unit) hardware. It may be used in conjunction with hardware or other hardware such as a field programmable gate array (“FPGA”: field programmable gate array).

FIG. 6B shows inference and / or training logic 615 according to at least one embodiment. In at least one embodiment, inference and / or training logic 615 may include, without limitation, hardware logic, in which computational resources are one or more layers of neurons in a neural network. Dedicated to the corresponding weight value or other information, or used only in combination with them in other ways. In at least one embodiment, the inference and / or training logic 615 shown in FIG. 6B is a Tensorflow® processing unit from Google, an inference processing unit (IPU) from Graphcore ™, or Nervana from Intel Corporation. It may be used in conjunction with an application specific integrated circuit (ASIC) such as a registered trademark) (eg, "Lake Crest") processor. In at least one embodiment, the inference and / or training logic 615 shown in FIG. 6B is a central processing unit (CPU) hardware, a graphics processing unit (“GPU”) hardware, or a field programmable gate array. It may be used in combination with other hardware such as (FPGA). In at least one embodiment, the inference and / or training logic 615 includes, without limitation, data storage 601 and data storage 605, which are used to weight values and / or bias values, gradient information, and so on. Other information may be stored, including momentum values and / or other parameters or hyperparameter information. In at least one embodiment shown in FIG. 6B, data storage 601 and data storage 605 are each associated with dedicated computational resources such as computational hardware 602 and computational hardware 606, respectively. In at least one embodiment, each of the computational hardware 606 performs one or more mathematical functions, such as linear algebraic functions, only on the information stored in data storage 601 and data storage 605, respectively. It is equipped with an ALU and the result is stored in the activation storage 620.

In at least one embodiment, the data storages 601 and 605, and the corresponding computing hardware 602 and 606, respectively, correspond to different layers of the neural network, thereby with the data storage 601 and the computing hardware 602, respectively. The resulting activation from one "storage / computational pair 601/602" of "with the following data storage 605 and computational hardware 606" to reflect the conceptual organization of the neural network. Provided as an input to "Storage / Calculation vs. 605/606". In at least one embodiment, storage / computational pairs 601/602 and 605/606 may correspond to two or more layers of neural networks. In at least one embodiment, an additional storage / calculation pair (not shown) is included in the inference and / or training logic 615 after, or in parallel with, the storage / calculation pair 601/602 and 605/606. It may be.

Training and Introducing Neural Networks Figure 7 shows another embodiment for training and introducing deep neural networks. In at least one embodiment, the untrained neural network 706 is trained using the training data set 702. In at least one embodiment, the training framework 704 is the PyTorch framework, while in the other embodiment, the training framework 704 is the Tensorflow, Boost, Cafe, Microsoft CognitiveToolkit / CNT, MXNet, Chainer, Keras, Depple. , Or other training framework. In at least one embodiment, the training framework 704 trains an untrained neural network 706, allowing it to be trained using the processing resources described herein, a trained neural network. Generate 708. In at least one embodiment, the weights may be randomly selected or may be selected by pre-training using a deep belief network. In at least one embodiment, training may be performed in a supervised, partially supervised, or unsupervised manner.

In at least one embodiment, the untrained neural network 706 is trained using supervised learning, where the training data set 702 contains an input paired with the desired output to the input, or training data. The set 702 includes inputs with known outputs, the outputs of the neural network 706 are manually graded. In at least one embodiment, the untrained neural network 706 is trained in a supervised manner, processing inputs from training data set 702, and using the resulting output as the expected or desired set of outputs. compare. In at least one embodiment, the error is then backpropagated through the untrained neural network 706. In at least one embodiment, the training framework 704 adjusts the weights that control the untrained neural network 706. In at least one embodiment, the training framework 704 is a trained neural network suitable for the untrained neural network 706 to generate the correct answer in results 714, etc., based on known input data, such as new data 712. Includes tools to monitor how well converged towards a model such as 708. In at least one embodiment, the training framework 704 iteratively trains the untrained neural network 706, while using tuning algorithms such as the loss function and stochastic gradient descent to output the untrained neural network 706. Adjust the weights to refine. In at least one embodiment, the training framework 704 trains the untrained neural network 706 until the untrained neural network 706 reaches the desired accuracy. In at least one embodiment, a trained neural network 708 can then be introduced to implement any number of machine learning actions.

In at least one embodiment, the untrained neural network 706 is trained using unsupervised learning, where the untrained neural network 706 attempts to train itself using unlabeled data. In at least one embodiment, the training data set 702 for unsupervised learning includes input data without any relevant output data or "ground truth" data. In at least one embodiment, the untrained neural network 706 can learn grouping within the training data set 702 and how the individual inputs relate to the untrained data set 702. It can be determined. In at least one embodiment, unsupervised training can be used to generate a self-organizing map, which can perform useful actions to reduce the dimensions of the new data 712. A type of trained neural network 708. In at least one embodiment, unsupervised training can also be used to perform anomaly detection, which identifies data points in the new data set 712 that deviate from the normal pattern of the new data set 712. It can be so.

In at least one embodiment, semi-supervised learning may be used, which is a technique in which labeled and unlabeled data are mixed in the training data set 702. In at least one embodiment, the training framework 704 may be used to perform gradual learning, such as by communication learning techniques. In at least one embodiment, gradual learning allows the trained neural network 708 to adapt to the new data 712 without forgetting the knowledge taught in the network during the initial training.

Data Center FIG. 8 shows an exemplary data center 800 in which at least one embodiment may be used. In at least one embodiment, the data center 800 includes a data center infrastructure layer 810, a framework layer 820, a software layer 830, and an application layer 840.

In at least one embodiment, as shown in FIG. 8, the data center infrastructure layer 810 is a resource orchestrator 812, a grouped computing resource 814, and a node computing resource (“Node C.I. R. ": node computing resources) 816 (1) to 816 (N) may be included, where" N "represents an arbitrary positive integer. In at least one embodiment, Node C.I. R. 816 (1)-816 (N) are any number of central processing units (“CPUs”) or other processors (including accelerators, field programmable gate arrays (FPGAs), graphics processors, etc.). Memory devices (eg, dynamic read-only memory), storage devices (eg, semiconductor or disk drives), network input / output (“NWI / O”: network input / output) devices, network switches, virtual It may include, but is not limited to, a machine (“VM”: virtual machine), a power supply module, and a cooling module. In at least one embodiment, Node C.I. R. One or more nodes C. of 816 (1) to 816 (N). R. May be a server having one or more of the computing resources described above.

In at least one embodiment, the grouped computing resource 814 is housed in one or more racks (not shown). R. It may contain multiple racks housed in a data center in separate groups of, or in various graphical locations (also not shown). Node C. in grouped computing resources 814. R. Separate groups of may include grouped compute resources, network resources, memory resources, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, some nodes C.I. R. May be grouped within one or more racks to provide compute resources to support one or more workloads. In at least one embodiment, the rack may also include any number of power supply modules, cooling modules, and network switches in any combination.

In at least one embodiment, the resource orchestra 822 has one or more nodes C.I. R. 816 (1)-816 (N) and / or grouped computing resources 814 may be configured or controlled in other ways. In at least one embodiment, the resource orchestrator 822 may include a software design infrastructure (“SDI”: software design infrastructure) management entity for the data center 800. In at least one embodiment, the resource orchestrator may include hardware, software, or any combination thereof.

In at least one embodiment shown in FIG. 8, framework layer 820 includes job scheduler 832, configuration manager 834, resource manager 836, and distribution file system 838. In at least one embodiment, the framework layer 820 may include software 832 of software layer 830 and / or a framework for supporting one or more applications 842 of application layer 840. In at least one embodiment, software 832 or application 842 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure, respectively. In at least one embodiment, framework layer 820 can use the distributed file system 838 for large-scale data processing (eg, "big data") Apache Spark® (hereinafter "Spark"). ), Etc., may be a type of free and open source software web application framework, but is not limited to this. In at least one embodiment, Jobscheduler 832 may include a Spark driver to facilitate scheduling of workloads supported by various layers of data center 800. In at least one embodiment, configuration manager 834 may configure different layers such as software layer 830, as well as framework layer 820 including Spark and distribution file system 838 to support large-scale data processing. There may be. In at least one embodiment, resource manager 836 is capable of managing clustered or grouped computing resources that are mapped or allocated to support distribution file system 838 and job scheduler 832. It may be. In at least one embodiment, the clustered or grouped computing resource may include the grouped computing resource 814 at the data center infrastructure layer 810. In at least one embodiment, the resource manager 836 may work with the resource orchestrator 812 to manage these mapped or allocated computing resources.

In at least one embodiment, the software 832 included in the software layer 830 is the node C.I. R. 816 (1)-816 (N), grouped computing resources 814, and / or software used by at least a portion of the distributed file system 838 of framework layer 820 may be included. One or more types of software may include, but is not limited to, Internet web page search software, email virus scanning software, database software, and streaming video content software.

In at least one embodiment, the application 842 included in the application layer 840 is the node C.I. R. 816 (1)-816 (N), grouped computing resources 814, and / or one or more types of applications used by at least a portion of the distributed file system 838 of framework layer 820. It may be included. One or more types of applications include any number of genomics applications, cognitive compute, and machine learning framework software, including training or inference software, machine learning framework software (eg, PyTorch, Tensorflow, Cafe, etc.). Alternatively, it may include, but is not limited to, other machine learning applications used in conjunction with one or more embodiments.

In at least one embodiment, any one of the configuration manager 834, the resource manager 836, and the resource orchestrator 812 is based on any amount and type of data obtained in any technically viable manner. , Any number and type of self-correction measures may be implemented. In at least one embodiment, self-correction measures prevent the data center operator of the data center 800 from determining potentially defective configurations, and are underutilized and / or poorly performing data centers. You may be able to eliminate the part of.

In at least one embodiment, the data center 800 trains one or more machine learning models, or uses one or more machine learning models according to one or more embodiments described herein. May include tools, services, software, or other resources for predicting or inferring information. For example, in at least one embodiment, the machine learning model may be trained by computing weight parameters according to a neural network architecture using the software and computing resources described above for the data center 800. .. In at least one embodiment, the trained machine learning model corresponding to one or more neural networks is data by using the weighting parameters calculated by one or more techniques described herein. The resources described above for the center 800 may be used to infer or predict information.

In at least one embodiment, the data center uses a CPU, application specific integrated circuit (ASIC), GPU, FPGA, or other hardware to perform training and / or inference using the resources described above. You may use it. In addition, one or more software and / or hardware resources mentioned above are used to enable the user to perform training or inference of information such as image recognition, speech recognition, or other artificial intelligence services. It may be configured as a service.

Inference and / or training logic 615 is used to perform inference and / or training operations related to one or more embodiments. In at least one embodiment, the inference and / or training logic 615 uses the training behavior of the neural network described herein, the function and / or architecture of the neural network, or the use case of the neural network. It may be used in the system of FIG. 8 for inference or prediction behavior, at least partially based on the calculated weight parameters.

As described herein, methods, computer readable media, and systems for estimating object-to-object attitudes using images of externally captured objects are disclosed. According to FIGS. 1-4, the embodiment can provide a neural network that can be used to perform inferring behavior and to provide inferred data, FIG. 6A and. As depicted in FIG. 6B, the neural network is stored in one or both of the data storages 601 and 605 (partially or wholly) within the inference and / or training logic 615. .. Training and deployment of neural networks can be performed as depicted in FIG. 7 and as described herein. The distribution of the neural network is depicted in FIG. 8 and can be performed using one or more servers in the data center 800 as described herein.

Claims (20)

A step of identifying an image of a first object and a target object, wherein the image is captured by a camera outside the first object and the target object.
A step of processing the image using a first neural network to estimate a first attitude of the target object with respect to the camera.
A step of processing the image using a second neural network to estimate a second orientation of the first object with respect to the camera.
A method comprising the step of calculating a third posture of the first object with respect to the target object using the first posture and the second posture. The method according to claim 1, wherein the image is a red-green-blue (RGB) image or a grayscale image. The method of claim 1, wherein the camera captures one of a wavelength of light or a wavelength of non-light. The method of claim 1, wherein the first object is a robotic gripping system. The method of claim 4, wherein the target object is a known object that is gripped by the robotic gripping system. 5. The method of claim 5, further comprising having the robotic gripping system grip the known object using the third posture. The method of claim 1, wherein the first attitude of the target object with respect to the camera comprises three-dimensional (3D) rotation and translation of the target object with respect to the camera. The method of claim 1, wherein the second posture of the first object with respect to the camera comprises 3D rotation and translation of the first object with respect to the camera. The method of claim 1, wherein the second neural network performs online calibration of the camera. The method according to claim 1, wherein the third posture is the posture of the target object with respect to the coordinate frame of the first object. The method of claim 1, wherein only the first neural network is trained on the target object. The method of claim 1, wherein only the second neural network is trained on the first object. The step of refining the first posture and
The method of claim 1, further comprising the step of refining the second posture. The step of refining the first posture is
13. According to claim 13, the image is repeatedly matched with the synthetic projection of the model by the first posture, and the parameter of the first posture is adjusted based on the result of the iterative matching. The method described. The step of refining the second posture is
13. Claim 13 is performed by iteratively matching the image with the synthetic projection of the model in the second posture and adjusting the parameters of the second posture based on the result of the iterative matching. The method described. A non-temporary computer-readable medium that, when executed by one or more processors, stores computer instructions that cause the one or more processors to perform the method.
A step of identifying an image of a first object and a target object, wherein the image is captured by a camera outside the first object and the target object.
A step of processing the image using a first neural network to estimate a first attitude of the target object with respect to the camera.
A step of processing the image using a second neural network to estimate a second orientation of the first object with respect to the camera.
A non-temporary computer-readable medium comprising the step of calculating a third posture of the first object with respect to the target object using the first posture and the second posture. A first neural network that receives images of a first object and a target object as input and processes the images to estimate a first attitude of the target object with respect to the camera, wherein the image is the first. A first neural network captured by a camera outside the object 1 and the target object.
A second neural network that receives the image as input and processes the image to estimate a second orientation of the first object with respect to the camera.
A system comprising a processor that calculates a third posture of the first object with respect to the target object using the first posture and the second posture. 17. The system of claim 17, wherein the camera is outside the system. 17. The system of claim 17, wherein the first object is a robotic gripping system and the target object is a known object. 19. The system of claim 19, wherein the processor causes the robotic gripping system to grip the target object using the third posture.

Images