JP7129065B2 - Object pose detection from image data - Google Patents

Description

The present invention relates to pose detection. More specifically, the present invention relates to pose determination functions trained on simulations of computer model poses.

The manufacturing of products employs an increasing number of robotics techniques. For example, an assembly line may employ robotic arms that detect, pick up, and group parts in assembling the final product. Human intervention may be increased to reduce the programming burden. For example, by manually placing the parts in proper position and orientation, the robotic arm needs only minimal detection capabilities. As robotic arms increase their ability to detect and manipulate objects, human intervention may be reduced, which may also reduce manufacturing costs.

To effectively manipulate an object, a robotic system recognizes how such an object is placed in 6D space (which can be defined by its position along and orientation around 3 axes). I need to be able to. In order to train such a robot system and evaluate its performance, a large amount of training data containing a lot of environmental diversity must be obtained. Designers of such robotic systems face challenges in attempting to maximize accuracy while keeping both run time and data modality requirements low.

According to one aspect of the invention, there is provided a computer program executable by a computer, comprising: obtaining a computer model of an object; simulating the computer model with a realistic environment simulator; wherein each pose representation includes an image of the computer model in one of a plurality of poses, one of the plurality of poses being a pose specification of the computer model shown in the image , and the image and pose specification of the computer model are specified by the simulator, the capture procedure, and the pose determination for applying a learning process to the pose representation and associating the image of the object with the pose specification A computer program is provided for performing an operation comprising generating a function.

This aspect may also include a computer-implemented method executing the instructions of a computer program, and apparatus for performing the method.

It is noted that the above summary of the invention is not an exhaustive list of all required features of embodiments of the invention. Sub-combinations of the above features may also constitute the invention.

FIG. 4 shows a diagram of the interaction between hardware and software elements from CAD model to corrected pose detection, according to one embodiment of the present invention.

4 illustrates an exemplary hardware configuration for pose detection, according to one embodiment of the present invention;

FIG. 4 illustrates an operational flow for pose detection, according to an embodiment of the present invention; FIG.

FIG. 4 illustrates an operational flow for simulation of a computer model for populating training data, according to one embodiment of the present invention; FIG.

FIG. 4 illustrates an operational flow for generating a pose determination function, according to one embodiment of the present invention; FIG.

4 illustrates an operational flow for determining pose designations, according to one embodiment of the present invention;

Illustrative embodiments of the invention are subsequently described. The illustrated embodiments do not limit the claimed invention, and the combination of features described in the embodiments is not necessarily essential to the invention.

FIG. 1 shows a diagram of the interaction between hardware and software elements from CAD model to corrected pose detection, according to one embodiment of the present invention. This figure shows a multi-step approach consisting of simulation, deep learning, and traditional computer vision. In this embodiment, a computer aided design (CAD) model 112 is obtained. CAD model 112 may be created from a 3D scan of the object or may be manually designed. In some cases, such as assembly lines, CAD models may already be created and may simply be reused.

One or more instances of CAD model 112 are used by simulator 104 to create images of instances of CAD model 112 in random poses. A simulator 104 may be used so that the actual pose of each instance of the CAD model 112, sometimes referred to as the "ground truth", can be readily output from the simulator. In this way, manual derivation of the actual pose is no longer necessary, which can be very tedious and time consuming. Random poses for each instance of CAD model 112 may be achieved by dropping, colliding, shaking, stirring, etc. objects within the simulation. Simulator 104 uses a physics engine to simplify these operations. Once each instance of CAD model 112 is anchored in a stationary position, an image is captured. Features that do not correlate with pose can be randomized. Thus, lighting effects can be varied, and surface color, texture, and gloss can all be randomized. Thereby, the learning process can be effectively focused on features that are correlated with the pose, such as shape data, edges, etc., to determine the pose. Noise may be added to the images so that the learning process can become accustomed to imperfections in the real image of the object. Lighting effects also play a role here, as real images are not always taken under ideal lighting conditions, which may leave some relevant aspects difficult to detect. be.

Each color image captured by simulator 104 is paired with a corresponding actual pose output from simulator 104 used as a label. In this embodiment, the learning process is an untrained convolutional neural network 117U, which is applied to each color image and label pair. Color images and label pairs form the training data. Training data may be generated prior to the training process or during the training process. In embodiments where the training data is generated using computational resources separate from the training process, it may be more temporally efficient to apply each pair as it is generated. During the training process, the outputs from the untrained convolutional neural network 117U are compared with the corresponding labels and the weights are adjusted accordingly. The training process continues until a condition is met that indicates that training is complete. This condition may be the application of a certain amount of training data, the settling of weights of the untrained convolutional neural network 117U, the output reaching a threshold accuracy, or the like.

Once training is complete, the resulting trained convolutional neural network 117T is ready to be used in a physical environment. In this embodiment, the physical environment includes objects that are the substance of CAD model 112 . These objects are photographed by camera 125, thereby providing one or more color images of the objects. A trained convolutional neural network 117T is applied to this color image to output the 6D pose of each object in the color image. Although the camera 125 may be a more basic, less powerful camera, and the lighting conditions may not be ideal, the trained convolutional neural network 117T will capture this color image in the same manner as the simulated image during training. should be able to handle properly.

Once the 6D pose of each object in the color image has been output, the last operation correction 109 is performed. Correction operation 109 again utilizes CAD model 112 to make fine adjustments to each detected 6D pose. In this embodiment, the CAD model 112 is used to recreate the images according to each output 6D pose and then adjust the 6D pose of any object that appears to be misaligned between images. As the 6D pose is adjusted, the recreated images are manipulated accordingly and the comparison continues until the images match. In this embodiment, the correction operation 109 is a traditional handwritten algorithm rather than another learning process.

Once the correction 109 is complete, the final pose 119 is output. Final pose 119 may be utilized in a variety of ways depending on the context of the embodiment. For example, on an assembly line, the robot arm can utilize the final pose 119 to strategically grab each object in a manner that allows the robot arm to perform the assembly steps. There are many uses other than robotic arms and other than assembly lines. The number of applications requiring proper pose detection is increasing.

FIG. 2 shows an exemplary hardware configuration for pose detection, according to one embodiment of the present invention. An exemplary hardware configuration includes pose detection device 220 in communication with network 228 and capable of interacting with CAD modeler 224 , camera 225 , and robotic arm 226 . Pose detection device 220 may be a host computer, such as a server computer or a mainframe computer that hosts client computers that run and use on-premises applications. In this case, pose detection device 220 may not be directly connected to CAD modeler 224 , camera 225 and robot arm 226 , but through network 228 . Pose detection device 220 may be a computer system including two or more computers. Pose detection device 220 may be a personal computer running an application for the user of pose detection device 220 .

Pose detection device 220 includes logic unit 200 , storage unit 210 , communication interface 221 and input/output controller 222 . Logic portion 200 is a computer that includes one or more computer-readable storage media collectively storing program instructions executable by a processor or programmable circuitry to cause the processor or programmable circuitry to perform various portions of the operations. It may be a program product. Logic portion 200 may alternatively be analog or digital programmable circuitry, or any combination thereof. Logic portion 200 may be comprised of physically separate enclosures or circuits that interact through communication. Storage unit 210 may be a non-volatile computer-readable medium capable of storing non-executable data for access by logic unit 200 during execution of the processes herein. The communication interface 221 reads transmission data that can be stored in a transmission buffer area provided in a recording medium such as the storage unit 210, transmits the read transmission data to the network 228, or records reception data received from the network 228. Write to the receiving buffer area provided on the medium. Input/output controller 222 connects to various input/output units such as CAD modeler 224, camera 225, and robotic arm 226 via parallel ports, serial ports, keyboard ports, mouse ports, monitor ports, etc. to issue commands. Accept and present information.

Acquisition unit 202 is the portion of logic unit 200 that performs acquisition of data from CAD modeler 224, camera 225, robotic arm 226, and network 228 in the course of pose detection. The acquisition unit may acquire a computer model 212 of the object. Acquisition unit 202 may store computer model 212 in storage unit 210 . Acquisition unit 202 may include subsections for performing additional functions described in the flow charts below. Such subsections may be referred to by names associated with their function.

Simulation portion 204 is the portion of logic portion 200 that simulates a computer model in a realistic environment. The simulation unit 204 may simulate computer models of objects in random poses. In doing so, the simulation unit 204 may include a physics engine, such as for inducing motion of the computer model. The simulation unit 204 may store simulation parameters 214 such as a physics engine in the storage unit 210 . The simulation portion 204 may include subsections for performing additional functions described in the flow charts below. Such subsections may be referred to by names associated with their function.

Acquisition unit 205 is the portion of logic unit 200 that acquires training data. The training data may include a plurality of pose representations 215, each pose representation 215 of the computer model in one of a plurality of poses paired with a label containing a pose designation of the computer model shown in the images. Contains images. Images and corresponding pose specifications are defined by simulation unit 204 . The capture unit 205 may store the pose representation 215 in the storage unit 210 . The ingestor 205 may include subsections for performing additional functions described in the flow charts below. Such subsections may be referred to by names associated with their function.

Function generator 206 is the portion of logic 200 that applies a learning process to the pose representation to generate a pose determination function during pose detection. For example, a pose determination function can associate an image of an object with a pose specification. Function generator 206 may store parameters of the trained learning process, such as pose determination function parameters 217 , in storage 210 . Function generator 206 may include subdivisions for performing additional functions described in the flow charts below. Such subsections may be referred to by names associated with their function.

Pose determiner 208 is the portion of logic 200 that determines the pose designation of an object by applying a pose determination function to an image of the object during pose detection. For example, 6D specification of pose specification position and orientation. In so doing, pose determiner 208 may utilize pose determination function parameters 217 stored in storage 210 and images of the same object as computer model 212 in the physical environment captured by camera 225, which gives the output of the 6D pose specification. Pose determiner 208 may include subsections for performing additional functions described in the flow charts below. Such subsections may be referred to by names associated with their function.

Pose corrector 209 is the portion of logic 200 that corrects the pose specification of the object during pose detection. In doing so, pose corrector 209 may utilize correction parameters 218 and computer model 212 stored in storage 210 to provide a corrected 6D pose specification output. The pose corrector 209 may include subsections for performing additional functions described in the flow charts below. Such subsections may be referred to by names associated with their function.

In this embodiment, pose detection device 220 generates training data, trains a learning process to generate a pose decision function, and then automatically generates a trained pose decision function simply by inputting a computer model. can allow you to use

In other embodiments, the pose detection device may be any other device capable of processing logic functions to perform the processes herein. The pose detection device may not need to be networked in environments where inputs, outputs and all information are directly connected. The logic and storage need not be entirely separate devices, but may share one or more computer-readable media. For example, the storage may be a hard drive that stores both computer-executable instructions and data accessed by logic, which is a combination of central processing unit (CPU) and random access memory (RAM). and in the logic portion, computer-executable instructions may be replicated in whole or in part for execution by the CPU during execution of the processes herein. One or more graphics processing units (GPUs) may be included in the logic, particularly in embodiments utilizing neural networks.

In embodiments in which the pose detection device is a computer, a program installed on the computer may cause the computer to function as the apparatus or one or more portions thereof (including modules, components, elements, etc.) of embodiments of the present invention. or may cause operations associated therewith and/or cause a computer to perform the processes of embodiments of the present invention or steps thereof. Such programs may be executed by the processor to cause the computer to perform certain operations associated with blocks in some or all of the flowcharts and block diagrams described herein.

In other embodiments, the camera may be a depth camera capable of capturing depth information for each pixel in addition to color information. In such an embodiment, the fetching unit would also fetch the depth information defined by the simulating unit, and the learning function would be trained accordingly. In other words, the image of the computer model may contain depth information, so capturing the image of the object also includes capturing the depth information. However, many depth cameras may not be accurate at close range. Depth cameras may therefore be more suitable for larger scale applications.

In some embodiments, multiple computer models can be used for a single application. Multiple computer models can be easily simulated in the simulation section, but more training may be required to generate a robust pose determination function. For example, if a single object contains two components that are connected but relatively moveable, such components may be treated as separate objects, and the learning function is trained accordingly. will be In further embodiments, the label may include parameters that define relationships between components. Objects that change shape in more complex ways, such as objects that flow, deform, or have many moving parts, may not be able to generate a reliable pose determination function at all.

FIG. 3 shows an operational flow for pose detection, according to one embodiment of the present invention. This operational flow may provide a pose detection method that may be performed by a pose detection device such as pose detection device 220 or any other device capable of performing the following operations.

At S330, an acquisition unit, such as acquisition unit 202, acquires a computer model. For example, the acquisition unit may acquire a computer model of the object from direct user input, such as from a CAD modeler, such as CAD modeler 224, over a network, such as network 228, or from another source. In some embodiments, the acquisition unit may generate a computer model by performing a 3D scan of the object.

At S340, a simulation unit, such as simulation unit 204, simulates the computer model in a realistic environment. For example, the simulation unit may simulate a computer model in a realistic environment. In some embodiments, the simulation unit may simultaneously simulate more than one instance of the computer model.

At S346, a capture unit such as capture unit 205 captures pose representation training data. For example, the importer may import a plurality of pose representations, each pose representation of the computer model in one of a plurality of poses paired with a label containing a pose designation of the computer model shown in the image. Contains images. Images and corresponding pose specifications are defined by the simulation unit. In embodiments where the simulation unit simulates more than one instance of the computer model, each image may also contain more than one instance of the computer model, each instance of the computer model being in a unique pose.

At S350, a function generator, such as function generator 206, generates a pose determination function. For example, the function generator may apply a learning process to the pose representation to generate a pose determination function that associates images of objects with pose specifications.

At S360, a pose determiner, such as pose determiner 208, determines a pose designation. For example, the pose determiner may determine the pose designation of an object by applying a pose determination function to an image of the object during pose detection. In some embodiments, a pose corrector, such as pose corrector 209, may correct the pose designation of the object. In some embodiments, the pose corrector performs Direct Image Alignment to reduce differences between an image of the computer model according to the object's pose specification and an image of the object in the physical environment. , DIA) can be applied. In some embodiments, such as those in which depth information is available, the pose corrector reduces differences between an image of the computer model according to the object's pose specification and an image of the object in the physical environment. For this purpose, Coherent Point Drift (CPD) can be applied.

At S370, a robotic arm, such as robotic arm 226, may be positioned. For example, the pose detection device may position the robotic arm according to the pose specification. In some embodiments, positioning the robotic arm may include determining the position of the object relative to the robotic arm based on the position of a camera that captured an image of the object, such as camera 225 .

FIG. 4 illustrates an operational flow for simulating a computer model to capture training data, such as S340 and S346 of FIG. 3, according to one embodiment of the invention. The operations in this operational flow may be performed by a simulation unit, such as simulation unit 204, or a correspondingly named subsection thereof, and an acquisition unit, such as acquisition unit 205, or a correspondingly named subsection thereof. .

At S442, an environment generator, such as simulation unit 204 or a subsection thereof, generates a simulation environment. For example, the environment generator may create a 3D space inside which is a computer model and part of which forms a platform. The remaining details of the environment, such as background color and objects, are of little importance, if any, for simulation purposes, and are further randomized to prevent the learning process from assigning values to them.

At S444, a random allocation unit, such as simulation unit 204 or a subsection thereof, randomly allocates colors, textures, and lighting. For example, the random assigner may randomly assign one or more surface colors to the computer model and platform for each pose within the realistic environment simulator. As another example, the random assigner may randomly assign one or more surface textures to the computer model and platform for each pose within the realistic environment simulator. As yet another example, the random assigner may randomly assign lighting effects in the environment for each pose within the realistic environment simulator. Such lighting effects may include at least one of brightness, contrast, color temperature, and direction.

At S445, a motion inducer, such as simulation unit 204 or a subsection thereof, induces motion of the computer model. For example, a motion inducer may induce motion of the computer model relative to a platform within the realistic environment simulator such that the computer model assumes a random pose. Examples of induced motion include dropping, rotating, and colliding the computer model against a platform or other instance of the computer model.

At S446, a capture unit, such as capture unit 205, may capture the image and pose specification. For example, the capturer may capture images of the computer model within the simulation, such as by defining a soft camera within the simulation and using the soft camera to capture images of the computer model. The importer may also import the pose specification of the computer model. Posing may be from the point of view of a soft camera. Alternatively, the pose specification may be from some other viewpoint, eg obtained by transforming the pose specification. In embodiments in which the simulation portion simulates more than one instance of the computer model, each image may also contain more than one instance of the computer model, each instance of the computer model being in a unique pose and thus a unique pose. Associated with a pose specification.

At S448, the simulation unit determines whether a sufficient amount of training data has been captured by the capture unit. If the amount of training data is insufficient, operational flow proceeds to S449 where the environment is reset in preparation for loading another training data. If there is a sufficient amount of training data, the operational flow ends.

FIG. 5 shows an operational flow for generating a pose determination function, such as S350 of FIG. 3, according to one embodiment of the invention. The operations in this operational flow may be performed by a function generator, such as function generator 206, or a correspondingly named subsection thereof.

At S552, a learning process definition unit, such as a function generator or a subsection thereof, defines a learning process. Defining the learning process may include defining the type of neural network, the dimensions of the neural network, the number of layers, and the like. In some embodiments, the learning process definer defines the learning process as a convolutional neural network.

At S554, a pose expression selector, such as a function generator or a subdivision thereof, selects a pose expression from among a plurality of pose expressions. To ensure that each pose expression is processed, only previously unselected pose expressions may be selected at S554 as the iterations of the operational flow for generating the pose determination function proceed. In embodiments where pose representations are processed as soon as they are captured, pose representation selection may not be necessary.

At S556, a learning process applicator, such as a function generator or a subsection thereof, applies a learning process to the image. Applying the learning process to the pose representation may include using the image as input to the learning process, such that the learning process produces an output. In embodiments where the learning process includes a neural network and the pose representations are simulated images, the learning process may output a 6D pose designation.

At S557, the learning process adjuster, such as the function generator or a subsection thereof, adjusts the learning process using the labels and pose designations defined by the simulation unit as goals. As the iterations of the operational flow for generating the pose decision function proceed, the learning process adjuster adjusts the parameters of the learning process, such as the pose decision function parameters 217, to train the learning process to the pose decision function. . In embodiments in which the learning process includes a neural network and the pose representations are simulated images, the learning process adjuster may adjust the weights of the neural network, and the learning process is a 6D model for each instance of the computer model in the image. It can be trained to output pose specifications. For example, after an image is input to the neural network, the error between the actual output of the neural network and the corresponding pose specification is calculated. Once the error is calculated, it is then back-propagated, ie the error is expressed as a derivative with respect to each weight of the network. Once the derivative is obtained, the neural network weights are updated according to the function of this derivative.

At S559, the function generator determines whether all pose expressions have been processed by the function generator. If there are pose representations left unprocessed, operational flow returns to S554 where another pose representation is selected for processing. If no pose representations remain unprocessed, the operational flow ends. As the operational flow of FIG. 5 is performed iteratively, the iterations of operations S554, S556, and S557 collectively result in operations that generate a pose determination function. At the end of the operational flow of FIG. 5, the learning process has been trained sufficiently to be the pose decision function.

Although in this embodiment training terminates when all pose representations have been processed, other embodiments may use a method to determine when training terminates, such as by the number of epochs or according to the amount of error. It may contain different criteria. Also, while in this embodiment the parameters of the learning process are adjusted after each pose representation is applied, other embodiments adjust the parameters at different intervals, e.g., once per epoch, or depending on the amount of error. may be adjusted. Finally, in this embodiment, the output of the learning process is the pose determination function, which means that the output of the learning function is the pose specification, whereas in other embodiments the learning process itself does not output the pose specification. may provide some output combined with the camera parameters that provide the pose specification. In these embodiments, the training data may be generated by removing such parameters from the pose specification of the camera in order to adequately define the target learning process output. In these embodiments, the pose determination function includes both a trained learning process and a function for combining the output with camera parameters.

FIG. 6 illustrates an operational flow for determining pose designations, such as S360 of FIG. 3, according to one embodiment of the invention. The operations in this operational flow are performed by a pose determiner such as pose determiner 208 or its correspondingly named subsection and a pose corrector such as pose corrector 209 or its correspondingly named subsection. can be

At S662, an image capture component, such as pose determiner 208 or a subsection thereof, captures an image of the object. For example, the image capturer may capture images of objects in the physical environment. The image capture unit may communicate with a camera, such as camera 225, or other photosensor for capturing images. Although the pose determination function can be effectively trained such that color information does not affect the pose specification that is output, images captured in color can provide more information, resulting in The information representing them is more misaligned at edges than, for example, in images captured in grayscale, thereby allowing the pose determination function to more easily detect edges that define objects in the image.

At S664, a pose determination function applicator, such as pose determiner 208 or a correspondingly named subsection thereof, applies the pose determination function to the image. Applying the pose determination function to the image may include using the image as an input to the pose determination function such that the pose determination function produces an output. In embodiments where the pose determination function includes a neural network, the neural network may output a 6D pose designation for each instance of the computer model in the image.

At S666, an image creator, such as pose corrector 209 or a correspondingly named subsection thereof, creates an image of the computer model. For example, the image creator may create an image of the computer model according to the pose specification of the object. In some embodiments, the image consists only of a computer model with a plain background and a pose specification.

At S667, an image comparator, such as the pose corrector 209 or a correspondingly named subsection thereof, compares the generated image with the captured image. For example, the image comparator may compare an image of the computer model according to the pose specification of the object with an image of the object in the physical environment. In some embodiments, for ease of comparison, silhouettes that may be generated by cropping the generated image are compared to the silhouette of the generated image calculated directly from the simulation. This comparison can be performed iteratively until the error is sufficiently minimized.

At S669, a pose adjuster, such as pose corrector 209 or a correspondingly named subsection thereof, adjusts the pose specification output from the pose determination function. For example, the pose adjuster may adjust pose specifications to reduce differences between the captured image and the generated image.

In many of the embodiments herein, the pose detection device generates training data, trains a learning process to generate a pose determination function, and then automatically trains a pre-trained function simply by inputting a computer model. It may be possible to use a pose determination function. By utilizing a simulator to generate training data, embodiments described herein may enable rapid image capture, including capture of pose specifications defined by the simulator as labels. Using pose specifications defined by the simulator also allows the labels to be very accurate. Existing simulators known for their realistic accuracy, such as the UNREAL® engine, not only increase confidence in accuracy, but may also be equipped with image processing and environmental randomization capabilities. .

Various embodiments of the present invention are described with reference to flowcharts and block diagrams, where blocks may represent (1) steps in a process in which operations are performed or (2) portions of an apparatus responsible for performing the operations. can do. Certain steps and portions may be performed by dedicated circuitry, programmable circuitry provided with computer readable instructions stored on a computer readable medium, and/or processor provided with computer readable instructions stored on a computer readable medium. can be implemented. Dedicated circuitry may include digital and/or analog hardware circuitry and may include integrated circuits (ICs) and/or discrete circuits. Programmable circuits include logic AND, OR, XOR, NAND, NOR, and other logic operations, flip-flops, registers, memory elements, etc., such as Field Programmable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs), and the like. It may include configurable hardware circuitry. A processor may include a central processing unit (CPU), a graphics processing unit (GPU), a mobile processing unit (MPU), and the like.

Computer-readable media may include any tangible device capable of storing instructions to be executed by a suitable device, such that computer-readable media having instructions stored thereon may be designated in flowcharts or block diagrams. It will comprise an article of manufacture containing instructions that can be executed to create means for performing the operations described above. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer readable media include floppy disks, diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), electrically erasable Programmable Read Only Memory (EEPROM) Static Random Access Memory (SRAM) Compact Disc Read Only Memory (CD-ROM) Digital Versatile Disc (DVD) BLU-RAY disc Memory Stick Integrated Circuit Card etc. can be included.

The computer readable instructions may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or an object oriented programming language such as Smalltalk®, JAVA®. ), C++, etc., and any combination of one or more programming languages, including conventional procedural programming languages such as the "C" programming language or similar programming languages, either source or object code may contain

The computer readable instructions may be transferred to a processor or programmable circuit of a general purpose computer, special purpose computer, or other programmable data processing apparatus, either locally or over a local area network (LAN), e.g., a wide area network (WAN) such as the Internet. and may be executed to create means for performing the operations specified in the flowchart or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and the like.

Many of the embodiments of the present invention involve artificial intelligence, learning processes, and neural networks, among others. Some of the above embodiments describe specific types of neural networks. However, the learning process usually starts by setting values such as weights with random numbers. Such untrained learning processes must be pretrained so that they can reasonably be expected to perform the function well. Many of the processes described herein are for the purpose of training the learning process for pose detection. Once trained, the learning process can be used for pose detection and may not require further training. Thus, the trained pose decision function is the product of a process of training an untrained learning process.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various modifications or amendments can be made to the above embodiments. It is clear from the description of the scope of claims that forms with such modifications or amendments can also be included in the technical scope of the present invention.

The operations, procedures, steps, and phases of each process performed by the apparatus, systems, programs, and methods depicted in the claims, embodiments, or drawings may be "prior to," "before," etc. in order. Unless indicated by , and unless the output from the earlier process is used by the later process, they can be executed in any order. Regarding the operation flow in the claims, specification, or drawings, even if it is described using "first" or "next" for convenience, it means that it is essential to execute in this order not a thing

Claims (25)

A computer program executable by a computer, comprising:
obtaining a computer model of an object;
simulating the computer model in a realistic environment simulator;
A procedure for capturing training data comprising a plurality of pose representations, each pose representation comprising an image of said computer model in one of a plurality of poses, one of said plurality of poses being said image. a capturing step paired with a label containing a pose specification of the computer model shown in , wherein the image and the pose specification of the computer model are defined by the realistic environment simulator;
applying a learning process to the plurality of pose representations to generate a pose determination function for associating an image of the object with a pose specification;
capturing an image of said object in a physical environment;
determining a pose designation of the object by applying the pose determination function to the image of the object;
correcting the pose specification of the object;
to perform an operation including
the image of the computer model includes depth information;
said capturing said image of said object further comprising capturing depth information;
The correcting procedure applies one of Direct Image Alignment (DIA) and Coherent Point Drift (CPD) to transform the computer model according to the pose specification of the object. reducing differences between the image of and the image of the object in the physical environment;
computer program. 2. The computer program according to claim 1 , further comprising the step of positioning a robot arm according to said pose specification. 3. The computer program product of claim 2 , wherein the step of positioning the robotic arm comprises determining the position of the object relative to the robotic arm based on the position of a camera that captured the image of the object. 4. A computer program product according to any one of claims 1 to 3 , wherein said step of correcting comprises creating an image of said computer model according to said pose specification of said object. The procedure for correcting
comparing said image of said computer model according to said pose specification of said object with said image of said object in said physical environment;
adjusting the pose specification to reduce differences between the captured image and the generated image;
5. The computer program of claim 4 , further comprising: 6. A computer program product according to any preceding claim, wherein the pose specification is a 6D specification of position and orientation. the step of simulating includes simulating more than one instance of the computer model;
each image includes the more than one instance of the computer model, each instance of the computer model being a unique pose;
Computer program according to any one of claims 1 to 6. the realistic environment simulator includes a physics engine;
the simulating step includes inducing motion of the computer model relative to a platform within the realistic environment simulator such that the computer model assumes a random pose;
Computer program according to any one of claims 1 to 7 . 9. The computer program product of claim 8 , wherein the motion-inducing procedure comprises at least one of dropping, rolling, and colliding procedures. 10. The computer program product of claim 8 or 9 , wherein the step of simulating comprises randomly assigning one or more surface colors to the computer model and platform on a pose-by-pose basis within the realistic environment simulator. A computer program executable by a computer, comprising:
obtaining a computer model of an object;
simulating the computer model in a realistic environment simulator;
A procedure for capturing training data comprising a plurality of pose representations, each pose representation comprising an image of said computer model in one of a plurality of poses, one of said plurality of poses being said image. a capturing step paired with a label containing a pose specification of the computer model shown in , wherein the image and the pose specification of the computer model are defined by the realistic environment simulator;
applying a learning process to the plurality of pose representations to generate a pose determination function for associating an image of the object with a pose specification;
to perform an operation including
the realistic environment simulator includes a physics engine;
the simulating step includes inducing motion of the computer model relative to a platform within the realistic environment simulator such that the computer model assumes a random pose;
the step of simulating includes randomly assigning one or more surface colors to the computer model and the platform for each pose within the realistic environment simulator;
computer program. 12. The step of simulating comprises randomly assigning one or more surface textures to the computer model and platform on a pose-by-pose basis within the realistic environment simulator. The computer program described. A computer program executable by a computer, comprising:
obtaining a computer model of an object;
simulating the computer model in a realistic environment simulator;
A procedure for capturing training data comprising a plurality of pose representations, each pose representation comprising an image of said computer model in one of a plurality of poses, one of said plurality of poses being said image. a capturing step paired with a label containing a pose specification of the computer model shown in , wherein the image and the pose specification of the computer model are defined by the realistic environment simulator;
applying a learning process to the plurality of pose representations to generate a pose determination function for associating an image of the object with a pose specification;
to perform an operation including
the realistic environment simulator includes a physics engine;
the simulating step includes inducing motion of the computer model relative to a platform within the realistic environment simulator such that the computer model assumes a random pose;
the step of simulating includes randomly assigning one or more surface textures to the computer model and the platform for each pose within the realistic environment simulator;
computer program. 14. The computer program product of any one of claims 1 to 13 , wherein the step of simulating comprises randomly assigning lighting effects in the environment from pose to pose within the realistic environment simulator. A computer program executable by a computer, comprising:
obtaining a computer model of an object;
simulating the computer model in a realistic environment simulator;
A procedure for capturing training data comprising a plurality of pose representations, each pose representation comprising an image of said computer model in one of a plurality of poses, one of said plurality of poses being said image. a capturing step paired with a label containing a pose specification of the computer model shown in , wherein the image and the pose specification of the computer model are defined by the realistic environment simulator;
applying a learning process to the plurality of pose representations to generate a pose determination function for associating an image of the object with a pose specification;
to perform an operation including
the step of simulating includes, within the realistic environment simulator, randomly assigning lighting effects in the environment for each pose;
computer program. 16. A computer program as claimed in claim 14 or 15, wherein the lighting effects include at least one of brightness, contrast, color temperature and direction. 17. A computer program product as claimed in any preceding claim, wherein the learning process is a convolutional neural network. obtaining a computer model of the object;
simulating the computer model in a realistic environment simulator;
capturing training data comprising a plurality of pose representations, each pose representation comprising an image of said computer model in one of a plurality of poses, one of said plurality of poses being said image; wherein said image and said pose designation of said computer model are defined by said realistic environment simulator;
applying a learning process to the plurality of pose representations to generate a pose determination function for associating an image of the object with a pose specification;
capturing an image of the object in a physical environment;
determining a pose designation of the object by applying the pose determination function to the image of the object;
correcting the pose designation of the object;
including
the image of the computer model includes depth information;
said capturing said image of said object further comprising capturing depth information;
the step of correcting applies one of Direct Image Alignment (DIA) and Coherent Point Drift (CPD) to generate the computer model according to the pose specification of the object; reducing differences between the image of and the image of the object in the physical environment;
Computer-implemented method. obtaining a computer model of the object;
simulating the computer model in a realistic environment simulator;
capturing training data comprising a plurality of pose representations, each pose representation comprising an image of said computer model in one of a plurality of poses, one of said plurality of poses being said image; wherein said image and said pose designation of said computer model are defined by said realistic environment simulator;
applying a learning process to the plurality of pose representations to generate a pose determination function for associating an image of the object with a pose specification;
including
the realistic environment simulator includes a physics engine;
the simulating step includes inducing motion of the computer model relative to a platform within the realistic environment simulator such that the computer model assumes a random pose;
said simulating includes randomly assigning one or more surface colors to said computer model and said platform for each pose within said realistic environment simulator;
Computer-implemented method. obtaining a computer model of the object;
simulating the computer model in a realistic environment simulator;
capturing training data comprising a plurality of pose representations, each pose representation comprising an image of said computer model in one of a plurality of poses, one of said plurality of poses being said image; wherein said image and said pose designation of said computer model are defined by said realistic environment simulator;
applying a learning process to the plurality of pose representations to generate a pose determination function for associating an image of the object with a pose specification;
including
the realistic environment simulator includes a physics engine;
the simulating step includes inducing motion of the computer model relative to a platform within the realistic environment simulator such that the computer model assumes a random pose;
said simulating includes randomly assigning one or more surface textures to said computer model and said platform for each pose within said realistic environment simulator;
Computer-implemented method. obtaining a computer model of the object;
simulating the computer model in a realistic environment simulator;
capturing training data comprising a plurality of pose representations, each pose representation comprising an image of said computer model in one of a plurality of poses, one of said plurality of poses being said image; wherein said image and said pose designation of said computer model are defined by said realistic environment simulator;
applying a learning process to the plurality of pose representations to generate a pose determination function for associating an image of the object with a pose specification;
including
the simulating step includes randomly assigning lighting effects in the environment to poses within the realistic environment simulator;
Computer-implemented method. an acquisition unit configured to acquire a computer model of an object;
a simulation unit configured to simulate the computer model in a realistic environment simulator;
a capture unit configured to capture training data comprising a plurality of pose representations and an image of the object in a physical environment , each pose representation of the computer model in one of a plurality of poses; an image, one of the plurality of poses being paired with a label including a pose specification of the computer model shown in the image, the image and the pose specification of the computer model being the realistic an acquisition section defined by the environment simulator;
a learning process applying unit configured to apply a learning process to the plurality of pose representations to generate a pose determination function for associating an image of the object with a pose specification;
a pose determination unit that determines a pose designation of the object by applying the pose determination function to the image of the object;
a pose correction unit that corrects the pose designation of the object;
with
the image of the computer model includes depth information;
The capturing unit further captures depth information,
The pose corrector applies one of Direct Image Alignment (DIA) and Coherent Point Drift (CPD) to transform the computer model according to the pose specification of the object. reducing differences between the image of and the image of the object in the physical environment;
Device. an acquisition unit configured to acquire a computer model of an object;
a simulation unit configured to simulate the computer model in a realistic environment simulator;
an importer configured to import training data comprising a plurality of pose representations, each pose representation comprising an image of the computer model in one of a plurality of poses; is paired with a label containing a pose designation of said computer model shown in said image, said image and said pose designation of said computer model being defined by said realistic environment simulator; ,
a learning process applying unit configured to apply a learning process to the plurality of pose representations to generate a pose determination function for associating an image of the object with a pose specification;
with
the realistic environment simulator includes a physics engine;
the simulation unit induces motion of the computer model relative to a platform within the realistic environment simulator such that the computer model assumes a random pose;
wherein the simulation unit randomly assigns one or more surface colors to the computer model and the platform for each pose within the realistic environment simulator;
Device. an acquisition unit configured to acquire a computer model of an object;
a simulation unit configured to simulate the computer model in a realistic environment simulator;
an importer configured to import training data comprising a plurality of pose representations, each pose representation comprising an image of the computer model in one of a plurality of poses; is paired with a label containing a pose designation of said computer model shown in said image, said image and said pose designation of said computer model being defined by said realistic environment simulator; ,
a learning process applying unit configured to apply a learning process to the plurality of pose representations to generate a pose determination function for associating an image of the object with a pose specification;
with
the realistic environment simulator includes a physics engine;
the simulation unit induces motion of the computer model relative to a platform within the realistic environment simulator such that the computer model assumes a random pose;
the simulation unit randomly assigning one or more surface textures to the computer model and the platform for each pose within the realistic environment simulator;
Device. an acquisition unit configured to acquire a computer model of an object;
a simulation unit configured to simulate the computer model in a realistic environment simulator;
an importer configured to import training data comprising a plurality of pose representations, each pose representation comprising an image of the computer model in one of a plurality of poses; is paired with a label containing a pose designation of said computer model shown in said image, said image and said pose designation of said computer model being defined by said realistic environment simulator; ,
a learning process applying unit configured to apply a learning process to the plurality of pose representations to generate a pose determination function for associating an image of the object with a pose specification;
with
wherein the simulation unit randomly assigns lighting effects in the environment to poses within the realistic environment simulator;
Device.

Images