WO2024257546A1 - ロボットシステム、ロボット制御方法、およびロボット制御プログラム - Google Patents

ロボットシステム、ロボット制御方法、およびロボット制御プログラム Download PDF

Info

Publication number
WO2024257546A1
WO2024257546A1 PCT/JP2024/018169 JP2024018169W WO2024257546A1 WO 2024257546 A1 WO2024257546 A1 WO 2024257546A1 JP 2024018169 W JP2024018169 W JP 2024018169W WO 2024257546 A1 WO2024257546 A1 WO 2024257546A1
Authority
WO
WIPO (PCT)
Prior art keywords
workpiece
robot
work
information
skeleton
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2024/018169
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
諒 増村
健斗 山田
大輔 九日
麻梨乃 樫山
俊樹 伊藤
享大 戸畑
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yaskawa Electric Corp
Original Assignee
Yaskawa Electric Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yaskawa Electric Corp filed Critical Yaskawa Electric Corp
Priority to JP2025527584A priority Critical patent/JPWO2024257546A1/ja
Priority to DE112024001966.8T priority patent/DE112024001966T5/de
Publication of WO2024257546A1 publication Critical patent/WO2024257546A1/ja
Priority to US19/415,781 priority patent/US20260097515A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1694—Program controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
    • B25J19/02—Sensing devices
    • B25J19/021—Optical sensing devices
    • B25J19/023—Optical sensing devices including video camera means
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1656—Program controls characterised by programming, planning systems for manipulators
    • B25J9/1661—Program controls characterised by programming, planning systems for manipulators characterised by task planning, object-oriented languages
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1656—Program controls characterised by programming, planning systems for manipulators
    • B25J9/1664—Program controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40053—Pick 3-D object from pile of objects

Definitions

  • One aspect of the present disclosure relates to a robot system, a robot control method, and a robot control program.
  • Robots that handle food, agricultural products, and other items have been developed for some time.
  • one aspect of this disclosure aims to improve the reliability of work performed on a workpiece.
  • a robot system includes a skeleton estimation unit that estimates skeleton information indicating the skeleton of a workpiece based on a workpiece image including the workpiece existing in real space, a completion state determination unit that determines the completion state of work performed on the workpiece based on constraint condition information indicating constraint conditions related to work on the workpiece and the estimated skeleton information, and a robot control unit that causes a robot positioned in real space to perform work according to the determined completion state.
  • FIG. 1 is a diagram illustrating an example of a configuration of a robot system.
  • FIG. 2 is a diagram showing a workpiece and its skeletal information.
  • FIG. 2 is a diagram showing a workpiece and its skeletal information.
  • FIG. 11 is a diagram illustrating an example of the operation of the robot system.
  • FIG. 11 is a diagram illustrating an example of the operation of the robot system.
  • FIG. 13 is a diagram for explaining the results of the operation of the robot system.
  • FIG. 13 is a diagram for explaining the results of the operation of the robot system.
  • FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer used for the robot system.
  • FIG. 13 is a diagram showing a configuration of a robot system according to a modified example.
  • FIG. 13 is a diagram showing an example of estimated workpiece keypoints.
  • FIG. 11 illustrates an example of a process for selecting a workpiece.
  • FIG. 13 is a diagram showing an example of a processed workpiece.
  • 13 is a diagram showing a configuration of a robot system according to a modified example.
  • FIG. 13 is a diagram illustrating an example of a user interface for generating constraint condition information.
  • FIG. 1 is a diagram showing an example of the configuration of the robot system 100.
  • Fig. 2A and Fig. 2B are both diagrams showing a workpiece W and skeletal information K of the workpiece W.
  • the robot system 100 is a control system for causing a robot R placed in real space to process a workpiece W present in the real space.
  • the robot system 100 includes a camera C, a skeleton estimation unit 110, a workpiece memory unit 112, a learning data generation unit 113, a learning unit 114, a workpiece selection unit 120, a start state determination unit 130, a work information memory unit 132, a completion state determination unit 140, a constraint condition information memory unit 141, an end state determination unit 142, a path generation unit 150, a robot control unit 160, and a robot R.
  • the robot R is a device or equipment that performs work on a workpiece W.
  • the robot R is placed in real space.
  • the robot R has an end effector that applies the action required for that work to the workpiece W.
  • the robot R is a vertical articulated robot, but the robot R may also be other types of robots, such as a horizontal articulated robot or a cartesian machine.
  • the operation refers to a process performed by the robot R on the workpiece W.
  • the operation can be a process that affects the shape of the workpiece W.
  • the operation includes at least one of handling the workpiece W and processing the workpiece W.
  • handling include pick-and-place, alignment, packing, unpacking, and assembly.
  • processing include welding, painting, grinding, polishing, cutting, and printing.
  • the robot system 100 can cause the robot R to perform work on various workpieces W.
  • the robot system 100 causes the robot R to perform work on an irregular workpiece W.
  • the irregular workpiece W can be broccoli, fried shrimp, fried chicken, and other workpieces that are difficult to deform but have individual differences in shape.
  • the irregular workpiece W can be a flexible object, such as a harness, cable, or handkerchief, in which the individual differences in shape can be ignored but the difference in the degree of deformation between individual objects can be large.
  • the irregular workpiece W can be a workpiece that combines the characteristics of these two types of workpieces. Broccoli is shown in Figure 2A as an example of an irregular workpiece W that has individual differences in shape.
  • the camera C captures an image of the workpiece W and generates a workpiece image that is an image that includes the workpiece W (i.e., an image that captures the workpiece W).
  • the image may be a two-dimensional image, or may include a distance image in addition to the two-dimensional image.
  • the camera C is fixed to the tip of the arm of the robot R.
  • the camera C may be fixed to another position on the robot R, or may be fixed to the ceiling or a pillar. Multiple cameras C may be arranged.
  • the skeleton estimation unit 110 is a functional module that estimates skeleton information K that indicates the skeleton of the workpiece W based on an image including the workpiece W.
  • the skeleton information K indicates the skeleton of the workpiece W to the extent that work on the workpiece W by the robot R can be planned.
  • the skeleton of the workpiece W represents the external shape of the workpiece W or the approximate positional relationship of the parts of the workpiece W.
  • the workpiece W does not need to have a "bone.”
  • broccoli has a structure in which multiple stalks extend from one stalk and each stalk is connected to a bunch.
  • the skeleton information K indicates the approximate positional relationship of such parts to the extent that the robot R can perform work on the broccoli.
  • the skeletal information K of the work W includes multiple keys, which are information for indicating the approximate positional relationship of parts of the work W.
  • the multiple keys include multiple key points, which are information indicating the positions of specific parts of the work W, and one or more skeletons, each of which is a virtual line connecting two key points.
  • a skeleton is also called a bone.
  • the skeletal information K of the work W includes multiple keys, which include multiple key points P (P1 to P5) and multiple skeletons S (S1 to S4) of the work W.
  • the skeleton estimation unit 110 can estimate the skeletal information K for each of the two or more works W.
  • the skeleton estimation unit 110 estimates the skeleton information K of the workpiece W using the skeleton estimation model 111.
  • the skeleton estimation model 111 is an inference device that accepts an input of a workpiece image including the workpiece W and estimates the skeleton information K of the workpiece W.
  • the skeleton estimation unit 110 estimates the skeleton information K by inputting the workpiece image of the workpiece W for which the skeleton information K is to be estimated to the skeleton estimation model 111.
  • the skeleton estimation model 111 can be realized using keypoint detection technology such as OpenPose, RTMPoses, and YOLOX-Pose.
  • a keypoint P the position of a specific part of the recognition target of the workpiece W is identified as a keypoint P, and a virtual line (edge) that represents the relationship between the two positions is inferred as a skeleton S (bone).
  • skeleton S a virtual line that represents the relationship between the two positions.
  • the skeleton estimation model 111 is generated in advance by machine learning using learning data including a sample image, which is an image based on a three-dimensional model (3D model) of the work W, and a plurality of data records in which a plurality of keys set in the 3D model are associated with each other. Therefore, the skeleton estimation model 111 is a trained model.
  • the multiple keys in each data record also include a plurality of key points and one or more skeletons, each of which connects two key points.
  • the robot system 100 has a work memory unit 112, a training data generation unit 113, and a learning unit 114 as components for performing the learning.
  • the work storage unit 112 is a functional module that stores at least one sample image based on a 3D model of the work W and skeletal information K of each 3D model as learning data.
  • the learning data generation unit 113 is a functional module that transforms a 3D model indicated by at least one of the multiple data records of the learning data stored in the work storage unit 112 to generate a new data record of the learning data, and stores the new data record in the work storage unit 112.
  • the skeletal information K of the original 3D model is known, the skeletal information K (multiple keys) of the transformed 3D model can be determined. Therefore, a new data record in which the transformed 3D model and the determined skeletal information K are associated with each other can be stored in the work storage unit 112.
  • the learning data generation unit 113 can increase the learning data by such processing.
  • the transformation of the 3D model may include not only a process of changing the outer shape of the 3D model to give individual differences in the outer shape of the work W, but also a process of giving individual differences in the degree of deformation of the 3D model of an item that is flexible and therefore deformable.
  • the learning unit 114 is a functional module that trains the skeleton estimation model 111 by machine learning using learning data that may include new data records, and generates the skeleton estimation model 111.
  • the generation of the skeleton estimation model 111 by the learning unit 114 may include updating the skeleton estimation model 111.
  • the work selection unit 120 is a functional module that selects at least one work W to be processed in the work based on skeletal information K estimated for each of the multiple work pieces W included in the work image. If the work image includes a single work piece W, the work selection unit 120 selects that work piece W. As shown in FIG. 2B, the work image may include multiple work pieces W. In this case, the work selection unit 120 selects at least one work piece W to be processed in the work from the work image.
  • the work selection unit 120 selects at least one work based on the skeletal information K of each of the multiple workpieces W.
  • the work selection unit 120 may select a workpiece W having skeletal information K with a degree of deformation within a predetermined range, such as a straight workpiece W or a workpiece W with a small degree of bending.
  • the work selection unit 120 may select the current workpiece W based on its relationship with previously selected workpieces W so that the variation in the skeletal information K of the workpieces W to be processed successively falls within a predetermined threshold value.
  • the work selection unit 120 calculates the area on the work image for each of the multiple workpieces W. Then, based on the area and skeletal information K of each of the multiple workpieces W, the work selection unit 120 selects a workpiece that is entirely exposed in the work image without being hidden or buried by other workpieces W. Based on the area and skeletal information K of each of the multiple workpieces W, the work selection unit 120 may select a workpiece W whose skeletal information K is entirely exposed, or a workpiece W whose proportion of exposed skeletal information K is equal to or greater than a predetermined threshold value.
  • the start state determination unit 130 is a functional module that determines the start state, which is the state of the robot R at the start of work, based on the skeletal information K of the workpiece W.
  • the start state represents, for example, at least one of the position and posture of the robot R when work on the workpiece W is started.
  • the position and posture of the robot R may be the position and posture of the end effector.
  • the start state represents, for example, the position and posture of the end effector when picking up the workpiece W.
  • the start state represents, for example, the position and posture of the end effector at the start of the process.
  • the start state determination unit 130 may analyze the skeletal information K of the workpiece W, convert the skeletal information K into at least one of the position and posture of the end effector when the end effector acts on the workpiece W (e.g., picks it), and determine the result of the conversion as the start state.
  • the start state determination unit 130 may perform the conversion using a function that receives input of the skeletal information K and calculates the start state, or may geometrically analyze the skeletal information K to extract the start state.
  • the start state determination unit 130 may perform the conversion using an inference unit that receives input of at least the skeletal information K and outputs the start state.
  • the start state determination unit 130 may simulate various start states based on the skeletal information K, evaluate each simulation result using a predetermined evaluation method, and determine the start state that will give the optimal simulation result.
  • the start state determination unit 130 may determine the start information based on the work information that is set according to the type of work W or the work and stored in the work information storage unit 132, and the estimated skeletal information K.
  • the work information is defined for each type of work W, regardless of individual differences such as the shape of the work W, and is associated with at least one of a plurality of keys of the work W.
  • the work information represents information regarding the work performed on the work W.
  • the work information includes at least one of the position and the posture of the end effector acting on the work W in the start state of the work performed on the work W.
  • the operation information includes pick information indicating whether or not the robot R can pick a position on the workpiece W corresponding to at least one key of the workpiece W.
  • the pick information may indicate whether or not the robot R can pick the periphery of each of the key points P and skeletons S.
  • the start state determination unit 130 determines the pick position A of the workpiece W by the robot R as the start state based on the pick information and the estimated skeleton information K.
  • the pick position A refers to the position at which the robot R acts on the workpiece W to pick it.
  • the pick position A indicates the gripping position
  • the pick position A indicates the suction position.
  • the operation information may indicate whether or not processing can be started around each of the multiple keys (each key point P and each skeleton S) of the workpiece W.
  • the completion state determination unit 140 is a functional module that determines the completion state of the work performed on the workpiece W based on the constraint condition information that represents the constraint conditions related to the work on the workpiece W and the estimated skeletal information K.
  • the robot system 100 causes the robot R to perform the work according to the determined completion state.
  • the constraint conditions represent conditions related to the work on the workpiece W, and are stored in advance as constraint condition information in the constraint condition information storage unit 141.
  • the constraint conditions are specified for each type of workpiece W, regardless of individual differences such as the shape of the workpiece W.
  • the constraint conditions include conditions for the state of one or more workpieces W after the work is completed for each of multiple types of workpieces W. If the work is handling, the constraint conditions may be conditions for the posture (e.g., orientation) of each workpiece W, conditions for the evaluation result in a state in which one or more workpieces W are arranged or aligned, or conditions for the state of the workpieces W after assembly is completed.
  • the evaluation result may be calculated by applying physical measurements of one or more arranged or aligned workpieces W to a predetermined algorithm, or may be calculated by inputting an image of one or more arranged or aligned workpieces W into a trained model such as a neural network. If the work is processing, the constraint conditions may be conditions for the state of one or more workpieces W after the processing is completed.
  • the completion state represents the predicted state of a specific workpiece W processed by the work to be performed by the robot R when the work to be performed is completed.
  • the constraint conditions are defined for each type of workpiece W, regardless of individual differences such as the shape of the workpiece W, whereas the completion state is defined according to the shape of the individual workpiece W to be processed by the work to be performed.
  • the constraint conditions may be a condition that a specific part of the workpiece W is placed in a specific direction, or a condition regarding the evaluation value of the state in which multiple workpieces W are placed.
  • the completion state determination unit 140 may determine, as the completion state, a post-placement state, which is a state after the workpiece W has been placed, according to the individual shape of the workpiece W (i.e., the skeletal information K).
  • the completion state determination unit 140 may at least determine the posture (e.g., orientation) of the placed workpiece W as the post-placement state, and may further determine at least one of the position and shape of the placed workpiece W.
  • the completion state determination unit 140 determines at least the posture of the placed work W by specifying not only the direction of a specific part of the work W but also the direction of other parts, and further determines at least one of the position and shape of the placed work W.
  • the completion state determination unit 140 may convert the constraint condition information into a value or condition specific to the individual work W to be processed using the estimated skeletal information K, and determine the conversion result as the completion state. For example, if the constraint condition information specifies the position and direction of a specific part of the work W, the completion state determination unit 140 identifies a specific part of the work W from the estimated skeletal information K, and specifies the position and direction of the individual work W so that the specified part is in the position and direction specified by the constraint condition information. Then, the completion state determination unit 140 determines the state of the work W arranged at the specified position and direction as the completion state.
  • the completion state determination unit 140 may determine the completion state using a function that receives input of the skeletal information K and calculates the completion state based on the constraint condition information, or may extract the completion state by geometrically analyzing the skeletal information K based on the constraint condition information. As another example, the completion state determination unit 140 may determine the completion state using an inference device that receives input of at least the constraint condition information and the skeletal information K and outputs the completion state. Alternatively, the completion state determination unit 140 may simulate various completion states based on the skeleton information K and the constraint condition information, evaluate each simulation result using a predetermined evaluation method, and determine the completion state that produces the optimal simulation result.
  • the robot system 100 causes the robot R to perform a task corresponding to the determined completion state.
  • the task corresponding to the completion state is a task that transitions the state of the workpiece W to the completion state, in which case the robot system 100 can achieve the completion state as a result of the task.
  • the robot system 100 can also achieve the completion state by determining the start state of the task.
  • the robot system 100 may determine an individual welding start position for the workpiece W as the completion state based on the constraint condition in accordance with individual differences in the workpiece W, and cause the robot R to start the task from that welding start position.
  • the end state determination unit 142 is a functional module that determines an end state, which is the state of the robot R in a determined completion state, based on the determined completion state. If the robot R can be operated by the completion state without using the end state, the process of determining the end state may be omitted. In this case, the end state can be appropriately interpreted as the completion state. Examples of cases where an end state is unnecessary include a case where the completion state directly represents the end state, and a case where the operation of the robot R is uniquely determined by determining the completion state.
  • the end state represents at least one of the position and posture of the robot R when the operation on the workpiece W is completed.
  • the position and posture of the robot R may be the position and posture of the end effector.
  • the end state represents the position and posture of the end effector when placing the workpiece W.
  • the end state represents the position and posture of the end effector when the process is completed.
  • the end state determination unit 142 may convert the completion state of the workpiece W based on the configuration of the robot R or the end effector, identify either the position and posture of the tip of the arm of the robot R or the end effector, and determine the identified result as the end state.
  • the scope of this disclosure also includes cases where the completion state of the workpiece W is determined indirectly rather than explicitly, that is, where the completion state determination unit 140 and the end state determination unit 142 are integrated into a series of processes, and the end state is determined as if from the constraint condition information and the skeleton information K.
  • the path generating unit 150 is a functional module that generates a path for the robot R based on the determined start state and end state.
  • the generation of a path is also called a path plan.
  • a path represents, for example, the trajectory of the movement of the robot R from the starting point to the end point.
  • the path generating unit 150 may collect information on the surrounding environment, such as obstacles, in addition to the start state and end state, and generate a path that prevents the robot R and the transported object from interfering with the obstacles. In the case of pick-and-place, the transported object is a workpiece W.
  • the path generating unit 150 may also generate a path using other methods.
  • At least one of the start state and the end state may include a predetermined motion of the robot R that is uniquely determined by that state, and the predetermined motion may include an approach point.
  • the path generating unit 150 may generate a path from the approach point or a path to the approach point.
  • the path generating unit 150 may execute at least one of the end state determination unit 142 redetermining the end state and the start state determination unit 130 redetermining the start state, and generate (regenerate) a path based on at least one of the redetermined end state and the redetermined start state.
  • the path generating unit 150 may plan to change the grip of the workpiece W during the process of generating or regenerating a path for a specified task such as pick-and-place. As an example, consider a case where the robot system 100 temporarily places the workpiece W on a placement table, changes the way the workpiece W is held, and continues the pick-and-place.
  • the robot system 100 may process, as pick-and-place, the task of transporting the workpiece W to the placement table and the task of transitioning the workpiece W placed on the placement table to a completed state.
  • the robot system 100 may divide the task of directly transitioning the workpiece W to the completed state into the task of placing the workpiece W on the placement table and the task of transitioning the workpiece W placed on the placement table to the completed state, and generate a path for each of the divided tasks.
  • the robot control unit 160 is a functional module that causes the robot R to perform a task corresponding to the determined completion state. For example, the robot control unit 160 controls the robot R so that the robot R performs a task of transitioning the workpiece W to a completion state using an end effector.
  • the robot control unit 160 may cause the robot R to perform a task corresponding to the completion state based on at least one of a start state and an end state.
  • the robot control unit 160 causes the robot R to perform a task based on a path generated based on the start state and the end state.
  • the robot control unit 160 may cause the robot R to perform a task based on a path generated (regenerated) based on at least one of a redetermined start state and a redetermined end state.
  • Such robot control allows a task to be performed according to the skeletal information K of the workpiece W, thereby improving the accuracy or reliability of the task performed on the workpiece W and increasing the variety of workpieces W processed by the robot R.
  • FIG. 3A is a diagram showing an example of the operation of the robot system 100 related to learning the skeleton estimation model 111.
  • Figure 3B is a diagram showing an example of the operation of the robot system 100 related to the control of the robot R.
  • the robot system 100 executes the processes shown in Figures 3A and 3B.
  • Figures 4A and 4B are diagrams for explaining the results of the operation of the robot system 100.
  • step S101 the learning data generation unit 113 acquires a 3D model and skeleton information K for a specific workpiece W that corresponds to the type of workpiece to be estimated by the skeleton estimation model 111.
  • the learning data generation unit 113 acquires a 3D model of the workpiece W based on a sample image stored in the work storage unit 112, and acquires skeleton information K that corresponds to the sample image.
  • This skeleton information K includes a plurality of keys set in the 3D model.
  • the multiple keys include a plurality of key points and one or more skeletons that each connect two key points.
  • the learning data generation unit 113 generates a new data record of the learning data.
  • the learning data generation unit 113 deforms the acquired 3D model of the work W, updates the skeletal information K (multiple keys) corresponding to the deformed 3D model, and generates a new sample image obtained from the deformed 3D model.
  • the learning data generation unit 113 associates the new sample image with the updated skeletal information K (multiple keys) to generate a new data record.
  • the learning data generation unit 113 stores the new data record in the work memory unit 112 as at least a part of the learning data.
  • step S103 the learning unit 114 checks whether the amount of learning data required for learning the skeletal estimation model 111 has been prepared. If the amount of learning data is insufficient, the process returns to step S102, and the learning data generation unit 113 further generates new data records. If a sufficient amount of learning data has been prepared, the process proceeds to step S104.
  • step S104 the learning unit 114 generates a skeletal estimation model 111 through machine learning using the training data.
  • step S105 the learning unit 114 stores the generated skeletal estimation model 111 in the skeletal estimation unit 110.
  • learning data can be prepared by deforming a 3D model of the workpiece W, and the skeleton estimation model 111 can be trained using the learning data. Since the amount of learning data required for learning can be prepared without having to manually create a large number of 3D models, the construction costs of the robot system 100 can be reduced.
  • step S201 the camera C captures a workpiece image including the workpiece W.
  • the camera C captures a workpiece image as shown in FIG. 2B.
  • step S202 the skeleton estimation unit 110 estimates skeleton information K of the workpiece W included in the workpiece image.
  • the skeleton estimation unit 110 estimates skeleton information K for each of the multiple workpieces (broccoli) W in the workpiece image.
  • step S203 the work selection unit 120 selects a work W to be processed in the next operation based on the skeletal information K of each of the one or more works W.
  • step S204 the start state determination unit 130 determines pick position A as the start state.
  • broccoli near the top center is selected, and the start state determination unit 130 determines pick position A for the selected broccoli.
  • step S205 the completion state determination unit 140 determines the completion state of the work performed on the selected work W.
  • FIGS. 4A and 4B show examples of the completion state of the work. In these examples, the work is assumed to be pick-and-place of the work (broccoli) W.
  • FIG. 4A shows a state in which the broccoli are placed so that their orientations are not uniform
  • FIG. 4B shows a state in which the broccoli are placed so that their orientations are uniform.
  • the completion state determination unit 140 determines a state in which the orientations of the left and right broccoli are uniform as the completion state when placing the left broccoli shown in FIG. 4B.
  • the completion state determination unit 140 determines the position and posture of the left broccoli after placement so that the orientations of the left and right broccoli are uniform.
  • step S206 the end state determination unit 142 determines the end state of the robot R in the determined completion state.
  • step S207 the path generation unit 150 generates a path connecting the start state and the end state. As a result, it is determined that the robot R picks up the workpiece W in the start state and places the workpiece W in the end state, and a path connecting the start state and the end state is prepared.
  • step S208 it is confirmed whether the path generation unit 150 was able to generate a path. If the path cannot be generated, the process returns to step S204. In this case, at least one of the end state determination unit 142 redetermines the end state and the start state determination unit 130 redetermines the start state, and the path generation unit 150 generates (regenerates) a path based on at least one of the redetermined end state and the redetermined start state. If the path can be generated, the process proceeds to step S209.
  • step S209 the robot control unit 160 causes the robot R to perform the task corresponding to the determined completion state. If the task is pick-and-place, the robot R picks up the workpiece W in the determined start state, transports the workpiece W along the generated path, and places the workpiece W in the determined end state. In the example of FIG. 4B, even if the workpiece W has an irregular shape such as broccoli, the robot system 100 picks and places each workpiece W so that the orientation of the multiple workpieces W is aligned. This mechanism makes it possible to improve the quality of the task and increase the types of workpieces that can be processed.
  • step S210 the work selection unit 120 checks whether or not there is a work W to be processed in the next task (i.e., whether or not the task is to be repeated). If such a work W exists, the robot system 100 repeats the process from step S201 or step S203. If such a work W does not exist, the robot system 100 ends the process.
  • the above-mentioned series of processes may be executed by dedicated hardware or software.
  • the series of processes is executed by software, the above-mentioned series of processes can be realized by causing a general-purpose or dedicated computer to execute a program.
  • the computer may have a CPU (Central Processing Unit), a recording device such as a HDD (Hard Disk Drive), a ROM (Read Only Memory), or a RAM (Random Access Memory), a communication device connected to a network such as a LAN (Local Area Network) or the Internet, an input device such as a mouse or a keyboard, a drive for reading and writing removable storage media such as a magnetic disk such as a flexible disk, various optical disks such as CDs (Compact Discs), MO (Magneto Optical) disks, and DVDs (Digital Versatile Discs), and semiconductor memories, and an output device such as a display device such as a monitor, and an audio output device such as a speaker or headphones.
  • This computer may execute the above series of processes by executing a program recorded on a recording device or a removable storage medium, or a program acquired via a network.
  • FIG. 5 is a diagram showing an example of the hardware configuration of a computer 1000 used for a robot system according to the present disclosure.
  • the computer 1000 includes a main body 1100, an output device 1200, and an input device 1300.
  • the main body 1100 is a device having a circuit 1600.
  • the circuit 1600 has a processor 1601, a memory 1602, a storage 1603, an input/output port 1604, and a communication port 1605.
  • the number of each hardware component may be one or more.
  • the storage 1603 records a program for configuring each functional module of the main body 1100.
  • the storage 1603 is a computer-readable recording medium such as a hard disk, a non-volatile semiconductor memory, a magnetic disk, or an optical disk.
  • the memory 1602 temporarily stores a program loaded from the storage 1603, the calculation results of the processor 1601, etc.
  • the processor 1601 configures each functional module by executing a program in cooperation with the memory 1602.
  • the input/output port 1604 inputs/outputs an electrical signal between the output device 1200 or the input device 1300 in response to a command from the processor 1601.
  • the input/output port 1604 may input/output an electrical signal between another device.
  • the communication port 1605 performs data communication with other devices via the communication network N according to instructions from the processor 1601.
  • the output device 1200 is a device for outputting information from the main body 1100.
  • Examples of the output device 1200 include display devices such as various displays and speakers.
  • the input device 1300 is a device for inputting information to the main body 1100.
  • Examples of the input device 1300 include operation interfaces such as a keypad, a mouse, and an operation controller.
  • the output device 1200 and the input device 1300 may be integrated as a touch panel.
  • the main body 1100, the output device 1200, and the input device 1300 may be integrated as in a tablet computer.
  • Each functional module of the robot system is realized by loading a robot control program onto the processor 1601 or memory 1602 and having the processor 1601 execute the program.
  • the robot control program includes code for realizing each functional module of the robot system.
  • the processor 1601 operates the input/output port 1604 and the communication port 1605 in accordance with the robot control program, and executes reading and writing of data in the memory 1602 or the storage 1603.
  • the robot control program may be provided in a form recorded on a non-transitory recording medium such as a CD-ROM, DVD-ROM, or semiconductor memory.
  • the robot control program may be provided via a communications network as a data signal superimposed on a carrier wave.
  • the start state determination unit 130 determines the pick position A as the start state based on the skeleton information K, but the start state determination unit may determine the pick position A directly from the workpiece image. This modified example will be described with reference to FIG.
  • FIG. 6 is a diagram showing the configuration of a robot system 200 according to a modified example.
  • the robot system 200 differs from the robot system 100 in that it includes a start state determination unit 230 instead of the start state determination unit 130. A description of the same configuration as the robot system 100 will be omitted.
  • the start state determination unit 230 has a pick position estimation model 231.
  • the pick position estimation model 231 is generated in advance by machine learning using learning data including a plurality of data records. Each data record indicates a correspondence between a sample image based on a 3D model of the work W and pick information indicating whether or not the robot R can pick a position on the 3D model corresponding to a key set in the 3D model.
  • the pick information is associated with at least one of a plurality of keys set in the 3D model.
  • the plurality of keys includes a plurality of key points P and one or more skeletons S each connecting two key points P.
  • the pick position estimation model 231 is learned by the pick information associated with the keys as learning data.
  • the pick position estimation model 231 accepts input of a work image including the work W, it processes the work image without using a key and outputs the pick position A. Therefore, the start state determination unit 230 inputs the work image to the pick position estimation model 231 to determine the pick position A of the work W by the robot R.
  • the pick position estimation model 231 may be realized, for example, by a neural network model such as CNN.
  • the robot control unit 160 causes the robot R to perform a task on the workpiece W based on the end state and the pick position.
  • the workpiece processed by the robot may be a flexible object in which the individual difference in shape can be ignored but the difference in the degree of deformation between the individual objects may be large.
  • the skeleton estimation unit 110 may estimate a number of key points of the workpiece, calculate scores for the number of key points, and estimate the posture (e.g., orientation) of the workpiece as at least a part of the skeleton information based on the scores.
  • FIG. 7 is a diagram showing an example of key points P of the workpiece W estimated using the skeleton estimation model 111.
  • the skeleton estimation unit 110 estimates the orientation of the workpiece W as at least a part of the skeleton information. This estimation is also called head-tail determination.
  • the workpiece W is a conversion cable for transmitting signals between input/output terminals with different specifications.
  • the skeleton estimation model 111 is assumed to have been trained to estimate up to five key points P for the workpiece W. In estimating the key points P, the skeleton estimation model 111 sets an identifier for each key point P to distinguish each individual key point P.
  • the skeleton estimation model 111 estimates five key points P with identifiers 0 to 4 in order from the input/output terminal on the head side (the terminal on the right side in FIG. 7) to the input/output terminal on the rear side (the terminal on the left side in FIG. 7), which is an ideal estimation.
  • the skeleton estimation model 111 estimates only four key points P, and the identifiers are not arranged in order along the extension direction of the workpiece W.
  • the skeleton estimation unit 110 performs processing to accurately perform head-to-tail determination even when the skeleton estimation model 111 does not accurately estimate the key point P, as in situation 302.
  • the skeleton estimation unit 110 defines a weight corresponding to an identifier for each of a plurality of key points P that can be detected in advance.
  • the skeleton estimation unit 110 also sets a status for each key point P indicating whether or not the key point has been detected. The status is 1 if the key point has been detected, and 0 if the key point has not been detected.
  • the skeleton estimation unit 110 divides the work W into two regions based on the midpoint of the line segment connecting the first end and the second end of the work W, and determines in which of the two divided regions each detected key point P is located.
  • the skeleton estimation unit 110 For each of a plurality of key points P that can be detected, the skeleton estimation unit 110 calculates the product of the weight, the status, and a predetermined area value assigned to the divided region in which the key point P is located. Then, the skeleton estimation unit 110 calculates a score based on the sum of multiple products corresponding to the plurality of key points P. If the score satisfies a predetermined condition, the skeleton estimation unit 110 determines that the first end is a head-side input/output terminal, and if the score does not satisfy the predetermined condition, the skeleton estimation unit 110 determines that the second end is a head-side input/output terminal.
  • the skeleton estimation unit 110 estimates the identifier and position of each of the multiple key points P of the workpiece W.
  • the skeleton estimation unit 110 then calculates a score based on the identifier and position of each of the multiple key points P, and estimates the posture of the workpiece W as at least a part of the skeleton information based on the score.
  • the work selection unit 120 may use the score to select at least one work from the multiple works included in the work image.
  • the skeleton estimation unit 110 calculates a score for each of the multiple works. Based on the score of each of the multiple works, the work selection unit 120 selects at least one work as a work to be processed in the operation. For example, the work selection unit 120 may select a work whose score is equal to or greater than a predetermined threshold value, or may select a work whose score is the highest.
  • FIG. 8 is a diagram showing an example of the work selection process.
  • the work image G includes multiple workpieces (conversion cables) W stored in a box.
  • the skeleton estimation unit 110 calculates a score for each of the multiple workpieces W, and performs head/tail determination for each workpiece W.
  • the work selection unit 120 selects the workpiece W located at the top of the box, i.e., the workpiece W located closest to the camera C, based on the scores of the multiple workpieces W.
  • the results of the head/tail determination for the selected workpiece W are shown by the labels "head" and "tail", respectively.
  • the robot system 100, 200 may check the processed workpiece.
  • the camera C photographs and acquires a new workpiece image including the workpiece processed in the work performed by the robot R.
  • the skeleton estimation unit 110 estimates new skeleton information indicating the skeleton of the workpiece based on the new workpiece image.
  • the completion state determination unit 140 determines whether the processed workpiece satisfies the constraint condition based on the new skeleton information. If the processed workpiece satisfies the constraint condition, the robot system 100, 200 determines whether to have the robot R perform work on another workpiece. This process corresponds to step S210 described above.
  • the robot control unit 160 may have the robot R perform work again according to the new skeleton information so that the workpiece satisfies the constraint condition.
  • the completion state determination unit 140 may notify the user of the robot system 100, 200 of a message indicating that the constraint condition is not satisfied.
  • the constraint conditions include a condition that the work (conversion cable) W is placed within the placement area D.
  • the skeleton estimation unit 110 estimates new skeleton information K including multiple key points P of the processed work W.
  • the completion state determination unit 140 judges whether the work W satisfies the constraint conditions based on the skeleton information K. For example, the completion state determination unit 140 judges that the constraint conditions are satisfied if the proportion of the work W that falls within the placement area D is equal to or greater than a predetermined threshold, and judges that the constraint conditions are not satisfied if the proportion is less than the threshold.
  • the completion state determination unit 140 may also judge that the constraint conditions are satisfied if the number of key points P that fall within the placement area D is equal to or greater than a predetermined threshold, and judges that the constraint conditions are not satisfied if the number is less than the threshold. In situation 311, the completion state determination unit 140 judges that the work W satisfies the constraint conditions. In situation 312, the completion state determination unit 140 determines that the workpiece W does not satisfy the constraint condition. As described above, in situation 312, the robot control unit 160 may cause the robot R to again perform the task of placing the workpiece W in the placement area D in accordance with the new skeleton information K.
  • the robot system may include a functional module that generates constraint condition information.
  • This modified example will be described with reference to Fig. 10.
  • Fig. 10 is a diagram showing the configuration of a robot system 100A according to the modified example.
  • the robot system 100A differs from the robot system 100 in that it further includes a constraint generating unit 143. Descriptions of the same configuration as the robot system 100 will be omitted.
  • the constraint generation unit 143 is a functional module that generates constraint condition information based on a teaching image that indicates the target state of the work when the work is completed.
  • the teaching image prepared in advance may be generated by photographing an actual work, or may be generated by drawing a virtual work using computer graphics.
  • the constraint generation unit 143 estimates teaching skeletal information that indicates the skeleton of the work in the teaching image, and generates constraint condition information based on this teaching skeletal information.
  • the teaching skeletal information also includes multiple keys, each of which includes multiple key points and one or more skeletons that connect two key points.
  • the constraint generation unit 143 may estimate the teaching skeletal information from the teaching image using the skeleton estimation model 111.
  • the constraint generation unit 143 may set the attitude (e.g., orientation) of the work indicated by the teaching skeletal information as a constraint condition, and generate constraint condition information indicating this constraint condition.
  • the constraint generation unit 143 may display a user interface for generating constraint condition information, and generate the constraint condition information based on the estimated teaching skeleton information and user input received via the user interface. As described above, the constraint condition information is generated for each type of workpiece.
  • FIG. 11 is a diagram showing an example of a user interface for generating constraint condition information for pick-and-place operations.
  • the user interface 400 shown in this example includes an image display area 401 that displays a teaching image T specified by the user, an add area button 402 that is a button for defining a placement area D in which a work W is placed by a pick-and-place operation, a label 403 for specifying the type of work, and a setting completion button 404 for confirming the constraint condition information.
  • the user first displays the desired teaching image T in the image display area 401.
  • the teaching image T is an image showing the ideal state of the work (conversion cable) W to be placed by the pick-and-place operation, that is, the state of the work W after placement that the user desires the robot R to have.
  • the user drags the label 403 corresponding to the work W in the teaching image T to the work W on the teaching image T, and sets the conversion cable as the type of the work W.
  • the constraint generation unit 143 estimates teaching skeleton information that indicates at least the posture of the work W represented by the skeleton of the work W. Then, the constraint generation unit 143 generates constraint condition information that indicates the placement area D as at least a part of the constraint condition of the conversion cable based on the estimated posture and the specified frame F. In response to the user clicking or tapping the setting completion button 404, the constraint generation unit 143 stores the generated constraint condition information in the constraint condition information storage unit 141.
  • the constraint condition information can be referenced by the completion state determination unit 140 to determine the completion state of the pick-and-place operation performed on each work (conversion cable) W.
  • the constraint generation unit 143 may generate constraint condition information indicating the placement area D based on the attitude of the workpiece indicated by the teaching skeleton information, without accepting input of the frame F by the user.
  • the constraint generation unit 143 may display a dialogue screen for adjusting the automatically set placement area D in response to a specific user operation on the user interface 400, and change the position, shape, orientation, etc. of the placement area D based on the user input on the dialogue screen.
  • process flow described in the flowchart includes not only processes that are performed chronologically according to the order described, but also processes that are not performed chronologically but are performed in parallel or individually. Needless to say, even when processes are performed chronologically, the order of each step can be changed.
  • the hardware configuration of the system is not limited to a configuration in which each functional module is realized by executing a program.
  • each functional module may be configured with a logic circuit specialized for that function, or may be configured with an ASIC (Application Specific Integrated Circuit) that integrates the logic circuit.
  • ASIC Application Specific Integrated Circuit
  • a skeleton estimation unit that estimates skeleton information indicating a skeleton of a work based on a work image including the work existing in real space; a completion state determination unit that determines a completion state of the work performed on the work based on constraint condition information that represents constraint conditions related to the work on the work and the estimated skeleton information; a robot control unit that causes a robot arranged in the real space to perform the task corresponding to the determined completion state;
  • a robot system comprising: In this case, the completion state of the work performed on the workpiece is determined based on the skeleton of the workpiece, and the robot is controlled based on the completion state.
  • This processing makes it possible to have the robot perform delicate work according to the shape of the workpiece actually being processed or changes in the shape. Therefore, the reliability of the work performed on the workpiece can be improved. For example, the accuracy of the work performed on the workpiece can be improved. In one example, it is possible to increase the types of workpieces processed by the robot, i.e., to widen the range of robot systems available.
  • the operation includes a pick-and-place operation of the workpiece by the robot,
  • the completion state determination unit determines, as the completion state, at least a post-placement state which is a state after the work is placed.
  • the robot system of claim 1 In this case, the state of the workpiece placed by the robot is determined to be the completed state, so that the pick-and-place operation can be controlled with higher precision.
  • the workpiece is an irregular workpiece
  • the operation includes a pick-and-place operation of the irregular workpiece by the robot
  • the completion state determination unit determines at least the posture of the placed work as the post-placement state. 3.
  • An end state determination unit that determines an end state, which is a state of the robot in the determined completion state, based on the determined completion state, The robot control unit causes the robot to perform the task based on the determined end state.
  • the robot system according to any one of aspects 1 to 3.
  • the end state which is the state of the robot when the work is completed, is determined based on the completion state, and the robot is controlled based on that end state, allowing the robot to operate more accurately in response to the shape of the workpiece more flexibly.
  • a start state determination unit that determines a start state, which is a state of the robot at the start of the work, based on the skeleton information; a robot control unit that causes the robot to perform the task based on the determined start state and end state; 5.
  • the starting state which is the state of the robot at the start of work, is determined based on the skeletal state, making it possible to increase the success rate of the robot starting work and to control with greater precision the results at the completion of the work, which are affected by the state at the start of the work.
  • a path generating unit that generates a path of the robot based on the start state and the end state, The path generation unit When the path cannot be generated, at least one of redetermining the end state by the end state determination unit and redetermining the start state by the start state determination unit is executed; generating the path based on at least one of the re-determined end state and the re-determined start state; a robot control unit that causes the robot to perform the task based on the generated path; 6.
  • the robot system according to claim 5. In this case, a path for the robot to perform the work is generated, and if the path cannot be generated, the end state and the start state are redetermined and then the path is generated.
  • This automatic path generation reduces the frequency of the robot being unable to perform the work due to an inability to generate a path, and increases the probability of completing the work while dealing with various situations. For example, it increases the probability of performing the work reliably depending on the surrounding environment such as obstacles or the state of the work.
  • the skeleton information includes a plurality of keys including a plurality of key points of the work and one or more skeletons each connecting two of the key points;
  • the start state determination unit determines the start state based on work information associated with at least one key among the plurality of keys set for each type of work and the estimated skeletal information.
  • the robot system according to claim 5 or 6. In this case, by referring to the work information corresponding to the work in addition to the skeleton information of the work, the robot can be made to start the work more reliably. Also, by setting the work information, the user's intention regarding the work can be reflected in the robot control.
  • the operation includes a pick-and-place operation of the workpiece by the robot, the work information includes pick information indicating, for each of the at least one key, whether or not the robot can pick a position on the workpiece corresponding to the key;
  • the start state determination unit determines, as the start state, a pick position of the workpiece by the robot based on the pick information and the skeleton information.
  • the robot system of claim 7. In this case, the pick position for each type of work is set as work information, and the pick position of the work to be processed by the work is determined according to the work information and the skeleton information. This process increases the success rate of pick and place and controls the state of the work after placement.
  • a start state determination unit determines a pick position of the workpiece by the robot, The operation includes a pick-and-place operation of the workpiece by the robot,
  • the start state determination unit is a pick position estimation model learned from the learning data, the pick information including a plurality of data records indicating a correspondence between a sample image based on a 3D model of the workpiece and pick information indicating whether the robot can pick a position on the 3D model corresponding to a key set in the 3D model, the pick information being associated with at least one of a plurality of keys set in the 3D model, the plurality of keys including a plurality of key points and one or more skeletons each connecting two of the key points; determining the pick position based on the workpiece image and the pick position estimation model; a robot control unit that causes the robot to perform the task based on the end state and the determined pick position; 5.
  • the pick position can be determined and the robot can be operated without determining the starting state by using a pick position estimation model that has been trained in advance using learning data that indicates the correspondence between the sample image and the pick information associated with the skeletal information. Also, by introducing the pick position estimation model, the cost of constructing the robot system can be reduced and the accuracy of the pick position can be improved.
  • a work selection unit that selects at least one work from the plurality of workpieces included in the workpiece image, The skeleton estimation unit estimates the skeleton information for each of the plurality of works, The work selection unit selects the at least one work as a work to be processed in the operation based on the skeleton information of each of the plurality of workpieces.
  • a robot system according to any one of aspects 1 to 9. In this case, work that is predicted to be suitable for processing is selected based on the structure of each work, thereby increasing the probability of completing the work and improving the quality of the work results.
  • the work selection unit selects a workpiece that is entirely exposed in the workpiece image as a workpiece to be processed in the operation based on the area of each of the plurality of workpieces and the skeleton information.
  • the robot system according to aspect 10. a workpiece that is entirely exposed in the workpiece image, i.e., a workpiece that is expected to be easy to process, is selected, thereby further increasing the probability of completing the task.
  • the workpiece selection unit selects a workpiece having a degree of deformation within a predetermined range as a workpiece to be processed in the operation based on the skeletal information of each of the plurality of workpieces. 11.
  • the robot system according to aspect 10. In this case, a workpiece that is relatively close to a typical posture of a workpiece, that is, a workpiece that is expected to be easy to process, is selected, so that the quality of the work result can be further improved.
  • the skeleton estimation unit includes: Calculating scores for a plurality of key points of the workpiece; Estimating a posture of the workpiece as at least a part of the skeletal information based on the score.
  • a robot system according to any one of aspects 1 to 12. In this case, since the orientation of the workpiece is estimated based on the scores calculated for a plurality of key points of the workpiece, the orientation of the workpiece can be estimated more accurately, thereby improving the accuracy of the work performed on the workpiece.
  • the skeleton estimation unit estimates new skeleton information indicating a skeleton of the processed workpiece based on a new workpiece image including the workpiece processed in the operation performed by the robot,
  • the completion state determination unit determines whether the processed workpiece satisfies the constraint condition based on the estimated new skeleton information.
  • a robot system according to any one of aspects 1 to 13. In this case, whether the processed workpiece satisfies the constraints is determined based on the skeleton of the workpiece. With this mechanism, it is possible to automatically monitor whether the robot has properly processed the workpiece.
  • the robot control unit causes the robot to perform the work again according to the estimated new skeleton information.
  • the robotic system of claim 14. In this case, if the processed workpiece does not satisfy the constraints, the robot reprocesses the workpiece based on the current skeleton of the workpiece, which can further improve the quality of the work result.
  • the operation includes a pick-and-place operation of the workpiece by the robot,
  • the constraint generation unit Estimating the posture of the workpiece in the teaching image as at least a part of the teaching skeleton information; generating the constraint information, based on the estimated posture, the constraint information representing a placement area, which is an area in which the work is placed by the pick-and-place operation, as at least a part of the constraint; 17.
  • the robot system of claim 16 In this case, since the work placement area is set based on the orientation of the work in the teaching image, an appropriate placement area suited to the type of work can be easily set.
  • a robot control method executed by a robot system having at least one processor comprising: A step of estimating skeleton information indicating a skeleton of a workpiece based on a workpiece image including the workpiece existing in a real space; determining a completion state of the work performed on the workpiece based on constraint condition information representing constraint conditions related to the work performed on the workpiece and the estimated skeleton information; causing a robot arranged in the real space to perform the task corresponding to the determined completion state;
  • a robot control method comprising: In this case, the completion state of the work performed on the workpiece is determined based on the skeleton of the workpiece, and the robot is controlled based on the completion state.
  • This processing makes it possible to have the robot perform delicate work according to the shape of the workpiece actually being processed or changes in the shape. Therefore, the reliability of the work performed on the workpiece can be improved. For example, the accuracy of the work performed on the workpiece can be improved. In one example, it is possible to increase the types of workpieces processed by the robot, i.e., to widen the range of robot systems available.
  • a robot control program that causes a computer to execute the above.
  • the completion state of the work performed on the workpiece is determined based on the skeleton of the workpiece, and the robot is controlled based on the completion state.
  • This processing makes it possible to have the robot perform delicate work according to the shape of the workpiece actually being processed or changes in the shape. Therefore, the reliability of the work performed on the workpiece can be improved. For example, the accuracy of the work performed on the workpiece can be improved. In one example, it is possible to increase the types of workpieces processed by the robot, i.e., to widen the range of robot systems available.
  • the skeleton estimation unit estimates the skeleton information based on the workpiece image using a skeleton estimation model learned from learning data including a sample image based on a 3D model of the workpiece and a plurality of data records including a plurality of keys set in the 3D model, the plurality of keys including a plurality of key points and one or more skeletons each connecting two of the key points;
  • the skeleton information is estimated by a skeleton estimation model that is trained in advance using learning data that indicates the correspondence between the sample image and a plurality of keys corresponding to the workpiece.
  • (Aspect 21) a training data generation unit that transforms the 3D model represented by one of the plurality of data records to generate a new data record of the training data; a learning unit that learns the skeleton estimation model using at least the new data record; 21.
  • a work selection unit that selects at least one work from the plurality of workpieces included in the workpiece image, The skeleton estimation unit calculates the score for each of the plurality of works, The work selection unit selects the at least one work as a work to be processed in the operation based on the scores of each of the plurality of works. 14.

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Multimedia (AREA)
  • Manipulator (AREA)
PCT/JP2024/018169 2023-06-15 2024-05-16 ロボットシステム、ロボット制御方法、およびロボット制御プログラム Ceased WO2024257546A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2025527584A JPWO2024257546A1 (https=) 2023-06-15 2024-05-16
DE112024001966.8T DE112024001966T5 (de) 2023-06-15 2024-05-16 Robotersteuerungssystem, Robotersteuerungsverfahren und Robotersteuerungsprogramm
US19/415,781 US20260097515A1 (en) 2023-06-15 2025-12-11 Robot control based on skeleton of workpiece

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2023098409 2023-06-15
JP2023-098409 2023-06-15

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US19/415,781 Continuation US20260097515A1 (en) 2023-06-15 2025-12-11 Robot control based on skeleton of workpiece

Publications (1)

Publication Number Publication Date
WO2024257546A1 true WO2024257546A1 (ja) 2024-12-19

Family

ID=93852081

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2024/018169 Ceased WO2024257546A1 (ja) 2023-06-15 2024-05-16 ロボットシステム、ロボット制御方法、およびロボット制御プログラム

Country Status (4)

Country Link
US (1) US20260097515A1 (https=)
JP (1) JPWO2024257546A1 (https=)
DE (1) DE112024001966T5 (https=)
WO (1) WO2024257546A1 (https=)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008087074A (ja) * 2006-09-29 2008-04-17 Fanuc Ltd ワーク取り出し装置
WO2022085408A1 (ja) * 2020-10-19 2022-04-28 三菱電機株式会社 ロボット制御装置およびロボット制御方法
US20230071488A1 (en) * 2021-09-01 2023-03-09 Mujin, Inc. Robotic system with overlap processing mechanism and methods for operating the same
JP2023518071A (ja) * 2020-03-18 2023-04-27 リアルタイム ロボティクス， インコーポレーテッド ロボットの動作計画に有用なロボット操作環境のデジタル表現

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021084587A1 (ja) 2019-10-28 2021-05-06 株式会社安川電機 機械学習データ生成装置、機械学習装置、作業システム、コンピュータプログラム、機械学習データ生成方法及び作業機械の製造方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008087074A (ja) * 2006-09-29 2008-04-17 Fanuc Ltd ワーク取り出し装置
JP2023518071A (ja) * 2020-03-18 2023-04-27 リアルタイム ロボティクス， インコーポレーテッド ロボットの動作計画に有用なロボット操作環境のデジタル表現
WO2022085408A1 (ja) * 2020-10-19 2022-04-28 三菱電機株式会社 ロボット制御装置およびロボット制御方法
US20230071488A1 (en) * 2021-09-01 2023-03-09 Mujin, Inc. Robotic system with overlap processing mechanism and methods for operating the same

Also Published As

Publication number Publication date
US20260097515A1 (en) 2026-04-09
JPWO2024257546A1 (https=) 2024-12-19
DE112024001966T5 (de) 2026-02-26

Similar Documents

Publication Publication Date Title
CN114599488B (zh) 机器学习数据生成装置、机器学习装置、作业系统、计算机程序、机器学习数据生成方法及作业机的制造方法
JP6640060B2 (ja) ロボットシステム
CN104057447B (zh) 机器人拾取系统以及被加工物的制造方法
JP2021536068A (ja) 物体姿勢推定方法および装置
CN110769985A (zh) 使用循环神经网络的机器人末端执行器的视点不变的视觉伺服
CN114516060A (zh) 用于控制机器人装置的设备和方法
JP6632095B1 (ja) 学習済モデル生成装置、ロボット制御装置、及び、プログラム
CN111015656A (zh) 一种机器人主动避障的控制方法、装置及存储介质
JP7259860B2 (ja) ロボットの経路決定装置、ロボットの経路決定方法、プログラム
WO2019064916A1 (ja) ロボットシミュレータ
JP2019171501A (ja) ロボットの干渉判定装置、ロボットの干渉判定方法、プログラム
JP2019171498A (ja) ロボットプログラム実行装置、ロボットプログラム実行方法、プログラム
CN114474106A (zh) 用于控制机器人装置的方法和机器人控制装置
CN115319734A (zh) 用于控制机器人设备的方法
US20250291333A1 (en) Self-learning process for a manufacturing and/or assembly process
JP7151713B2 (ja) ロボットシミュレータ
EP4052865A1 (en) Information processing apparatus, robot system, information processing method, program, and recording medium
CN118893633A (zh) 一种模型训练方法、装置和机械臂系统
CN117260702A (zh) 用于控制机器人来操纵、尤其是拾取对象的方法
WO2020059342A1 (ja) ロボットシミュレータ
WO2024257546A1 (ja) ロボットシステム、ロボット制御方法、およびロボット制御プログラム
JPWO2019064919A1 (ja) ロボット教示装置
JP7832919B2 (ja) 制御支援システム、制御支援方法、および制御支援プログラム
CN113867137A (zh) 用于操作机器的方法和装置
JP7167925B2 (ja) ロボット教示装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24823168

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2025527584

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 112024001966

Country of ref document: DE

WWP Wipo information: published in national office

Ref document number: 112024001966

Country of ref document: DE