JP2018051652A - Robot system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a system which allows an operator to show working motion to a robot, so that the robot which has a mechanism different from an operator can achieve motion which the operator intends to achieve, even in an environment different from an environment at the time when the robot is taught.SOLUTION: A robot system according to one embodiment of the invention memorizes a plurality of kinds of original scenes, combinations of operation information on a robot and information on an object, and memorizes a plurality of derived scenes derived from the original scenes, associating the derived scenes with the original scenes; when operating the robot according to teaching motion of an operator measured by a measuring portion, firstly selects an original scene most similar to the teaching motion of the operator and teaching information constituted from kinds of objects out of the plurality of kinds of original scenes; subsequently selects a derived scene most similar in position and size to a measured object out of derived scenes derived from the selected scene; and operates the robot on the basis of operation information included in the selected derived scene.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a system and method for teaching an operator task to a robot.

  Automation by robots is performed in a wide range of fields such as manufacturing and transportation. In order to automate a predetermined work with a robot, a work for teaching the robot the procedure of the operation is necessary, and this is usually performed in advance by an operator.

  There is a teaching playback method as a method of teaching the robot the operation. Teaching playback is an operation in which the operator teaches the operation procedure of each joint angle of the robot in advance, and the robot reproduces the procedure during automatic operation. However, teaching by teaching playback involves inputting the motion of each joint one by one, and the cost of teaching is high.

  On the other hand, Patent Document 1 discloses a method that can easily teach the robot the same operation as the operation of the operator. The robot control system described in Patent Document 1 acquires the operation of the operator using teaching information acquisition means such as a camera, and converts the acquired finger position (coordinate) information into joint angle information of the robot hand, Create robot hand teaching data. Based on the created robot hand teaching data, the robot hand is driven.

JP 2011-110620 A

  According to the method disclosed in Patent Document 1, even if the mechanisms of the robot and the operator are different, the operation of the operator can be correctly transmitted to the robot. However, if the position or size of the work object handled by the robot changes, it is necessary to change the robot operation accordingly.

  For this reason, every time the position or size of the target object changes, it is necessary to teach the robot the operator's action on the changed target object. Therefore, it is necessary to teach a very large number of types of operator actions.

  An object of the present invention is to provide a method for causing a robot to realize an operation intended by an operator even in an environment different from that at the time of teaching.

  A robot system according to an embodiment of the present invention includes an operation measuring unit that measures a teaching operation of an operator, an object measuring unit that measures the type, position, and size of an object operated by the operator or the robot, A scene storage unit that stores a plurality of types of scenes that are sets of motion information and object information, a derived scene storage unit that stores a plurality of derived scenes derived from a scene in association with the scene, and a scene storage unit A scene selection unit that acquires one scene from the scene, a derived scene selection unit that acquires one derived scene from the derived scene storage unit, and a robot that operates the robot based on operation information included in the derived scene And an operation execution unit. Here, the information on the target object included in the derived scene is obtained by changing the position and size of the target object included in the scene associated with the derived scene. This is operation information for operating an object included in the derived scene.

  When the robot is operated in accordance with the operator's teaching operation, the scene having the highest similarity with the teaching information of the operator's teaching operation and the type of the object among the plurality of scenes stored in the scene storage unit by the scene selection unit Select. The derivation scene selection unit selects a derivation scene having the highest measured position / size and similarity among the derivation scenes derived from the scene selected by the scene selection unit. Based on the motion information included in the selected derived scene, the robot motion execution unit operates the robot.

  According to the present invention, it is possible for the robot having a mechanism different from that of the operator to realize the operation intended by the operator even in an environment different from that at the time of teaching by showing the operation of the operation to the robot. .

1 is a functional configuration diagram of a robot system according to Embodiment 1. FIG. It is a figure which shows the physical structure of a robot system. It is a figure explaining the outline | summary of the procedure at the time of operating a robot. It is explanatory drawing of a measurement part. It is a figure explaining the outline | summary of a scene selection part. It is a figure explaining the outline | summary of the derived scene preparation part. It is a figure explaining the outline | summary of the derived scene selection part. It is an example of the interface which a scene addition part provides. It is a figure explaining the outline | summary of a robot operation | movement selection part. It is a figure explaining the outline | summary of an operation correction part. It is an example of the correction method of the robot operation | movement by an operation correction part. It is a figure explaining the outline | summary of a collision avoidance part. It is a figure explaining the outline | summary of an operation | movement update part. FIG. 6 is a functional configuration diagram of a robot system according to a second embodiment. FIG. 9 is a functional configuration diagram of a robot system according to a third embodiment.

  Hereinafter, embodiments of the present invention will be described. First, an outline of the present invention will be described. A robot system according to an embodiment of the present invention registers information related to robot operations in advance in a database, and selects the robot operation closest to the operator's operation from the database when teaching the operator's operation to the robot. In this way, the robot performs the same operation as the operator with only one teaching.

  Here, when selecting the robot motion information on the database closest to the taught operator motion, it is necessary to determine the similarity between the operator motion and the robot motion. However, since the operator motion and the robot motion are different in shape and mechanism, the similarity between them cannot be directly calculated. On the other hand, by using a conventional machine learning method and giving a sample of the above correspondence example in advance, the similarity calculation method can be estimated, and when a new operator action is input as teaching information In addition, it is possible to select the closest data from the database.

  However, in machine learning, it is necessary to collect a sufficient number of samples in order to estimate the correspondence, but in order to be able to teach a wide variety of motions, the types, objects, and sizes of robot motions It is necessary to register all the various conditions such as position and location in the database, and to create a large number of correspondence examples that can cover them. In general, collecting human motion samples is expensive and it is difficult to prepare all environmental conditions in advance. In other words, the correspondence between the operator movement and the robot movement is estimated by machine learning, the shape of the object and the operation method are recognized, and if the operation method of the object is known, the movement is corrected even if some environmental changes occur Only by using the operation correction method such as performing the above, there is a problem that it is necessary to collect samples regarding objects of all shapes, positions, and sizes, which is expensive.

  In the robot system according to the embodiment of the present invention, the association between the operator action and the robot action is only for a typical case (referred to as a “scene” in this specification) related to the robot action / work object. A more specific example (referred to as a “derived scene” in this specification) in which detailed information such as the position and size of the object is added to the scene is created by simulation. , And select according to the state of the environment.

  For example, in the case of grasping as a target, the database relating to the scene includes a method of grasping the object (gripping from above, grasping from the side, grasping with both hands, etc.), type of object shape (rectangular column, sphere, cylinder, etc.) Only information related to) is registered in advance. However, when the learning cost does not increase, such as when there are few types of object shapes, position and size information may be added to the scene. If the position / size information is also added to the scene, the difference between the scene and a derived scene to be described later is reduced, so that the robot motion in the derived scene can be easily calculated.

  In the database relating to the derived scene, information including the position and size of the object with respect to the scene, and the operation procedure of the robot corresponding to the information are registered. By doing in this way, the sample of the example necessary for calculating the correspondence relationship between the operator motion and the robot motion described above is only two conditions regarding the gripping method and the shape type, and the sample collection cost can be reduced. In addition, the accuracy of calculating the similarity increases as the number of conditions decreases. Furthermore, the derived scene is selected based on the difference in the position and size of the object according to the state of the environment. Here, for the similarity between the information such as the position and size of the target object in the environment where the robot actually operates and the information recorded in the derived scene, the degree of coincidence of the shapes in the space may be obtained. Easy calculation is possible.

  A robot system according to an embodiment of the present invention includes a representative example (scene) of a robot operation / work object and a specific example (derivation) in order to select a robot operation similar to a teaching operation by an operator. Scenes) are separately registered in the database, and the selection process is divided into selection of the scene closest to the operator action and selection of the derived scene closest to the actual environment. As a result, it is possible to separate the similarity calculation processing of the different motions of the operator's motion and the robot motion and the processing of comparing the degree of coincidence of the shape such as the comparison of the position / size of the object that is easy to calculate. This makes it possible to operate the robot in any situation while suppressing the number of example samples required for calculating the correspondence between the human motion and the robot motion.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a functional configuration diagram of the robot system according to the first embodiment. First, the motion measuring unit 100 is for measuring an operator's motion, and the object measuring unit 101 is for measuring an article to be operated. The environment measuring unit 102 is for measuring surrounding conditions when the robot is operating.

  The scene storage unit 103 stores in advance typical cases (scenes) of robot operation / work objects. In an example in the present embodiment, information on an operation method such as how to hold an object and shape examples of the operation object are recorded. Shape examples may be classified according to the type of shape related to the operation method of the robot. For example, if there are three types of gripping methods according to the type of shape, three types of scenes may be prepared.

  The scene selection unit 104 selects a combination of the operator action measured by the action measurement unit 100 and the type of the object measured by the object measurement unit 101 from the scenes stored in the scene storage unit 103 (this is “teach” The most similar scene is selected. Since the information recorded in the scene is information about the operation method (robot operation) of the robot, the scene selection unit 104 needs to obtain the similarity between different types of information, that is, the operator operation and the robot operation. Therefore, in the robot system according to the present embodiment, the learning unit 115 learns the correspondence between the operator motion and the robot motion in advance, and the scene selection unit 104 selects a scene based on the learned relationship. Details will be described later.

  The derived scene creation unit 105 is a functional block for creating a derived scene of the scene stored in the scene storage unit 103. Usually, a plurality of derived scenes are created for one scene. A “derived scene” is a scene in which the position and size of an object recorded in the original scene are changed. The derived scene is associated with a robot operation method for appropriately operating the object whose position and size are changed. The created derived scene is stored in the derived scene storage unit 106.

  The derived scene selection unit 108 compares the shape of the object measured by the object measurement unit 101 with the shape of the object stored in the derived scene storage unit 106, so that the environment in which the robot actually operates (hereinafter referred to as the environment) Then, this is called “real environment”).

  In the robot system according to the present embodiment, the robot motion execution unit 112 drives the robot's hands and fingers based on the derived scene selected by the derived scene selection unit 108, so that the robot performs according to the operation taught by the operator. To work. Note that the environment in which the operator performs the teaching operation may not always match the environment in which the robot is placed. In this case, since it is necessary to correct the robot motion recorded in the derived scene, the robot system may include a robot motion selection unit 109, a motion correction unit 110, and a collision avoidance unit 111 described below.

  The robot motion selection unit 109 selects which operation is appropriate from the situation of the entire environment when two or more types of robot motions are recorded in association with the derived scene selected by the derived scene selection unit 108. To do. Selection criteria include selecting a more stable joint angle or a shorter trajectory.

  When the position / size of the target object in the real environment differs from the position of the target object stored in the derived scene, the motion correcting unit 110 matches the position / size of the target object in the real environment. Correct the robot movement. As a correction method, the above-described linear interpolation or the like is preferably used.

  The collision avoidance unit 111 changes the movement trajectory from the state of the surrounding environment measured by the environment measurement unit 102 so that the robot does not collide with surrounding objects. For example, the collision avoidance unit 111 changes the path until the robot's hand reaches the target and the shape of the arm other than the hand. The robot motion execution unit 112 controls the robot based on the robot motion information corrected by the robot motion selection unit 109, the motion correction unit 110, and the collision avoidance unit 111. For example, the robot operation execution unit 112 issues a command group for changing the angle of each joint to the robot, and the robot operates in response to this.

  The scene addition unit 107 and the derived scene addition unit 113 are functional blocks for creating a scene / derived scene and adding the created scene / derived scene to the scene storage unit 103 / derived scene storage unit 106. For example, it is used when it is desired to add a scene in which a new type of motion is recorded or when an unlearned situation occurs as a result of the robot motion. In this embodiment, since the creation (addition) of the scene (derived scene) is performed by the user (operator) of the robot system, the scene adding unit 107 (derived scene adding unit 113) displays information that the operator configures the scene. A user interface (eg, a graphical user interface) used to create is provided to the operator.

  The user of the robot system uses the scene addition unit 107 to create information on the robot operation to be created and information on the target article operated by the operation, and stores the information in the scene storage unit 103. The target article is created by a CAD tool or the like, and information on the robot's operation with respect to the article is set using a robot simulator tool. Further, after setting the robot operation, the user of the robot system associates the created scene with operator operation information (teaching information) corresponding to the scene, and sets the associated scene and teaching information as a correspondence storage unit. The machine learning is performed again on all the learning data stored in 114 and stored in the correspondence storage unit 114.

  The derived scene adding unit 113 adds a derived scene that becomes the position / size of a new object to a predetermined scene. At the time of addition, the position / size information of the object is input and passed to the derived scene creation unit 105, so that the derived scene creation unit 105 creates a derived scene by calculating the robot motion for the object. To the derived scene storage unit 106.

  FIG. 2 is a diagram illustrating a physical configuration of the robot system according to the present embodiment. The robot system according to this embodiment includes a processor 1400, a memory 1401, an auxiliary storage device 1402, a communication interface 1403, an input interface 1404, an output interface 1405, and a sensing device 1406. In FIG. 2, “interface” is abbreviated as “I / F”. Also in this specification, “interface” may be abbreviated as “I / F”.

  The processor 1400 executes a program stored in the memory 1401. The memory 1401 includes a ROM that is a nonvolatile storage device and a volatile RAM. The ROM stores an invariant program (for example, BIOS). The RAM is a volatile storage device such as a DRAM (Dynamic Random Access Memory), and temporarily stores a program executed by the processor 1400 and data used when the program is executed.

  The auxiliary storage device 1402 is a large-capacity non-volatile storage device, and stores a program executed by the processor 1400 and data (for example, a scene and a derived scene) used when the program is executed. That is, the program is read from the auxiliary storage device 1402, loaded into the memory 1401, and executed by the processor 1400. As the auxiliary storage device 1402, for example, a magnetic storage device such as a hard disk drive (HDD) or a storage device using a nonvolatile semiconductor memory such as Solid State Drive (SSD) is used.

  In the robot system according to the present embodiment, a program loaded in the memory 1401 is executed by the processor 1400, so that the computer 1410 and the sensing device 1406 are connected to the motion measuring unit 100, the object measuring unit 101, and the like described in FIG. Environment measurement unit 102, scene storage unit 103, scene selection unit 104, derived scene creation unit 105, derived scene storage unit 106, scene addition unit 107, derived scene selection unit 108, robot motion selection unit 109, motion correction unit 110, collision The avoidance unit 111, the robot operation execution unit 112, the derived scene addition unit 113, the correspondence storage unit 114, and the learning unit 115 function. The storage area of the auxiliary storage device 1402 is used as a storage area in which the scene storage unit 103, the derived scene storage unit 106, and the correspondence storage unit 114 store information such as scenes.

  The communication interface 1403 is an interface device that communicates with other devices according to a predetermined protocol. The interface device is a port capable of communication such as Ethernet, USB, and RS-232C. For example, in the case of Ethernet, communication is performed between devices using TCP communication, UDP communication, or the like. The robot system may be configured to be connectable to a wide area network 1420 such as the Internet via the communication interface 1403. In that case, the robot system may download data from a base having a large-capacity storage means such as the data center 1100, or conversely upload data held by itself to the data center 1100.

  The input interface 1404 is an interface device for connecting input devices such as a keyboard 16 and a mouse 17 used by a user (operator) of the robot system. The output interface 1405 is an interface device for connecting a display device such as the display 19. The robot system accepts input from an operator such as scene creation and additional learning through the input interface 1404, and displays the processing result on the display 19 through the output interface 1405.

  The sensing device 1406 is a device for measuring a teaching operation by an operator and measuring an object, an ambient environment, and the like. The sensing device 1406 is an operation measuring unit 100, an object measuring unit 101, an environment described with reference to FIG. It is a device constituting the measurement unit 102. The sensing device 1406 is a device such as a camera, a motion sensor, or a laser distance sensor. However, any device may be used for the sensing device 1406 as long as the device can measure the position, size, or operation of the article.

  The robot system may include a plurality of sensing devices 1406. Alternatively, the robot system may include a plurality of types of sensing devices 1406. For example, the sensing device 1406 may include a camera and a motion sensor. The robot system transmits a control command to the robot 1407 while acquiring information of the sensing device 1406 via the communication interface 1403.

  The physical configuration of the robot system is not necessarily limited to the configuration shown in FIG. For example, in FIG. 2, only one computer 1410 is shown, but a robot system may be composed of a plurality of computers. Alternatively, the components in the computer 1410 may be incorporated in the robot 1407, and the robot 1407 may be configured to execute each function of the robot system described below. Further, the sensing device 1406 may be incorporated in the robot.

  Next, the procedure when the robot system according to the present embodiment operates the robot using the functional blocks described above will be outlined with reference to FIG.

  In the robot system according to the present embodiment, the process for operating the robot is roughly divided into two processes, a teaching process 1200 and an operation process 1201. In the teaching process 1200, the motion measuring unit 100 and the object measuring unit 101 measure a teaching operation (how an operation object is operated and how) performed by the operator 1202, and based on the measurement result. Determine the action (robot action) to be executed by the robot. FIG. 3 shows an example in which an operation in which the operator 1202 holds the object 1203 with one hand is measured as an example. However, the type of motion is not limited to gripping as long as the motion measurement unit 100 can measure.

  When the teaching operation of the operator 1202 is measured, the scene selection unit 104 selects a scene most similar to the combination of the measured operation of the operator 1202 and the object 1203 from the scenes stored in the scene storage unit 103. To do. Subsequently, the derived scene selection unit 108 extracts (a plurality of) derived scenes associated with the selected scene from the derived scene storage unit 106. Then, the derived scene selection unit 108 compares the position / size of the object recorded in each extracted derived scene with the position / size of the measured object 1203, and compares the measured object 1203 with the measured object 1203. A derived scene in which an object having the most similar position and size is recorded is selected. This is the process performed in the teaching process 1200.

  When the derived scene is selected, the robot system next performs an operation step 1201 of the robot. In the operation step 1201, the robot 1204 is driven based on the selected derived scene. For example, the robot motion execution unit 112 creates a group of commands (such as a command to change the angle of each joint) for actually moving the robot from the information about the robot motion recorded in the selected derivative scene. And issued to the robot 1204. The robot 1204 operates based on the received command group. By performing such a process, in the robot system according to the present embodiment, the robot 1204 operates in real time according to the operation taught by the operator.

  When the robot operation execution unit 112 operates the robot 1204, the object measurement unit 101 recognizes the detailed position of the object 1205 that is the operation target of the robot, and the operation correction unit 110 records it in the derived scene based on the position. Information on the robot operation being performed (such as a trajectory during gripping) may be corrected. Further, the robot operation may be changed so that the arm of the robot does not collide with the presence object 1206 in the environment based on the measurement result of the environment avoidance unit 111 environment measurement unit 102.

  Next, details of each functional block will be described. First, each functional block used mainly in the teaching process will be described.

  FIG. 4 shows the measurement units (the motion measurement unit 100, the object measurement unit 101, and the environment measurement unit 102). The motion measurement unit 100 measures the motion of the operator 200 and passes the motion information of the operator 200 to the scene selection unit 104 or the like. As a sensing device that constitutes the motion measurement unit 100, a device that can measure the motion of the operator 200, such as a motion sensor or a camera, may be used.

  The object measurement unit 101 measures the object 201 that is an operation target of the operator 200 and the object 201 that is an operation target of the robot. Information measured by the object measuring unit 101 is passed to the scene selection unit 104, the derived scene selection unit 108, the operation correction unit 110, and the like. As a sensing device constituting the object measuring unit 101, a device capable of measuring the position, posture, and shape of the object 201 such as a camera or a laser distance sensor may be used. In the actual environment, when not only the object 201 but also the surrounding shape are measured together, only the object 201 may be extracted by an ICP algorithm, template matching, or the like.

  The environment measurement unit 102 measures the shape of the environment 202 and passes the shape information of the environment 202 to the collision avoidance unit 111 and the like. As a sensing device constituting the environment measuring unit 102, a device capable of measuring the shape of the surrounding environment such as a camera or a laser distance sensor may be used.

  Different devices may be used as the sensing devices constituting the motion measurement unit 100, the object measurement unit 101, and the environment measurement unit 102, respectively. Other configuration examples include a case where only the motion measurement unit 100 is a motion sensor, the object measurement unit 101 and the environment measurement unit 102 are one shape measurement sensor, or a case where all are one shape measurement sensor. In a case where one sensing device functions as two or more types of measuring units, a means for extracting or separating necessary information (for example, information on an object or information on the surrounding environment) from sensor information obtained from the sensing device Should be provided.

  When the functions of the object measurement unit 101 and the environment measurement unit 102 are realized by a single sensing device, the object 201 may be extracted from the environment 202 by template matching, an ICP algorithm, or the like as described above. Further, when one sensing device also functions as the motion measuring unit 100, it is preferable to provide means for extracting person information from the sensor information and extracting it as the motion of the operator 200. As the extraction method, an existing person recognition method may be used. Furthermore, in the scene selection unit 104, even if the operation of the operator 200 is not explicitly measured, if only the appearance image of the environment 202, the object 201, and the operator 200 is associated with the scene, the teaching operation of the operator 200 is measured. It is not necessary to classify in.

  The robot system may have two object measuring units 101. In that case, one object measuring unit 101 measures an object (hereinafter referred to as “teaching object”) that is operated (gripped or the like) by the operator when the operator performs the teaching operation. The other object measuring unit 101 is used for observing an object to be operated by the robot when the robot performs an execution operation (referred to as an “operation object” in this specification). .

  FIG. 5 shows an outline of processing performed by the scene selection unit 104. The scene selection unit 104 selects from the scene storage unit 103 the scene closest to the operation of the operator 300 and the shape of the target object 301 measured by the motion measurement unit 100 and the target object measurement unit 101. The scene is recorded by pairing the type (shape) of the object 301 and the operation method (robot operation) of the robot with respect to the object 301, and a plurality of scenes are recorded in the scene storage unit 103. .

  When a scene (or a derived scene) is stored in the scene storage unit 103 or the like, it is necessary to convert the shape of the object and the robot operation into information of some form and store the information. May be adopted. In one example of the present embodiment, the shape of the object is expressed by CAD data, and as an operation method (robot operation) by the robot, information on how much force is applied to which position on the surface of the object, and an arm It is assumed that the set value of the joint angle when extending is recorded. The information on the target object included in the scene only needs to include information that can specify the type (shape) of the target object, but may include the position and size of the target object. In this embodiment, the scene will be described on the assumption that the position and size of the object are included unless otherwise specified.

  When the scene selection unit 104 selects information on the robot motion that is closest to the measured motion of the operator 300, it is necessary to determine the similarity of how similar the motion of the operator 300 is to the robot motion. However, since the operation of the operator 300 and the robot operation have different qualities such as differences in shape and mechanism, the similarity between the two cannot be directly calculated by the difference.

  Therefore, in the robot system according to the present embodiment, the correspondence between both objects having different qualities is solved by machine learning. Specifically, in the robot system, the operation result of the operator 300 and the measurement result (teaching information) of the object 301 and the corresponding case of the scene are stored in the correspondence storage unit 114 in advance, the teaching information is input data, and the scene is the teacher. The learning unit 115 learns the classification problem as data. As a machine learning method, a method such as SVM, Boosting, Neural Network may be used. In addition, machine learning may not be used as long as it is a technique for solving two types of correspondence relationships having different qualities.

  The types of measurement results given as input include the joint angle of the finger of the operator 300 and the shape of the object, or image data of the entire space. In the former case, it is necessary to cut out only the motion sensor and the object, but the number of learning samples can be reduced because the dimension of the input data is low. When learning is performed only from an image as in the latter case, it is not necessary to combine the measurement units into one camera for classification, but the number of samples used for learning increases.

  Using the learning instruction unit 1000, the operator associates the teaching information with the corresponding robot operation information and stores them in the correspondence storage unit 114, and issues a learning instruction to the learning unit 115. The learning unit 115 performs learning for selecting a scene closest to the operation of the operator, so that the teaching information is the information constituting the scene (the information on the shape of the object and the robot operation, and is the same as the scene or the derived scene). Deriving rules for conversion to information containing types of information). As an example, the learning unit 115 generates a function having the teaching information as input and the information constituting the scene as output.

  Then, when the scene selection unit 104 selects a scene from the scene storage unit 103, the operator's motion and target measurement obtained from the motion measurement unit 100 and the target measurement unit 101 are used using the function generated by the learning unit 115. The result (teaching information) is converted into information constituting the scene. Then, the scene selection unit 104 obtains the similarity between each scene recorded in the scene storage unit 103 and the information obtained by conversion, and selects the scene with the highest similarity. Since the information constituting the scene is composed of the same type of information as the scene, the similarity can be easily obtained using a known method or the like.

  In addition, when the operator gives a teaching to the robot and the robot selects an incorrect scene, the operator may set a correct scene corresponding to the operator's teaching through the learning instruction unit 1000. The learning unit 115 uses the measurement results (that is, teaching information) of the motion measurement unit 100 and the object measurement unit 101 at this time and the correct scene correspondence set by the learning instruction unit 1000 as a new learning example sample. In addition to the correspondence storage unit 114, the learning unit 115 performs learning by machine learning. Based on the learning result, the scene selection unit 104 selects a scene again using the measurement result.

  FIG. 5 shows an example in which three types of circles, squares, and triangles are stored as shapes of objects recorded in the scene storage unit 103, and two types of one hand and both hands are stored as one way of holding. Yes. In this case, the scene storage unit 103 stores six types of scenes as combinations of objects and holding methods. On the other hand, when the measurement unit (the motion measurement unit 100, the object measurement unit 101) measures the operation of the operator 300 holding the round object 301 with one hand, the scene selection unit 104 causes the robot 304 to be round with one hand. The scene 302 that holds the object 303 is selected from the scene storage unit 103. In principle, the scene selection unit 104 selects a scene based on the operation (how to hold) of the operator 300 and the type (shape, etc.) of the object 301. At this time, the position and size of the measured object 301 and the object 303 recorded in the scene do not need to match.

  FIG. 6 shows an outline of the operation of the derived scene creation unit 105. The scene 302 records the type (shape, etc.) of the object 303 and the operation method of the robot 304, while the derived scene changes the position and size of the object recorded in the scene. Is generated by the derived scene creation unit 105.

  The derived scene creation unit 105 stores a large number of derived scenes in the derived scene storage unit 106 for each scene in association with the scene. In this embodiment, since the object recorded in the scene is CAD data, the position / size of the CAD data is changed in the derived scene to obtain the derived scene shape data. A large number of changes in position and size may be created at random or by changing the position and size in a predetermined step size.

  When an object whose position / size has been changed is stored in the derived scene storage unit 106, the derived scene creation unit 105 further generates robot operation information that can operate the object, and changes the position / size. The generated robot motion information is recorded in association with the target object. When the derived scene creation unit 105 generates an operation method (robot operation) of an object whose position / size is changed, the operation method of the object stored in the scene storage unit 103 is used as the position / size. The operation method may be generated by correcting the object according to the shape of the object after the change.

  For example, when recording a motion of gripping an object, the derived scene creation unit 105 should store the gripping method recorded in the scene as a derived scene by performing linear interpolation on the object after the position / size change. Find gripping action. Or you may search for the holding method which can hold | grip a target object stably using physical simulation etc. For the trajectory of the robot arm, the trajectory of the arm up to the holding is calculated by a route search method. As a search method, Probabilistic Roadmap, Rapidly-exploring Random Trees, or the like may be used. When there is a difference in shape that cannot be calculated by linear interpolation, the operator obtains the shape and operation method of the object (manually), and uses the scene addition unit 107 to add a new scene to the scene storage unit 103 To do. In addition, when a robot motion is associated with an object included in each derived scene, a plurality of types of motions such as grabbing from the front or grasping from the side may be associated.

  After generating the motion of the robot in the derived scene, the derived scene creating unit 105 stores the created derived scene in the derived scene storage unit 106. Further, when there are two or more robot operation methods, the derived scene creation unit 105 may record information on two or more robot operations in the derived scene. The scene 302 shown in the example of FIG. 6 records the movement of the robot holding a round object with one hand. As derived scenes associated with this, there are three types of object sizes and positions. It is assumed that nine types of derived scenes are recorded by combining three types. Note that FIG. 6 shows only derived scenes in the case of handling a round object with one hand, but derived scenes are also created for the other five types of scenes listed in the example of FIG.

  FIG. 7 shows an outline of the operation of the derived scene selection unit 108. The derived scene selection unit 108 selects a derived scene 501 closest to the real environment from among a plurality of derived scenes associated with the scene 302 selected by the scene selection unit 104. In the derived scene, the shape, position, and size of the object are stored as CAD data in addition to the information of the scene. Since the object measurement unit 101 measures the shape, position, and size of the object 301 in the real environment, the derived scene selection unit 108 determines that the measured object and the object recorded in the derived scene are the most. A derived scene 501 that matches is selected. Here, the matching method is calculated by dividing the space into fine grids, turning on the area where the object exists, and turning off the area where the object does not exist. Can be calculated.

  In the example of FIG. 7, first, the scene 302 is selected based on the measured operation of the operator 300 and the type of the object 301. Up to this point, the processing performed by the scene selection unit 104 is the same as the processing shown in FIG. Subsequently, the derivation scene selection unit 108 compares the position / size of the target object recorded in the derivation scene group associated with the scene 302 with the measured position / size of the target object 301 to thereby derive the derivation scene 501. Select. In FIG. 7, the derived scene in which the small round object 500 is arranged on the back side of the desk is closest, and as a result, the derived scene 501 that matches the operation of the operator 300 and the position and size of the object 301 is selected. Will be.

  When it is attempted to directly associate measurement results related to human movements / objects with a large number of derived scenes by machine learning, there are a large number of correspondence candidates, and it is necessary to prepare a large amount of learning data. On the other hand, the robot system according to the present embodiment is configured to associate the measurement results related to the operation / object of the operator with a small number of scenes as representative examples, and then associate the derived scene with the scene. Yes. As a result, the robot system according to the present embodiment can reduce the number of candidates for association between measurement results related to human movements / objects and scenes, and requires less learning data when executing machine learning. Since the derivative scene can be created by simulation, it can cope with various arrangement situations of articles.

  FIG. 8 shows an example of a user interface provided by the scene addition unit 107 when the operator creates and adds a new scene. The scene addition unit 107 displays an object 1301 being edited and a reference plane 1302 on which the object is arranged on the screen 1300. The reference surface 1302 is a surface on which the object is arranged, and is a surface corresponding to, for example, a desk in the real environment.

  When the operator creates a new scene, first, the shape of the object is selected from the primitive shape group 1303, and the selected shape is displayed on the screen. For example, when creating a scene of a gripping operation of a cylindrical object, a cylinder may be selected from the primitive shape group 1303. Subsequently, the operator uses the editing tool group 1304 to set the size, position, and orientation of the selected object. Further, the operator uses the operation tool group 1305 for editing the operation position of the robot to set how the robot operates the object 1301. For example, in the case of the gripping operation in the present embodiment, the position 1306 to be gripped and 1307 in which the direction in which the force is applied are displayed on the screen are written. The scene addition unit 107 generates information constituting the scene based on the information input by the operator, and stores the generated information in the scene storage unit 103. Since the derived scene adding unit 113 provides the same function as the scene adding unit 107, the description of the derived scene adding unit 113 is omitted.

  Subsequently, each functional block used in the operation process will be described.

  FIG. 9 shows an operation example of the robot operation selection unit 109. The robot operation selection unit 109 is executed when the robot operation execution unit 112 operates the robot. If two or more types of robot motions are associated with the derived scene, the robot motion selection unit 109 selects which robot motion is appropriate from the situation of the entire environment. Conversely, when only one type of robot motion is associated with the derived scene, the robot motion selection unit 109 does not have to be executed.

  Examples of selection criteria for robot motion include selecting a joint angle that allows more stable motion and a shorter trajectory. For example, if stable motion is used as a reference, the robot motion selection unit 109 has a function for evaluating the stability of each joint of the robot in advance, and any robot motion is stable for a plurality of robot motions. Evaluate if there is. As an evaluation function, it is only necessary to calculate how much the position of the hand changes when each joint is moved minutely, and to stabilize the smaller the amount of change. When selecting the length of the trajectory as a selection criterion, the movement distance of the hand may be compared. In FIG. 9, the motion candidate A600 is compared with the motion candidate B601, and the motion candidate B601 having a short trajectory length is selected.

  FIG. 10 shows the operation correction unit 110. When the position / size of the target object in the real environment is different from the position of the target object in the derived scene stored in advance, the motion correcting unit 110 positions / sizes of the target object in the real environment. Modify the robot movement to match. First, the difference between the position / size of the CAD data of the object 700 in the derived scene and the position / size of the object 701 in the real environment is calculated. As a calculation method, a technique such as an ICP algorithm or template matching is used. It is calculated by linearly interpolating the operation 704 recorded in the derived scene, how the robot operation can be performed on the position of the target object 701 in the real environment. Alternatively, the robot's own hand may be measured using the environment measurement unit 102 and controlled using visual feedback so that the shape of the real environment does not collide with the robot.

  FIG. 11 shows an operation method in which the motion correction unit 110 linearly interpolates the motion, changes the position of the fingertip in the present embodiment, and can grasp even when the environment obtained by pre-calculation is different from the actual environment. An example of processing to be created is shown. In this embodiment, the gripping operation is enabled by linearly interpolating the operations 1501, 1502, and 1503 of each finger mechanism of the gripper 1500 according to the gripping position 1505 on the object 1504. By transforming the operation 1501 obtained by pre-calculation in a derived scene or the like, the operation 1501 is transformed. The amount of deformation at this time can be obtained by calculating a bending condition such that the gripping position 1505 of the actual object 1504 is closest to the trajectory of the finger mechanism defined in the original operation 1501. Similarly, the basic operation 1503 is transformed into an operation 1207. This processing is obtained by converting the change of the fingertip position linearly, but if the position of the target fingertip is obtained, the arm motion can be calculated by calculating the posture that satisfies the position obtained by the fingertip. Good. In the case of an arm with redundant degrees of freedom, the posture cannot be determined uniquely, but by setting constraints such as making the joint angle within a certain range, it is not possible to uniquely determine the target posture of all joints. Is possible. This process may be used by the derived scene creation unit 105.

  FIG. 12 shows the collision avoidance unit 111. The collision avoidance unit 111 changes the locus of the robot recorded in the derived scene from the state of the surrounding environment measured by the environment measurement unit 102 to a movement locus that does not collide with surrounding entities. For example, in the case of a gripping operation, the hand position that is the operation target is unchanged, and the route to reach the position and the shape of the arm other than the hand are changed. As a change method, existing route searching means (Probabilistic Roadmap, Rapidly-exploring Random Trees) or the like may be used. In FIG. 12, an obstacle 801 is measured on a predetermined track 800 and is changed to a track 802 that does not collide with the obstacle 801. At this time, the state of the hand 803 is not changed, and the object can be achieved by a change in the trajectory / posture of only the arm. If it is necessary to change the hand position, which is an operation target, how the operation can be achieved can be calculated by physical simulation, and the operation can be executed accordingly. The target here is dependent on the scene, and if it is a gripping operation, a condition that satisfies the condition is calculated on condition that the object is not dropped.

  The above is the description of each functional block used in the operation process. In the above description, the example in which the robot operation is corrected by linearly interpolating the operation method of the object recorded in the derived scene so as to match the object in the real environment during the robot operation has been described. However, in such a method, sufficient accuracy may not be obtained in the actual environment due to external factors (the influence of gravity and friction) existing in the actual environment. Therefore, as another embodiment, the motion correcting unit 110 may correct the robot motion in accordance with the actual environment.

  As a specific method, the motion correcting unit 110 grasps the shape of the target object in the real environment by the target measuring unit 101, and uses the original derived scene information 901 as an initial state to correct the motion according to the shape. As a correction method, it is possible to perform a simulation of which joint is moved while approaching the target posture, receive the feedback of the operation result, and repeat the simulation again to hold the target object stably. The operation is carried out while searching for directions.

  Further, the result information obtained by correcting the robot motion information recorded in the derived scene by the motion correcting unit 110 may be reflected in the derived scene storage unit 106. An example in that case is shown in FIG. As described above, in the operation process, information on the derived scene selected by the derived scene selection unit 108 (in FIG. 13, this is expressed as “derived scene information 901”) is matched with the actual environment by the operation correcting unit 110. To correct. When a plurality of robot motions are associated with the derived scene, one robot motion is selected by the robot motion selection unit 109, but the description of the robot motion selection unit 109 is omitted in FIG. .

  The motion update unit 902 overwrites the robot motion information recorded in the derived scene storage unit 106 using the derived scene information 901 modified by the motion modification unit 110. As a result, the robot motion information stored in the derived scene storage unit 106 is rewritten to the robot motion information considering the real environment, and when the robot performs the same motion in the future, it is more suitable for the state of the real environment. Robot operation becomes possible.

  The difference from the simulation when the derived scene is created is that, in the derived scene creation, how the robot moves is pre-calculated for the joint angle control command, so an error corresponding to the number of commands accumulates. On the other hand, since the operation correction unit 110 always feeds back the result of the operation by the control command at the time of operation, it is possible to grasp the error of the control command and further send a control command to reduce the error. Given.

  Subsequently, Example 2 will be described. The robot system according to the second embodiment has the same functional blocks as the robot system described in the first embodiment, and the hardware configuration is the same as that described in the first embodiment.

  However, the robot system according to the second embodiment has a data sharing function added to the robot system according to the first embodiment. Specifically, the data sharing function is a function that enables a plurality of robot systems to use data such as a scene (or a derived scene) used in the robot system.

  A robot system according to the second embodiment will be described with reference to FIG. Each of the robot systems 1410-1 and 1410-2 in the second embodiment includes the functional blocks described in the first embodiment, and further includes a download unit 1101 and an upload unit 1102. The robot systems (1410-1, 1410-2) are connected to a data center (central storage unit) 1100 via a wide area network 1420 or the like. The data center 1100 has a function of collecting and storing data from a plurality of robot systems (1410-1, 1410-2). For example, the data center 1100 is configured by a large-scale storage device such as a file server. In FIG. 14, only two robot systems are shown, but three or more robot systems may be connected to the data center 1100.

  The download unit 1101 acquires data stored in the data center 1100. On the other hand, the upload unit 1102 transmits the data stored in the robot system (1410-1, 1410-2) to the data center 1100 and stores it in the data center 1100. An example of data stored in the data center 1100 is a scene created in each robot system (1410-1, 1410-2) or a derived scene.

  Each robot system (1410-1, 1410-2) according to the second embodiment is similar to the robot system according to the first embodiment in that the created (or modified) scene and the derived scene are respectively stored in the scene storage unit 103 and the derived scene. Store in the storage unit 106. The robot system according to the second embodiment uses the upload unit 1102 together with it to transmit the created scene and the derived scene to the data center 1100 and store them in the data center 1100. By storing scenes and derived scenes created by a plurality of robot systems in the data center 1100, a large number of scenes and derived scenes are collectively stored in the data center 1100.

  Further, the robot system (for example, the robot system 1410-1) in the second embodiment acquires a scene (or a derived scene) created by another robot system (1410-2) from the data center 1100 by the download unit 1101, and has the own robot system. The data can be stored in the scene storage unit 103 and the derived scene storage unit 106 and used when the robot is operating (particularly in the teaching process). By doing in this way, the robot system in Example 2 can share and use data created (updated) in other robot systems.

  Data other than a scene (or a derived scene) may be shared. For example, information stored in the correspondence storage unit 114 (information on correspondence between teaching information and scenes) may also be shared by a plurality of robot systems via the data center 1100.

  Subsequently, Example 3 will be described. FIG. 15 is a functional configuration diagram of the robot system according to the third embodiment. The robot system according to the third embodiment is mainly different from the robot system according to the first embodiment in having an operation procedure storage unit 120 and an operation instruction unit 130. The operation procedure storage unit 120 is a storage area for storing the derived scene selected by the derived scene selection unit 108. In the following description, a set of functional blocks used in the teaching process (a group of functional blocks included in the dotted frame 11 in the figure) is referred to as a “teaching unit 11”, and functional blocks used in the robot operation process ( A set of functional blocks included in a dotted frame 12 in the drawing is referred to as an “operation unit 12”.

  When the operator performs some operation in the teaching process, as described in the first embodiment, the derived scene selection unit 108 included in the teaching unit 11 selects a derived scene similar to the operator's teaching operation. In the robot system according to the third embodiment, the derived scene selection unit 108 stores the selected derived scene in the operation procedure storage unit 120, and the teaching process ends.

  The trigger for starting the robot operation process is determined by an instruction from a user (operator) of the robot system. The operation instruction unit 130 is a user interface for notifying the robot system that the operator starts an operation process. When the operator uses the operation instruction unit 130 to instruct the start of the operation process, the robot operation selection unit 109 selects the derived scene recorded in the operation procedure storage unit 120.

  Subsequent operations are the same as those described in the first embodiment, and the robot operation execution unit 112 drives the robot based on the robot operation information recorded in the derived scene. Further, at that time, the robot motion information recorded in the derived scene is corrected by the motion correction unit 110 and the collision avoidance unit 111 as in the case described in the first embodiment.

  In the robot system according to the third embodiment, the robot operation (derived scene) is recorded in the operation procedure storage unit 120, and the robot can be operated based on the recorded robot operation later. That is, the robot does not necessarily have to operate in synchronization with the operator's operation, and the robot system according to the third embodiment can cause the robot to reproduce and execute the operation taught by the operator at a timing desired by the operator.

  DESCRIPTION OF SYMBOLS 100 ... Motion measurement part, 101 ... Object measurement part, 102 ... Environment measurement part, 103 ... Scene memory | storage part, 104 ... Scene selection part, 105 ... Derived scene creation part, 106: Derived scene storage unit, 107: Scene addition unit, 108 ... Derived scene selection unit, 109 ... Robot motion selection unit, 110 ... Motion correction unit, 111 ... Collision avoidance unit 112 ... Robot motion execution unit, 113 ... Derived scene addition unit

Claims (14)

A robot system for operating a robot according to an operator's teaching operation,
The robot system includes:
An operation measuring unit for measuring the teaching operation of the operator;
An object measuring unit for measuring the type, position, and size of the object operated by the operator or the robot;
A scene storage unit that stores a plurality of types of scenes that are sets of information on the robot and motion information;
A derived scene storage unit that stores a plurality of derived scenes derived from the scene in association with the scene;
A scene selection unit that acquires one of the scenes from the scene storage unit;
A derivation scene selection unit for obtaining one derivation scene from the derivation scene storage unit;
A robot operation execution unit that operates the robot based on the operation information included in the derived scene;
Have
The information on the target object included in the derived scene is obtained by changing the position and size of the target object included in the scene associated with the derived scene. The motion information is motion information for operating the object included in the derived scene,
The scene selection unit includes a plurality of the scenes stored in the scene storage unit, a teaching operation of the operator measured by the motion measurement unit, and a type of the target measured by the target measurement unit. Select the scene with the highest similarity to the teaching information consisting of
The derivation scene selection unit selects a derivation scene having the highest position / size and similarity of the measured object among the derivation scenes derived from the scene selected by the scene selection unit,
The robot motion execution unit operates the robot based on the motion information included in the derived scene selected by the derived scene selection unit.
A robot system characterized by this. The robot system further includes:
A learning unit that converts the teaching operation of the operator and information on the object into information included in the scene;
The scene selection unit uses the learning unit to convert the teaching information into information included in the scene, thereby obtaining a similarity between the teaching information and the plurality of scenes stored in the scene storage unit. ,
The robot system according to claim 1. The robot system further includes:
An environment measuring unit that measures the surrounding objects of the robot;
When operating the robot, correct the operation of the robot so as not to collide with the existence measured by the environment measurement unit,
The robot system according to claim 1, wherein: The derived scene includes a plurality of types of the operation information,
When operating the robot, from the selected derived scene, one of the operation information is selected according to a predetermined criterion, and the robot is operated based on the selected operation information.
The robot system according to claim 1, wherein: The robot system includes:
When there is a difference between the measured position / size of the object and the position / size of the object stored in the derived scene, the image processing apparatus includes an operation correcting unit that corrects the operation information included in the derived scene.
The robot system according to claim 1, wherein: Using an operation update unit that updates the derived scene stored in the derived scene storage unit using the operation information of the robot modified by the operation modification unit;
The robot system according to claim 5, wherein: In the robot system, the operator
The scene is stored in the scene storage unit, and the derived scene is stored in the derived scene storage unit.
Configured to be appendable,
The robot system according to claim 1, wherein: The learning unit is configured to determine a rule for converting the teaching information into information included in the scene by performing learning using a plurality of learning information including the teaching information and a scene corresponding to the teaching information. Being
The robot system according to claim 2, wherein: The robot system has a communication interface for connecting to a network,
The robot system further includes:
An upload unit for storing the scene and the derived scene in a storage unit on the network via the communication interface;
A download unit for acquiring the scene and the derived scene stored in the storage means on the network by another robot system via the communication interface;
The robot system according to claim 1, comprising: An operation measuring unit for measuring the teaching operation of the operator;
An object measuring unit for measuring the type, position and size of the object operated by the operator or the robot;
A scene storage unit that stores a plurality of types of scenes, which is a set of robot operation information and information on the object;
A derived scene storage unit that stores a plurality of derived scenes derived from the scene in association with the scene;
A scene selection unit that acquires one of the scenes from the scene storage unit;
A derivation scene selection unit for obtaining one derivation scene from the derivation scene storage unit;
A robot operation execution unit that operates the robot based on the operation information included in the derived scene;
A robot control method by a robot system having
The information on the target object included in the derived scene is obtained by changing the position and size of the target object included in the scene associated with the derived scene. The motion information is motion information for operating the object included in the derived scene,
A first step of measuring the teaching motion of the operator by the motion measuring unit and measuring information of the target by the target measuring unit;
The degree of similarity between the plurality of scenes stored in the scene storage unit by the scene selecting unit and the teaching information obtained in the first step and including the teaching operation of the operator and the type of the object A second step of selecting the scene with the highest,
Of the derived scenes derived from the scene selected in the second step by the derived scene selection unit, the derivation having the highest position / size and similarity of the object measured in the first step A third step of selecting a scene;
A fourth step of operating the robot based on the motion information included in the derived scene selected in the third step by the robot motion execution unit;
The robot control method characterized by performing. The robot system further includes:
A learning unit that converts the teaching operation of the operator and information on the object into information included in the scene;
In the second step, the learning information is converted into information included in the scene by the learning unit, thereby obtaining a similarity between the teaching information and the plurality of scenes stored in the scene storage unit. ,
The robot control method according to claim 10, wherein: The learning unit is configured to determine a rule for converting the teaching information into information included in the scene by performing learning using a plurality of learning information including the teaching information and a scene corresponding to the teaching information. Being
The robot control method according to claim 11, wherein: In the fourth step, when there is a difference between the measured position / size of the object and the position / size of the object stored in the derived scene, the operation information included in the derived scene is corrected. Including operation modification process,
The robot control method according to claim 11, wherein: The fourth step is executed at the time when there is an instruction from the operator after the third step is executed.
The robot control method according to claim 11, wherein:

Images