JP2021084218A - Information presentation device, information presentation method, and information presentation program - Google Patents

Abstract

To provide a technique for increasing efficiency of work in setting relative and physical relations of a plurality of objects in a process that a robot device executes a task.SOLUTION: An information presentation device according to one aspect of the present invention displays a first screen which shows a state transition diagram representing a sequence of target states transiting in a process from a start to an end of a task to be executed by a robot device, and receives any selection among a plurality of nodes and correction related to the transition of target states. Further, the information presentation device displays a second screen which shows relative and physical relations of a plurality of objects that are defined for the target states corresponding to the selected node, in accordance with the selection among the plurality of nodes, and receives correction related to the relative and physical relations between the plurality of objects for the displayed target states.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information presentation device, an information presentation method, and an information presentation program.

Various types of robot devices such as manipulators are used in production lines that produce products. The components of the robot device such as the manipulator mechanism, end effector, and workpiece have many variations depending on the task to be performed, etc., and the operation procedure of the robot device corresponding to all of them is manually created. It is difficult to teach the robot device the task of interest. Therefore, conventionally, after determining the types of components such as mechanisms, end effectors, and workpieces, the robot device is manually moved to directly teach the task to be performed while recording the posture in a series of movements to be executed. The method is adopted.

However, in this method, the robot device is instructed to perform the task each time the components such as the mechanism, the end effector, and the work are changed. Therefore, it is too costly to teach the robot device the task to be performed. Therefore, in recent years, research has been conducted on improving the efficiency of a method for making a robot device learn the task to be performed. For example, Non-Patent Document 1 proposes a method of causing a robot device to learn an operation of gripping an object based on image data obtained from a camera by reinforcement learning. According to this method, at least a part of a series of processes for teaching the robot device the operation of gripping the object can be automated. Therefore, it is possible to reduce the cost of teaching the task to the robot device.

Dmitry Kalashnikov, et al. "QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation" arXiv preprint arXiv: 1806.10293, 2018.

The present inventors have found that the conventional control method of the robot device as described above has the following problems. In the conventional control method, time-series control commands given to the robot device for the task to be performed are learned. That is, the learned time series control commands are directly associated with the task. If at least one of the environment and the object to perform the task changes even a little, the content of the task also changes substantially, and it is difficult to perform the task properly with the associated time series control command. Become. Therefore, the robot device may not be able to properly perform the task unless the time-series control command is newly learned for the changed task.

For example, it is assumed that the task taught to the robot device is to carry the object C existing in the posture B at the point A to the point D. In this case, if the object C is accurately arranged at the point A in the posture B, the robot device can appropriately perform the task based on the learning result. However, if the object C is placed slightly offset from the point A, or if the object C is placed at the point A but tilted from the posture B, the position and posture at which the robot device grips the object C changes. The content of the task to be performed changes due to reasons such as doing so. That is, in this case, the task to be performed is to "carry the object C that is deviated or tilted from the point A to the point D" and "to carry the object C that exists in the posture B at the point A". It is different from the original task of "carrying to point D". Then, even if the robot device operates based on the learning result, it may not be able to properly perform this task. Therefore, the conventional control method has a problem that the ability to perform the task to be learned is not versatile. Due to this problem, in order to operate the robot device for general purposes, control commands are learned for each different task, and the cost for teaching the task to the robot device is still high.

In order to solve this problem, the inventors of the present invention control the operation of the robot device based on the relative and physical relationship (relative relationship amount) of a plurality of objects existing in the environment in which the task is performed. Propose a control method to be used. In this control method, a sequence of relative relationships of a plurality of objects in the process in which the robot device performs a task is defined, and a control command to the robot device is associated with the amount of change in the relative relationships. This makes it possible to teach a time-series control command given to the robot device for changing the relative relationship without depending on the content of the task. That is, as long as the task can be performed with the same change in the relative relationship amount, even if the content of the task is slightly changed (in the above example, the position of the object C is displaced or the posture is tilted). ), The robot device can properly perform those tasks. Therefore, according to the control method proposed by the inventors, the versatility of the ability to perform the task to be acquired by the robot device can be increased, and the cost for teaching the task to the robot device can be reduced. .. However, it is assumed that it takes time and effort to set the relative and physical relationships of a plurality of objects in the process of the robot device performing the task.

The present invention, on the one hand, has been made in view of such circumstances, and an object of the present invention is to set relative and physical relationships between a plurality of objects in a process in which a robot device performs a task. It is to provide the technology to improve the efficiency of.

The present invention employs the following configuration in order to solve the above-mentioned problems.

That is, the information presenting device according to one aspect of the present invention is an information acquisition unit that acquires state transition information indicating a series of target states that transition in the process from the start to the end of the task performed by the robot device. Each target state included in the series includes an information acquisition unit and an acquired state transition defined by a relative and physical relationship between a plurality of objects to be operated by the robot device in the task. A first state transition diagram showing a state transition diagram representing a series of the target states based on information, including a plurality of nodes representing each of the target states and an edge representing the transition between the target states. A first display unit that displays a screen on a display device, a first reception unit that accepts selection of one of the plurality of nodes on the first screen, and a modification related to a transition of the target state, and the plurality of units. Depending on the selection of any of the nodes, the display device displays a second screen showing the relative and physical relationship between the plurality of objects defined in the target state corresponding to the selected node. A second display unit is provided, and a second reception unit that receives modifications regarding relative and physical relationships between the plurality of objects in the displayed target state on the second screen.

According to this configuration, in order to set the relative and physical relationships of a plurality of objects in the process in which the robot device performs a task, a first screen showing a state transition diagram and a second screen showing the relative relationships are shown. Two types of screens are prepared. According to the first screen, a series of target states that define the relative and physical relationships to be achieved of each of the plurality of objects involved in the task is appropriately set for the process in which the robot device performs the task. You can check if it is. Further, when there is a defect in the sequence of the target states, the transition between the target states can be corrected. On the other hand, according to the second screen, it is possible to confirm whether or not the relative and physical relationships of the plurality of objects are appropriately defined for each target state. Furthermore, if there is a deficiency in the relationship, it is possible to make corrections regarding the relationship. Therefore, according to the first screen and the second screen provided, it is possible to confirm and set the flow of the entire process of executing the task and each target state. This makes it possible to improve the efficiency of the work of setting the relative and physical relationships of a plurality of objects in the process in which the robot device performs a task.

In the information presenting device according to the one aspect, the information acquisition unit may further acquire environmental information that defines the initial state of the robot device and each object. Further, the information presenting device according to the one aspect includes a simulator unit that simulates the process of the robot device performing the task based on the acquired state transition information and the environmental information, and the result of the simulation. A third display unit for displaying the third screen shown on the display device may be further provided. According to this configuration, it is possible to confirm whether or not the target task can be appropriately performed by the set target state sequence based on the simulation result. By determining each target state while confirming this, the relative and physical relationships of a plurality of objects in the process of the robot device performing the target task can be appropriately set according to the target task. Will be able to.

In the information presenting apparatus according to the one aspect, the third screen is a timeline representing the elapsed time from the start of execution of the task, and a transition in the timeline for achieving a transition between the target states. It may include a block representing a time interval in which the robot apparatus performs the work on the simulation. According to this configuration, it is possible to improve the visibility of the operation of the robot device and the change in the relative and physical relationship between the objects on the third screen. As a result, it is possible to easily confirm whether or not the target task can be appropriately performed by the set target state sequence.

In the information presenting device according to the one aspect, the task may be performed by a plurality of robot devices, and at least one of the plurality of robot devices is used for the transition work for achieving the transition between the target states. The block may be arranged in association with the robotic apparatus used for the corresponding transition work. According to this configuration, the blocks representing the time intervals are arranged in association with the robot device used for the corresponding transition work, thereby enhancing the visibility of the robot device that performs each transition work on the third screen. be able to.

The information presenting device according to the one aspect further includes a third reception unit that receives a change in the robot device used for the corresponding transition work by an operation of changing the robot device associated with the block on the third screen. May be good. According to this configuration, it is possible to select an appropriate robot device for performing each transition work by using the result of the simulation. Thereby, for the task in which a plurality of robot devices cooperate, the work of each robot device can be assigned so that the task can be appropriately performed.

In the information presenting device according to the one aspect, the task may be performed by a plurality of robot devices, and the second reception unit may perform a transition between the target states from the plurality of robot devices on the second screen. May further accept the selection of at least one robotic apparatus used in the transition task to achieve. According to this configuration, it is possible to select an appropriate robot device for achieving each target state while checking each target state. Thereby, for the task in which a plurality of robot devices cooperate, the work of each robot device can be assigned so that the task can be appropriately performed.

In the information presenting device according to the one aspect, the robot device may be a manipulator, and the plurality of objects may include an end effector of the manipulator and parts of a product to be moved by the manipulator. According to this configuration, it is possible to improve the efficiency of the work of setting the relative and physical relationships of a plurality of objects in the process of performing the task by the manipulator.

As another aspect of the information presenting device according to each of the above embodiments, one aspect of the present invention may be an information processing method or a program that realizes each configuration of the above information presenting device. It may be a storage medium that stores such a program and can be read by a computer or the like. A storage medium that can be read by a computer or the like is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action.

For example, the information presentation method according to one aspect of the present invention is a step in which a computer acquires state transition information indicating a series of target states that transition in the process from the start to the end of a task performed by a robot device. , Each target state included in the series is defined by a relative and physical relationship between a plurality of objects that are the targets of the operation of the robot device in the task, and the step and the acquired state transition information. A first screen showing a state transition diagram showing a series of target states, including a plurality of nodes representing each target state and edges representing transitions between target states. On the display device, on the first screen, a step of accepting a selection of any of the plurality of nodes and a modification regarding the transition of the target state, and any of the plurality of nodes. Depending on the selection, a step of displaying on the display device a second screen showing a relative and physical relationship between a plurality of objects defined in the target state corresponding to the selected node, and the second step. It is an information processing method that executes a step of accepting a modification regarding a relative and physical relationship between the plurality of objects in the displayed target state on the screen.

For example, the information presentation program according to one aspect of the present invention is a step of acquiring state transition information indicating a series of target states that transition in the process from the start to the end of a task performed by a robot device to a computer. , Each target state included in the series is defined by a relative and physical relationship between a plurality of objects that are the targets of the operation of the robot device in the task, and the step and the acquired state transition information. A first screen showing a state transition diagram showing a series of target states, including a plurality of nodes representing each target state and edges representing transitions between target states. On the display device, on the first screen, a step of accepting a selection of any of the plurality of nodes and a modification regarding the transition of the target state, and any of the plurality of nodes. Depending on the selection, a step of displaying on the display device a second screen showing a relative and physical relationship between a plurality of objects defined in the target state corresponding to the selected node, and the second step. It is a program for executing a step of accepting a modification regarding a relative and physical relationship between the plurality of objects in the displayed target state on the screen.

According to the present invention, it is possible to improve the efficiency of the work of setting the relative and physical relationships of a plurality of objects in the process in which the robot device performs a task.

FIG. 1 schematically shows an example of a situation in which the present invention is applied. FIG. 2A schematically shows an example of the relative relation amount according to the embodiment. FIG. 2B schematically shows an example of the relative relation amount according to the embodiment. FIG. 3 schematically shows an example of the hardware configuration of the information presentation device according to the embodiment. FIG. 4 schematically shows an example of the software configuration of the information presentation device according to the embodiment. FIG. 5 is a flowchart showing an example of a processing procedure of the information presenting apparatus according to the embodiment. FIG. 6 schematically shows an example of the state transition screen according to the embodiment. FIG. 7A schematically shows an example of the state screen according to the embodiment. FIG. 7B schematically shows an example of the state screen according to the embodiment. FIG. 8 schematically shows an example of a simulation screen according to the embodiment. FIG. 9 schematically shows an example of the sequence screen according to the embodiment.

Hereinafter, embodiments according to one aspect of the present invention (hereinafter, also referred to as “the present embodiment”) will be described with reference to the drawings. However, the embodiments described below are merely examples of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. In carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted. Although the data appearing in the present embodiment is described in natural language, more specifically, it is specified in a pseudo language, a command, a parameter, a machine language, etc. that can be recognized by a computer.

§1 Application example FIG. 1 schematically illustrates an example of application situations of the present invention. The information presenting device 1 according to the present embodiment is a computer configured to present information on the process of executing the task of the robot device 2 and to accept the correction of the target state transitioning in the process of executing the task. The information presenting device 1 according to the present embodiment acquires state transition information 120 indicating a series of target states that transition in the revision from the start to the end of the task executed by the robot device 2. Each target state included in the sequence is defined by a relative and physical relationship (hereinafter, also simply referred to as “relative relationship”) between a plurality of objects to be operated by the robot device 2 in the task. In this embodiment, the task is divided into transition tasks that achieve transitions between target states.

In the example of FIG. 1, two robot devices 2 are used to perform a task, and each robot device 2 is a manipulator. However, the number of the robot devices 2 is not limited to two, and may be one or three or more. Further, the type of the robot device 2 is not particularly limited as long as the object can be operated, and may be appropriately selected according to the embodiment. The robot device 2 may include, for example, an industrial robot such as the manipulator, an automatically movable moving body, or the like. The industrial robot may include, for example, a vertical articulated robot, a SCARA robot, a parallel link robot, a Cartesian robot, a cooperative robot, and the like. Further, the automatically movable moving body may include, for example, a drone, a vehicle configured to be self-driving, a mobile robot, and the like.

In this embodiment, the end effector T and each work (W, V) are examples of objects. The end effector T may be, for example, a gripper, an aspirator, a screwdriver, or the like. Each work (W, V) may be a component of a product such as a connector, a peg, a socket, a hole, a gear, a bolt, or a nut. The product may be, for example, a home appliance, a computer device, an automobile, or the like. As described above, the plurality of objects may include the end effector of the manipulator and the parts of the product to be moved by the manipulator. However, the type of the object does not have to be limited to such an example as long as it can be related to the operation of the robot device 2, and may be appropriately selected depending on, for example, a task to be performed. .. The object may include, for example, an obstacle in addition to the end effector and the work. Obstacles may include, for example, worktables, trays, fixed sensors, and the like.

The type of the task is not particularly limited as long as it is a task to be performed by the robot device 2, and may be appropriately selected according to the embodiment. The task may be, for example, parts transportation, parts fitting, screwdriver, and the like. The task may be, for example, a simple task such as gripping a work or releasing a work.

Relative and physical relationships may relate, for example, to at least one of position, force, dimension (volume), and state. The relative and physical relationship may be defined by the relative relationship quantity. The relative relation quantity indicates the physical quantity of the attribute related to the relative relation between a plurality of objects. The relative relation amount is, for example, relative coordinates (relative position, relative posture), force acting between objects (for example, load, etc.), relative dimensions, state between objects (for example, whether or not they are connected), or It may be composed of a combination of these.

2A and 2B schematically illustrate an example of a relative relation amount composed of relative coordinates. Relative coordinates are the coordinates when the other object is viewed from one object. One of the two objects may be selected as the relative coordinate reference. Relative coordinates may include at least one of relative position and relative orientation. In the three-dimensional space, the relative position may be represented by three axes of front-back, left-right, and up-down, and the posture may be represented by the rotation (roll, pitch, yaw) of each axis. Relative coordinates may be represented by a three-dimensional relative position and a three-dimensional relative posture. However, the number of dimensions of the relative coordinates does not have to be limited to such an example, and may be appropriately determined according to the embodiment. For example, the number of dimensions of the relative position and the relative posture may be reduced as appropriate. Also, for example, the relative attitude may be represented by a quaternion or a rotation matrix.

FIG. 2A schematically illustrates a scene in which the end effector T tries to hold the first work W. As in this scene, while the end effector does not hold the target work, the task state may be defined by the relative relationship between the end effector and the work. In the example of FIG. 2A, the state of the task may be defined by the relative coordinates RC1 between the end effector T and the first work W. The relative coordinates RC1 represent the local coordinate system CW whose origin is the reference point W0 of the first work W as viewed from the local coordinate system CT whose origin is the reference point T0 of the end effector T.

On the other hand, FIG. 2B schematically illustrates a scene in which the first work W held by the end effector T is transported toward the second work V. As in this scene, while the end effector holds the target work, the task state may be defined by the relative relationship between the target work and the target to be transported (for example, other work). .. In the example of FIG. 2B, the task state may be defined by the relative coordinates RC2 between the first work W and the second work V. The relative coordinates RC2 represent the local coordinate system CV whose origin is the reference point V0 of the second work V as seen from the local coordinate system CW whose origin is the reference point W0 of the first work W.

Each reference point (T0, W0, V0) may be set arbitrarily. Further, the method of giving the relative coordinates is not limited to the above example, and may be appropriately determined according to the embodiment. For example, the relative coordinates RC1 are configured to represent the local coordinate system CT whose origin is the reference point T0 of the end effector T as seen from the local coordinate system CW whose origin is the reference point W0 of the first work W. As described above, the relationship between the relative coordinates (RC1, RC2) may be reversed. Similar to the above relative coordinates, relative relational quantities other than the relative coordinates may be appropriately set according to each physical quantity.

Returning to FIG. 1, the information presenting device 1 according to the present embodiment is a state transition diagram representing a series of target states based on the acquired state transition information 120, and a plurality of nodes representing each target state. A state transition screen 30 showing a state transition diagram including 302 and an edge 305 expressing the transition between the target states is displayed on the display device 15. The state transition screen 30 is an example of the first screen. In this embodiment, the state transition diagram further includes a node 301 that represents the initial state of the task. The node 301 is connected to the node 302 representing the first target state (first target state) via the edge 305. The transition relationship represented by the edge 305 may be, for example, any of permutations, branches, joins, and iterations. In the example of FIG. 1, n + 1 target states are set (n is a natural number), the n + 1th target state is the final target state, and the nodes 302 of each target state are connected in a permutation. Each target state and transition relationship may be given as appropriate. The information presenting device 1 according to the present embodiment receives the selection of any one of the plurality of nodes 302 and the modification regarding the transition of the target state on the state transition screen 30. The modification relating to the transition of the target state may be composed of, for example, addition, deletion of the target state, change of the transition source or transition destination, and a combination thereof.

Further, the information presenting device 1 according to the present embodiment is relative to a plurality of objects defined in the target state corresponding to the selected node 302 according to the selection of any one of the plurality of nodes 302. Moreover, the state screen 32 showing the physical relationship is displayed on the display device 15. The state screen 32 is an example of the second screen. The information presenting device 1 according to the present embodiment receives a modification regarding the relative and physical relationship between a plurality of objects in the displayed target state on the state screen 32. Modifications related to relative relations consist of at least one of directly modifying the relative relation quantity and indirectly modifying the relative relation quantity by changing the physical attribute value of each object. Good. The physical attribute values of each object may be related to, for example, position, posture, force (for example, weight, acting force, etc.), dimensions, state (for example, presence / absence of connection), and the like.

As described above, the information presenting device 1 according to the present embodiment displays the state transition screen 30 showing the state transition diagram and the state screen 32 showing the relative relationship in each target state on the display device 15. According to the state transition screen 30, whether or not a series of target states that define the relative relationship to be achieved for each of the plurality of objects involved in the task is appropriately set in the process of the robot device 2 performing the task. Can be confirmed. Furthermore, if there is a defect in the sequence of the target states, it is possible to correct the transition between the target states. On the other hand, according to the state screen 32, it is possible to confirm whether or not the relative relationship of the plurality of objects is appropriately defined for each target state. Furthermore, if the relative relationship is inadequate, the relative relationship can be modified. Therefore, according to the displayed state transition screen 30 and state screen 32, it is possible to confirm and set the flow of the entire process of executing the task and each of the individual target states. Thereby, the efficiency of the work of setting the relative relationship of a plurality of objects in the process of the robot device 2 performing the task can be improved.

§2 Configuration example [Hardware configuration]
FIG. 3 schematically illustrates an example of the hardware configuration of the information presentation device 1 according to the present embodiment. As shown in FIG. 3, the information presenting device 1 according to the present embodiment is a computer in which a control unit 11, a storage unit 12, an external interface 13, an input device 14, a display device 15, and a drive 16 are electrically connected. is there. In FIG. 3, the external interface is described as "external I / F".

The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, which are hardware processors, and is configured to execute information processing based on a program and various data. To. The storage unit 12 is an example of a memory, and is composed of, for example, a hard disk drive, a solid state drive, or the like. In the present embodiment, the storage unit 12 stores various information such as the information presentation program 81, the state transition information 120, the CAD (computer-aided design) information 123, and the environmental information 125.

The information presentation program 81 is a program for causing the information presentation device 1 to execute information processing (FIG. 5) described later regarding the presentation of information related to the process of executing the task of the robot device 2. The information presentation program 81 includes a series of instructions for the information processing. The state transition information 120 indicates a series of target states that transition in the revision from the start to the end of the task executed by the robot device 2. The CAD information 123 includes configuration information indicating a geometrical configuration such as a model (for example, a three-dimensional model) of the robot device 2 and each object (end effector T, first work W, second work V). The environmental information 125 defines the initial state of the robot device 2 and each object. In addition to these, the environment information 125 may further include information indicating an environment for performing a task.

The external interface 13 is, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like, and is an interface for connecting to an external device. The type and number of the external interfaces 13 may be appropriately selected according to the type and number of connected external devices. The information presenting device 1 may be connected to each robot device 2 via the external interface 13.

The input device 14 is, for example, a device for inputting a mouse, a keyboard, or the like. The display device 15 is an example of an output device, for example, a display. The operator can operate the information presenting device 1 via the input device 14 and the display device 15. The display device 15 may be a touch panel display. In this case, the input device 14 may be omitted.

The drive 16 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading various information such as a program stored in the storage medium 91. The storage medium 91 electrically, magnetically, optically, mechanically or chemically acts on the information of the program or the like so that the computer or other device, the machine or the like can read various information of the stored program or the like. It is a medium that accumulates by. At least one of the information presentation program 81, the state transition information 120, the CAD information 123, and the environmental information 125 may be stored in the storage medium 91. The information presenting device 1 may acquire at least one of the information presenting program 81, the state transition information 120, the CAD information 123, and the environmental information 125 from the storage medium 91. Note that FIG. 3 illustrates a disc-type storage medium such as a CD or DVD as an example of the storage medium 91. However, the type of the storage medium 91 is not limited to the disc type, and may be other than the disc type. Examples of storage media other than the disk type include semiconductor memories such as flash memories. The type of the drive 16 may be arbitrarily selected according to the type of the storage medium 91.

Regarding the specific hardware configuration of the information presentation device 1, components can be omitted, replaced, or added as appropriate according to the embodiment. For example, the control unit 11 may include a plurality of hardware processors. The hardware processor may be composed of a microprocessor, an FPGA (field-programmable gate array), a DSP (digital signal processor), or the like. The storage unit 12 may be composed of a RAM and a ROM included in the control unit 11. At least one of the external interface 13, the input device 14, the display device 15, and the drive 16 may be omitted. The information presenting device 1 may further include a communication interface so that data communication via a network can be executed. The information presenting device 1 may further include an output device other than the display device 15. The information presenting device 1 may be composed of a plurality of computers. In this case, the hardware configurations of the computers may or may not match. Further, the information presentation device 1 may be a general-purpose server device, a desktop PC (Personal Computer), a tablet PC, a PLC (programmable logic controller), or the like, in addition to an information processing device designed exclusively for the provided service. ..

[Software configuration]
FIG. 4 schematically illustrates an example of the software configuration of the information presentation device 1 according to the present embodiment. The control unit 11 of the information presentation device 1 expands the information presentation program 81 stored in the storage unit 12 into the RAM. Then, the control unit 11 controls each component by interpreting and executing the instruction included in the information presentation program 81 expanded in the RAM by the CPU. As a result, as shown in FIG. 4, the information presenting device 1 according to the present embodiment has the information acquisition unit 171 and the first display unit 172, the first reception unit 173, the second display unit 174, and the second reception unit 175. It operates as a computer including a simulator unit 176, a third display unit 177, a third reception unit 178, and an operation control unit 179 as software modules. That is, in the present embodiment, each software module of the information presenting device 1 is realized by the control unit 11 (CPU).

The information acquisition unit 171 acquires state transition information 120 indicating a series of target states that transition in the revision from the start to the end of the task executed by the robot device 2. The first display unit 172 is a state transition diagram representing a series of target states based on the acquired state transition information 120, and displays a plurality of nodes 302 representing each target state and transitions between the target states. The state transition screen 30 showing the state transition diagram including the edge 305 to be expressed is displayed on the display device 15. The first reception unit 173 receives the selection of any one of the plurality of nodes 302 and the modification regarding the transition of the target state on the state transition screen 30.

The second display unit 174 determines the relative and physical relationship between the plurality of objects defined in the target state corresponding to the selected node 302 according to the selection of any one of the plurality of nodes 302. The state screen 32 shown is displayed on the display device 15. The second reception unit 175 receives a modification regarding the relative and physical relationship between a plurality of objects in the displayed target state on the state screen 32.

The information acquisition unit 171 may further acquire the environmental information 125 that defines the initial state of the robot device 2 and each object. The simulator unit 176 simulates the process of the robot device 2 performing a task based on the acquired state transition information 120 and environment information 125. The third display unit 177 displays the sequence screen 36 showing the result of the simulation on the display device 15. The sequence screen 36 is an example of the third screen.

The task may be performed by a plurality of robot devices 2, and at least one of the plurality of robot devices 2 may be used for the transition work for achieving the transition between the target states. The third reception unit 178 receives the change of the robot device 2 used for the corresponding transition work on the sequence screen 36. The motion control unit 179 controls the motion of the robot device 2 based on the state transition information 120 and the environment information 125 so as to sequentially achieve each target state from the initial state to the final target state.

Each software module of the information presenting device 1 will be described in detail with reference to an operation example described later. In this embodiment, an example in which each software module of the information presenting device 1 is realized by a general-purpose CPU is described. However, some or all of the above software modules may be implemented by one or more dedicated processors. Further, with respect to the software configuration of the information presentation device 1, software modules may be omitted, replaced, or added as appropriate according to the embodiment.

§3 Operation example Next, an operation example of the information presenting device 1 will be described with reference to FIG. FIG. 5 is a flowchart showing an example of the processing procedure of the information presentation device 1 according to the present embodiment. The processing procedure described below is an example of an information presentation method. However, the processing procedure described below is only an example, and each step may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S101)
In step S101, the control unit 11 operates as the information acquisition unit 171 to acquire the state transition information 120.

The relative relationship in each target state indicated by the state transition information 120 may be determined by any method. A known method such as operator input or direct teaching may be adopted for determining the relative relationship. The state transition information 120 can be created by appropriately determining each target state from the start to the end of executing the task and the relative relationship in each target state. When the state transition screen 30 and the state screen 32, which will be described later, are used to determine the target state and the relative relationship in the target state, the state transition information 120 may be initially given by a template or is empty. It may be information. The template may be appropriately determined to give a series of typical goal states and relative relationships in each goal state, depending on the task.

The state transition information 120 may be generated by the information presenting device 1 or by a computer other than the information presenting device 1. When the information presenting device 1 generates the state transition information 120, the control unit 11 determines each target state and the relative relationship in each target state automatically or by input of the operator via the input device 14. The state transition information 120 can be generated. On the other hand, when the state transition information 120 is generated by another computer, the control unit 11 acquires the state transition information 120 generated by the other computer via, for example, a network or a storage medium 91.

Further, in the present embodiment, in step S101, the control unit 11 may further acquire CAD information 123 and environmental information 125. CAD information 123 may be generated by known software. The initial state of the robot device 2 and each object indicated by the environmental information 125 is defined by physical attributes such as a position and a posture in which the robot device 2 is arranged. This initial state may be determined by any method as in the case of each of the target states described above. Similar to the state transition information 120, the CAD information 123 and the environment information 125 may be generated by the information presenting device 1 or by a computer other than the information presenting device 1. Further, when the attribute values of the robot device 2 and each object in the initial state are determined by using the state screen 32 or the simulation screen 34 described later, the environment information 125 may be initially given by the template. Alternatively, it may be empty information. When various information is acquired, the control unit 11 proceeds to the next step S102.

(Step S102 to Step S105)
In step S102, the control unit 11 displays the corresponding screen on the display device 15 in response to the operation of the input device 14 by the operator. In the present embodiment, the screen displayed on the display device 15 in step S102 is any one of the state transition screen 30, the state screen 32, the simulation screen 34, and the sequence screen 36 of FIGS. 6 to 9, which will be described later. In step S103, the control unit 11 receives an operator's operation on the screen displayed on the display device 15. In the present embodiment, in step S103, the operator edits the state of the robot device 2 and each object in the process of executing the task by operating the display screen.

In step S104, the control unit 11 determines whether or not the operation received on the displayed screen involves a screen transition. When the received operation involves a screen transition, the control unit 11 returns the process to step S102 and displays the corresponding screen on the display device 15. On the other hand, if this is not the case, the control unit 11 proceeds to the next step S105. In step S105, the control unit 11 determines whether or not the received operation is related to the end of the editing work. When the received operation is related to the end of the editing work, the control unit 11 ends the processing procedure according to this operation example. On the other hand, if this is not the case, the control unit 11 returns the process to step S103 and further accepts the operator's operation on the screen displayed on the display device 15.

FIG. 6 schematically illustrates an example of the state transition screen 30 according to the present embodiment. In step S102, the control unit 11 operates as the first display unit 172, and based on the acquired state transition information 120, displays the state transition screen 30 illustrated in FIG. 6 as the first display screen on the display device 15. May be displayed. The state transition screen 30 shows a state transition diagram showing a series of target states. The state transition diagram includes a node 301 representing the initial state of the task, a plurality of nodes 302 representing each target state, and an edge 305 representing the transition between the target states. A thumbnail representing the robot device 2 and each object in each state or a description of each state may be drawn on each node (301, 302). Node 301 may be omitted.

After displaying the state transition screen 30, in step S103, the control unit 11 operates as the first reception unit 173, and on the state transition screen 30, the selection of any of the plurality of nodes 302 and the target state are selected. Accepts transition corrections. In the present embodiment, the state transition screen 30 is provided with six buttons 313-1-16. The control unit 11 receives various operations including selection of the node 302 and modification of the transition of the target state by each of the buttons 31 to 316.

Buttons 311 and 312 are used to accept modifications relating to the transition of the target state. After accepting the designation of the node 302 in the state transition diagram, the control unit 11 accepts the deletion of the designated node 302 and the corresponding target state in response to the operation of the button 311. Further, the control unit 11 accepts the addition of a new target state and the corresponding node 302 in response to the operation of the button 312. However, the method of accepting the deletion and addition of the target state does not have to be limited to such an example. As an example of another method, either deletion or addition of the target state may be directly accepted on the state transition diagram without going through the button (311, 312).

Further, the control unit 11 accepts the deletion and addition of the edge 305 representing the transition between the target states on the state transition diagram as a modification regarding the transition of the target states. The deletion and addition of the edge 305 may be accepted by any operation. As an example, the control unit 11 may delete the designated edge 305 according to the operation of designating the edge 305. Further, the control unit 11 may add an edge 305 between the designated target states according to the operation of designating the node 302 of the transition source and transition destination target states. The deletion and addition of the edge 305 corresponds to the change of the transition source or transition destination of the target state. In addition, the control unit 11 may accept a change of the node 302 connecting the edge 305.

The button 313 is used to instruct the robot device 2 (actual machine) to execute an operation based on the sequence of target states shown in the state transition diagram. The control unit 11 operates as the operation control unit 179 in response to the operation of the button 313, and achieves each target state in order from the initial state to the final target state based on the state transition information 120 and the environment information 125. Control the operation of the robot device 2. The control command given to the robot device 2 may be composed of, for example, a target control amount, an operation amount, and the like. The control command may be appropriately determined to achieve the transition between each target state. Any method may be adopted for determining the control command, and for example, a known method such as operator input or direct teaching may be adopted. The control unit 11 may directly control the operation of the robot device 2. Alternatively, the control unit 11 may indirectly control the operation of the robot device 2 by giving a control command to a control device such as a PLC.

Button 314 is used to accept the execution of the simulation. The button 315 is used to receive a transition to the sequence screen 36 showing the result of the simulation. Button 315 becomes operable after executing the simulation.

Button 316 is used to accept the selection of any of the plurality of nodes 302. After accepting the designation of the node 302 in the state transition diagram, the control unit 11 determines that the designated node 302 has been selected in response to the operation of the button 316. That is, in the present embodiment, selecting one of the plurality of nodes 302 corresponds to designating the node 302 in the state transition diagram and operating the button 316. However, the method of accepting the selection of the node 302 does not have to be limited to such an example. As an example of another method, the selection of the node 302 may be accepted directly in the state transition diagram without going through the button 316.

7A and 7B schematically illustrate an example of the state screen 32 according to the present embodiment. In step S104, the control unit 11 determines that the transition operation to the state screen 32 has been accepted in response to the operation of the button 316 on the state transition screen 30 (that is, the selection of any of the plurality of nodes 302). .. Then, in step S102, the control unit 11 operates as the second display unit 174, and displays the state screen 32 exemplified in FIGS. 7A and 7B on the display device 15.

The state screen 32 shows the relative relationship between a plurality of objects defined in the target state corresponding to the selected node 302. The state transition information 120 includes information that can reproduce the relative relation amount in each target state. The information that can reproduce the relative relation amount may be composed of, for example, the relative relation amount itself, or may be composed of the physical attribute value of each object. By referring to the state transition information 120, the control unit 11 identifies the relative relationship defined in the target state corresponding to the selected node 302. The method of displaying the relative relationship may be arbitrary. In the present embodiment, the control unit 11 uses the CAD information 123 on the state screen 32 to draw a model of each object so as to represent the target state corresponding to the selected node 302. Display the relative relationship between multiple objects specified in the target state.

In the example of FIG. 7A, it is assumed that the end effector T corrects the relative relation amount of the operation for holding the first work W, and the drawing of the model of the second work V irrelevant to this operation is omitted. ing. In this way, the drawing of the object irrelevant to the relative quantity to be modified may be omitted. On the other hand, in the example of FIG. 7B, it is assumed that the first work W held by the end effector T is transported toward the second work V, and the end effector T, the first work W, and the second work V Each model is drawn on the state screen 32. The object for which the relative relation amount is to be modified may be appropriately selected.

After displaying the state screen 32, in step S103, the control unit 11 operates as the second reception unit 175, and on the state screen 32, receives a modification regarding the relative relationship between the plurality of objects in the displayed target state. .. The method of accepting the modification may be appropriately determined according to the embodiment. In the present embodiment, the state screen 32 is provided with a box 321 according to a modifiable attribute relating to the relative relationship.

Each box 321 receives an input via the operator's input device 14 to display the position (X, Y, Z) and posture (R, P,) of each object (end effector T, first work W, second work V). It is configured to accept changes in Y). In the example of FIG. 7A, two boxes 321 that accept changes in the positions and postures of the end effector T and the first work W are displayed. In the example of FIG. 7B, two boxes 321 that accept changes in the positions and postures of the first work W and the second work V are displayed. By changing at least one of the values of the position and orientation of the corresponding object in at least one of the boxes 321, the relative coordinates with respect to the object can be modified. Further, the control unit 11 may accept a change in at least one of the positions and postures of each object by operating the model of each object. In this case, the control unit 11 may modify the position and posture values of each object displayed in each box 321 so as to match the current values of the model according to the operation of the model of each object.

However, the attributes that can be changed by each box 321 need not be limited to such an example. Each box 321 may be configured to accept changes in physical attribute values other than position and orientation, such as forces, dimensions, states, etc., instead of or with position and orientation. Further, the control unit 11 has at least one of the position and orientation of the reference point in the local coordinate system of each object as an attribute of each object on the model drawn by each box 321 or on the state screen 32. You may accept the correction of. In the example of FIGS. 7A and 7B, the control unit 11 is a reference point (T0, W0, V0) of each local coordinate system (CT, CW, CV) of the end effector T, the first work W, and the second work V. At least one modification of position and orientation may be accepted. Further, the object to be changed by each box 321 is not limited to the physical attribute value of each object. As another example, each box 321 may be configured to directly accept a change in the relative quantity.

As illustrated in FIG. 7B, when two or more objects are integrated to form one aggregate, such as when the end effector T holds the first work W, this is displayed on the state screen 32. Modifications to the relative relationship between the aggregate and other objects may be accepted. In this case, the attribute of one object included in the aggregate may be treated as the attribute of the aggregate. That is, on the state screen 32, the modification of the relative relation amount between one object representing the aggregate and the other object may be accepted.

One object representing the aggregate may be appropriately selected. For example, in the example of FIG. 7B, since it is assumed that the first work W is transported toward the second work V, this operation is mainly performed in the aggregate (end effector T and the first work W). The target object is the first work W. Therefore, in the state screen 32 of FIG. 7B, the first work W represents the aggregate. In this way, the main object for the operation of the relative relation amount corrected on the state screen 32 may be selected as the object representing the aggregate. Other objects may also be aggregates.

The amount of relative relation between each object in the aggregate may be set as appropriate. For example, the control unit 11 may further accept the setting or modification of the relative relationship amount between each object in the aggregate on the state screen 32 for modifying the relative relationship amount with another object. Alternatively, the control unit 11 may accept the setting or modification of the relative relation amount between each object in the aggregate on another state screen 32. When the relative relation amount between each object in the aggregate is set, the attribute of one object representing the aggregate is changed by the modification of the relative relation quantity, and the remaining objects are changed. The attribute of may be changed to satisfy the relative relation amount between each object set in the aggregate. For example, in the example of FIG. 7B, the position or orientation of the end effector T is changed so that the end effector T satisfies the state of holding the first work W in response to the change of the position or orientation of the first work W. May be changed. The control unit 11 may appropriately perform an operation that reflects such a change in the attributes of some objects to other objects according to the operation of the operator.

Further, in the present embodiment, the state screen 32 is provided with four buttons (323 to 325, 331). Two of these buttons (323, 324) are used to limit the possible range of relative quantities. Button 323 is used to specify the reachable range of the object. The method of specifying the contactable range may be arbitrary. For example, the contactable range of each object may be specified by specifying the area on the model of each object drawn on the state screen 32. In the example of FIG. 7A, the designation of the contactable range is performed to determine the range in which the end effector T is allowed to contact in the first work W when the end effector T holds the first work W. It's okay. Further, in the example of FIG. 7B, the contactable range may be specified to determine the range in which the arrangement of the first work W is permitted in the second work V. The method of designating the contactable range is not limited to such a method of directly designating. The contactable range may be indirectly specified, for example, by specifying the non-contactable range.

On the other hand, the button 324 is used to specify an allowable range of the relative relation amount. The method of specifying the allowable range of the relative relation quantity may be arbitrary. For example, the control unit 11 may display the slide bar according to the relative relation amount for which the designation is accepted according to the operation of the button 324, and may accept the designation of the maximum value and the minimum value on the slide bar. As a result, the control unit 11 may accept the designation of the allowable range of the relative relation amount. In the example of FIG. 7A, the specification of this allowable range is performed to determine the range of the relative relationship amount between the end effector T and the first work W that is allowed when the end effector T holds the first work W. You can. Further, in the example of FIG. 7B, the designation of this allowable range determines the range of the relative relationship amount between the first work W and the second work V that are allowed when the first work W is arranged on the second work V. May be done to.

In addition, at an arbitrary timing, such as when the relative relationship amount between any of the objects is changed by the operation of each of the boxes 321 above, the control unit 11 sets each range specified by each button (323, 324). It may be determined whether or not the condition is satisfied. Then, when any of the ranges is not satisfied, the control unit 11 may display a warning for notifying the fact on the state screen 32.

For example, in the scene of FIG. 7A, after the contactable range on the first work W is specified, the attribute of the end effector T is such that the end effector T holds a place other than the contactable range of the first work W. Is changed. In this case, the control unit 11 may display a warning on the state screen 32 for notifying that the end effector T holds a portion where contact of the first work W is not permitted.

Further, for example, in the scene of FIG. 7A, after the range of relative coordinates between the end effector T and the first work W allowed when the end effector T holds the first work W is specified, this range is not satisfied. It is assumed that the attribute of the end effector T is changed as described above. In this case, the control unit 11 may display a warning on the state screen 32 for notifying that the end effector T holds the first work W at unacceptable relative coordinates.

Further, when the relative relationship amount between the objects does not satisfy the condition of each specified range, the control unit 11 may appropriately modify the relative relationship amount between the objects. For example, the control unit 11 may change the relative relationship amount between the objects at random or according to a predetermined rule. A predetermined rule for changing the relative relation amount may be given by the operator's designation, the setting in the program, or the like. In this case, the control unit 11 calculates a plurality of candidates that satisfy the conditions of the contactable range and the allowable range, and among the calculated plurality of candidates, the amount of change is the smallest (that is, the relative relationship amount specified by the operator). The closest) candidate may be presented on the status screen 32. Then, the control unit 11 may accept the selection of whether or not to adopt this candidate on the state screen 32.

Further, the control unit 11 may determine at an arbitrary timing whether or not physical interference occurs between the objects due to the relative relationship amount specified on the state screen 32. In the present embodiment, the control unit 11 is in a state where the model of each object cannot be physically taken based on the CAD information 123 (for example, the model of one object is buried in the model of another object). It may be determined whether or not it is. For example, in the scene of FIG. 7A, the control unit 11 may determine whether or not interference occurs between the end effector T and the first work W. Further, in the scene of FIG. 7B, the control unit 11 determines whether or not interference occurs between the end effector T and the second work V, and at least one of the first work W and the second work V. You may.

Further, the control unit 11 considers the transition of the target state specified in the state transition diagram, and specifies the relative relationship amount in the target state of the target displayed on the state screen 32. It may be determined whether or not physical interference occurs with each object for which the relative relation amount is specified in the target state to be transitioned later (for example, the target state to be transitioned to next). For example, it is assumed that the target state of FIG. 7B is the target state that next transitions to the target state of FIG. 7A. In this case, in the state transition information 120, the relative relationship amount between the end effector T and the second work V is specified via the first work W. When the state screen 32 of FIG. 7A is displayed, the control unit 11 determines whether or not interference occurs between the end effector T and the second work V based on the state transition information 120 and the CAD information 123. You may judge. As a result, the relative relationship amount in the target state of the target can be adjusted in consideration of the target state to be transitioned later.

When it is determined that interference occurs in each scene, the control unit 11 may indicate the object on which the interference occurs and display a warning for notifying the occurrence of the interference on the state screen 32. Further, similarly to the above, the control unit 11 may appropriately modify the amount of relative relationship between the objects in which interference occurs. In this case, the control unit 11 may calculate a plurality of candidates having a relative relationship amount that does not cause interference, and may present the candidate having the smallest change amount among the calculated plurality of candidates on the state screen 32. Then, the control unit 11 may accept the selection of whether or not to adopt this candidate on the state screen 32.

Button 325 is used to select the robot device 2. The task may be performed by a plurality of robot devices 2. In this case, the control unit 11 may further accept the selection of at least one robot device 2 used for the transition work for achieving the transition between the target states from the plurality of robot devices 2 on the state screen 32. The button 325 is used to select at least one robot device 2 to be used for this transition operation.

Specifically, the control unit 11 can accept the selection of at least one robot device 2 used for the transition work to achieve the displayed target state in response to the operation of the button 325. The selection method of the robot device 2 to be used may be appropriately determined according to the embodiment. As an example, the control unit 11 may display a list of robot devices 2 that can be used for the transition work to achieve the displayed target state in response to the operation of the button 325. Then, the control unit 11 may accept the selection of at least one robot device 2 according to the displayed list.

When the robot device 2 to be used for the transition work has not been determined, the robot device 2 to be used for the transition work can be designated by this operation. Further, when the robot device 2 used for the transition work is determined in advance, the robot device 2 used for the transition work can be modified (changed) by this operation. On the state screen 32, the operator can select an appropriate robot device 2 for achieving each target state while checking each target state. As a result, it is possible to assign the work of each robot device 2 so that the task can be appropriately performed for the task in which the plurality of robot devices 2 cooperate. In response to the modification of the robot device 2 to be used, the control unit 11 changes the model drawn on the state screen 32 from the object related to the robot device 2 before the change to the object related to the robot device 2 after the change. May be good.

Button 331 is used to return to the state transition screen 30. In step S104, the control unit 11 determines that the transition operation to the state transition screen 30 has been accepted in response to the operation of the button 331. Then, in step S102, the control unit 11 displays the state transition screen 30 illustrated in FIG. 6 again on the display device 15, and executes the processes after step S103.

The node 301 expressing the initial state may be treated in the same manner as the node 302 expressing each target state. That is, the control unit 11 may accept the selection of the node 301 on the state transition screen 30. Then, depending on the selection of the node 301, the control unit 11 may display the state screen 32 showing the relative relationship between the plurality of objects in the initial state on the display device 15. Then, on the state screen 32, a correction regarding the relative relationship between a plurality of objects in the initial state may be accepted. Further, in addition to the buttons (323 to 325, 331), the state screen 32 may be provided with a button for selecting an object to be modified for the relative relation amount. For example, in the example of FIG. 7A, a model of the second work V may appear on the state screen 32 in response to the operation of this button, and the modification of the relative relationship amount with the second work V may be accepted.

FIG. 8 schematically illustrates an example of the simulation screen 34 according to the present embodiment. In step S104, the control unit 11 determines that the transition operation to the simulation screen 34 has been accepted in response to the operation of the button 314 on the state transition screen 30. Then, in step S102, the control unit 11 displays the simulation screen 34 illustrated in FIG. 8 on the display device 15. The configuration of the simulation screen 34 may be arbitrary. In the present embodiment, the control unit 11 draws a model of the robot device 2 and each object in the initial state on the simulation screen 34 based on the CAD information 123 and the environment information 125. Similar to the state transition information 120, the environment information 125 includes information that can reproduce the initial state such as the attribute value of the robot device 2 and the attribute value of each object.

The simulation screen 34 is provided with three buttons (341, 343, 351). The button 341 is used to set the attributes of the robot device 2 and each object in the initial state. The attributes of the robot device 2 may be related to, for example, the type of robot, the initial angle of joints, the installation position, the position where the end effector is attached, and the like. The attributes of each object may be related to, for example, an installation position, an installation angle (posture), and the like. The method of setting the initial state may be appropriately determined according to the embodiment. As an example, the control unit 11 displays a box corresponding to the changeable attribute according to the operation of the button 341, and the robot device 2 in the initial state is input to the displayed box. And the specification of the attribute value of each object may be accepted. Alternatively, the control unit 11 may accept the designation of the attribute values of the robot device 2 and each object in the initial state according to the operation of the model of the robot device 2 and each object on the simulation screen 34. If the initial state has not been determined in advance, the attributes of the robot device 2 and each object in the initial state can be specified by this operation. On the other hand, when the initial state is determined in advance, the attributes of the robot device 2 and each object in the initial state can be modified by this operation.

Button 343 is used to direct the execution of the simulation. The control unit 11 operates as the simulator unit 176 in response to the operation of the button 343, and simulates the process in which the robot device 2 executes a task based on the state transition information 120 and the environment information 125. The control unit 11 simulates the operation of the robot device 2 so as to sequentially achieve each target state from the initial state to the final target state, similarly to the operation of the operation control unit 179. The simulation may be performed in any way. Known software may be used for the simulation. The simulation may also be executed when controlling the operation of the actual machine by operating the button 313.

The button 351 is used to return to the state transition screen 30 in the same manner as the button 331. In step S104, the control unit 11 determines that the transition operation to the state transition screen 30 has been accepted in response to the operation of the button 351. Then, in step S102, the control unit 11 displays the state transition screen 30 illustrated in FIG. 6 again on the display device 15, and executes the processes after step S103. At this time, the control unit 11 activates the button 315.

FIG. 9 schematically illustrates an example of the sequence screen 36 according to the present embodiment. In step S104, the control unit 11 determines that the transition operation to the sequence screen 36 has been accepted in response to the operation of the button 315 on the state transition screen 30. Then, in step S102, the control unit 11 operates as the third display unit 177, and displays the sequence screen 36 illustrated in FIG. 9 on the display device 15. The timing of displaying the sequence screen 36 does not have to be limited to such an example. As another example, the control unit 11 may automatically display the sequence screen 36 on the display device 15 when the execution of the simulation by the operation of the button 343 is completed.

The sequence screen 36 according to the present embodiment includes a timeline 361 and each block (3631 to 3634, 3643-1643). The timeline 361 represents the elapsed time since the start of task execution. Each block (3631 to 3634, 3643-1643) represents a time interval in which the robot device 2 executes a transition operation for achieving a transition between target states on a simulation in the timeline 361. As described above, the task may be performed by a plurality of robot devices 2, and at least one of the plurality of robot devices 2 may be used for the transition work. Correspondingly, each block (3631 to 3634, 3643-1643) may be arranged in association with the robot device 2 used for the corresponding transition work. As a result, the visibility of the robot device 2 that performs each transition work can be improved.

In the example of FIG. 9, it is assumed that two robot devices 2 are used to perform the task. For convenience of explanation, they are referred to as a first robot and a second robot, respectively. In addition, as shown in FIG. 6, four target states are set in the task, the second robot is in charge of the transition work from the second target state to the third target state, and the remaining transition work is performed. It is assumed that the first robot is in charge. As shown in FIG. 9, for convenience of explanation, the operations of each transition operation are referred to as "first operation", "second operation", and the like in order.

Each block 3631 to 3634 is associated with the first robot and arranged in the upper stage. On the other hand, each block 3641 to 643 is associated with the second robot and is arranged in the lower stage. The blocks (3631, 3632, 3634) indicate that the first robot is executing each operation related to each transition work. Correspondingly, the blocks (3641 and 3443) indicate that the second robot is waiting while the first robot is executing each operation. Similarly, block 3642 indicates that the second robot is executing the third operation related to the transition work from the second target state to the third target state. Correspondingly, the block 3633 indicates that the first robot is waiting while the second robot is executing the third operation. In the example of FIG. 9, one robot device 2 is in charge of each operation. However, the number of robot devices 2 used for each operation is not limited to such an example, and may be two or more.

After displaying the sequence screen 36, in step S103, the control unit 11 operates as the third reception unit 178, and on the sequence screen 36, the robot device 2 that associates each block (3631 to 3634, 3643-1643) is provided. By the change operation, the change of the robot device 2 used for the corresponding transition work may be accepted. As an example, it is assumed that the control unit 11 has received an operation of moving the block 3642 from the lower stage to the upper stage or an operation of moving the block 3633 from the upper stage to the lower stage from the operator via the input device 14. In this case, the control unit 11 changes the robot device 2 used for the third operation from the second robot to the first robot. Thereby, the robot device 2 used for the corresponding transition work can be modified. On the sequence screen 36, the operator can select an appropriate robot device 2 for performing each transition work while checking the result of the simulation. As a result, it is possible to assign the work of each robot device 2 so that the task can be appropriately performed for the task in which the plurality of robot devices 2 cooperate.

An indicator 362 is provided on the timeline 361 according to the present embodiment. Further, the sequence screen 36 according to the present embodiment is provided with two display areas (367, 368). The operator can specify an arbitrary time on the timeline 361 by operating the indicator 362. In the display area 367, the states of the robot device 2 and each object reproduced by the simulation are displayed at the time specified by the indicator 362. On the other hand, in the display area 368, the state of the relative relationship of each object reproduced by the simulation is displayed at the time specified by the indicator 362.

Further, the sequence screen 36 according to the present embodiment is provided with a button 371. The button 371 is used to return to the state transition screen 30 in the same manner as the button 331 and the like. In step S104, the control unit 11 determines that the transition operation to the state transition screen 30 has been accepted in response to the operation of the button 371. Then, in step S102, the control unit 11 displays the state transition screen 30 illustrated in FIG. 6 again on the display device 15, and executes the processes after step S103.

The operator executes the task by operating each screen (30, 32, 34, 36) via the input device 14 while checking the contents displayed on each screen (30, 32, 34, 36). It is possible to edit the state of the robot device 2 and each object in the process of performing. The correction regarding the initial state received in this editing is reflected in the environment information 125 at an arbitrary timing after the correction is received. Similarly, the modification regarding the transition of the target state, the modification regarding the relative relationship between the plurality of objects in each target state, and the modification of the robot device 2 to be used are the state transition information 120 at an arbitrary timing after receiving the modification. It is reflected in. Reflection of the correction may be configured by generating new data or updating existing data. The state transition information 120 and the environment information 125 may be structured by a language such as XML (Extensible Markup Language). In this case, the control unit 11 can acquire the state transition diagram and the information of each state by parsing the structured state transition information 120 and the environment information 125.

The configuration of each screen (30, 32, 34, 36) does not have to be limited to the above example. With respect to the configuration of each screen (30, 32, 34, 36), components may be omitted, replaced, and added as appropriate according to the embodiment. For example, each screen (30, 32, 34, 36) may be further provided with a button (not shown) for ending the editing work. In response to the operation of this button, the control unit 11 may determine in step S105 that the editing work is completed, and may end the processing procedure according to this operation example.

[Features]
As described above, in the present embodiment, two types of screens, a state transition screen 30 showing the state transition diagram and a state screen 32 showing the relative relationship in each target state, are displayed on the display device 15 by the step S102. According to the state transition screen 30, whether or not a series of target states that define the relative relationship to be achieved for each of the plurality of objects involved in the task is appropriately set in the process of the robot device 2 performing the task. Can be confirmed. Further, if there is a defect in the sequence of the target states, the transition between the target states can be corrected in step S103. On the other hand, according to the state screen 32, it is possible to confirm whether or not the relative relationship of the plurality of objects is appropriately defined for each target state. Further, when the relative relationship is inadequate, the relative relationship can be corrected in step S103. Therefore, according to the displayed state transition screen 30 and state screen 32, it is possible to confirm and set the flow of the entire process of executing the task and each of the individual target states. Thereby, the efficiency of the work of setting the relative relationship of a plurality of objects in the process of the robot device 2 performing the task can be improved.

Further, in the present embodiment, the sequence screen 36 showing the result of the simulation is displayed on the display device 15 by the step S102. According to the sequence screen 36, it is possible to confirm whether or not the target task can be appropriately performed by the set target state sequence based on the simulation result. In addition, by determining each target state on the state transition screen 30 and the state screen 32 based on this confirmation result, the robot device 2 is relative to a plurality of objects in the process of performing the target task. Relationships can be set appropriately according to the target task. Further, in the present embodiment, on the sequence screen 36, the result of the simulation is represented by the timeline 361 and each block (3631 to 3634, 3643-1643). As a result, the visibility of the operation of the robot device 2 and the change in the relative relationship between the objects can be improved, and it is confirmed whether or not the target task can be appropriately performed by the set target state sequence. It can be made easier.

§4 Modifications Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in all respects. Needless to say, various improvements or modifications can be made without departing from the scope of the present invention. For example, the following changes can be made. In the following, the same reference numerals will be used for the same components as those in the above embodiment, and the same points as in the above embodiment will be omitted as appropriate. The following modifications can be combined as appropriate.

<4.1>
In the above embodiment, the control unit 11 displays each screen (30, 32, 34, 36) on the display device 15 included in the information presentation device 1. However, the display device for displaying each screen (30, 32, 34, 36) does not have to be limited to such an example. The control unit 11 may display each screen (30, 32, 34, 36) on a display device provided in another computer.

<4.2>
In the above embodiment, each operation of each screen (30, 32, 34, 36) may be omitted as appropriate. The method of each operation may be appropriately changed according to the embodiment. For example, the operation of changing the robot device 2 to be used on the state screen 32 may be omitted. In this case, the button 325 may be omitted from the status screen 32. Further, for example, on the state screen 32, at least one of the two buttons (323, 324) may be omitted. Further, for example, the operation of changing the robot device 2 to be used on the sequence screen 36 may be omitted. In this case, the third reception unit 178 may be omitted from the software configuration of the information presentation device 1.

In the sequence screen 36, the arrangement of each block (3631 to 3634, 3643-1643) does not have to be associated with the robot device 2 used for the corresponding transition work. In this case, the correspondence between each block (3631 to 3634, 3643-1643) and the robot device 2 may be expressed by another method such as the color of the block. If the result of the simulation can be confirmed, the sequence screen 36 may show the result of the simulation by a method other than the method used in the timeline 361 and each (3631 to 3634, 364-13643).

In the above embodiment, the execution of the simulation may be omitted. In this case, the display of the simulation screen 34 and the sequence screen 36 may be omitted. In the state transition screen 30, each button (314, 315) may be omitted. The simulator unit 176 and the third display unit 177 may be omitted from the software configuration of the information presentation device 1. Further, in the above embodiment, the execution of the operation of the actual machine may be omitted. In this case, the button 313 may be omitted on the state transition screen 30. The operation control unit 179 may be omitted from the software configuration of the information presentation device 1.

1 ... Information presentation device,
11 ... Control unit, 12 ... Storage unit, 13 ... External interface,
14 ... Input device, 15 ... Display device, 16 ... Drive,
91 ... Storage medium, 81 ... Information presentation program,
120 ... state transition information, 123 ... CAD information,
125 ... Environmental information,
171 ... Information acquisition department,
172 ... 1st display, 173 ... 1st reception,
174 ... 2nd display unit, 175 ... 2nd display unit,
176 ... Simulator department,
177 ... 3rd display unit, 178 ... 3rd reception unit,
179 ... Motion control unit,
2 ... Robot device,
T ... End effector (object),
T0 ... reference point, CT ... local coordinate system,
W ... 1st work (object),
W0 ... reference point, CW ... local coordinate system,
V ... 2nd work (object),
V0 ... reference point, CV ... local coordinate system,
RC1 and RC2 ... Relative coordinates,
30 ... State transition screen,
301/302 ... node, 305 ... edge,
311-16 ... Button,
32 ... Status screen (second screen),
321 ... Box,
325/331 ... button,
34 ... Simulation screen,
341, 343, 351 ... Button,
36 ... Sequence screen (3rd screen),
361 ... Timeline, 362 ... Indicator,
3631-3634 ... Block,
3641-643 ... Block,
367/368 ... Display area,
371 ... button

Claims (9)

An information acquisition unit that acquires state transition information indicating a series of target states that transition in the process from the start to the end of a task executed by a robot device, and each target state included in the series is the target state in the task. The information acquisition unit, which is defined by the relative and physical relationships between multiple objects that are the objects of operation of the robot device,
Based on the acquired state transition information, it is a state transition diagram representing a series of the target states, and includes a plurality of nodes representing each of the target states and an edge representing the transition between the target states. A first display unit that displays the first screen showing the figure on the display device, and
On the first screen, a first reception unit that accepts selection of any of the plurality of nodes and corrections related to the transition of the target state, and
The display shows a second screen showing the relative and physical relationships between the plurality of objects defined in the target state corresponding to the selected node according to the selection of any of the plurality of nodes. The second display unit displayed on the device and
On the second screen, a second reception unit that accepts corrections regarding relative and physical relationships between the plurality of objects in the displayed target state, and
To prepare
Information presentation device. The information acquisition unit further acquires environmental information that defines the initial state of the robot device and each object.
The information presenting device is
A simulator unit that simulates the process of the robot device performing the task based on the acquired state transition information and the environment information.
A third display unit that displays a third screen showing the result of the simulation on the display device, and
Further prepare
The information presenting device according to claim 1. The third screen is a timeline that expresses the elapsed time from the start of the execution of the task, and the robot device performs the transition work for achieving the transition between the target states on the simulation on the timeline. Contains a block that represents the time interval to execute,
The information presenting device according to claim 2. The task is performed by a plurality of robot devices.
At least one of the plurality of robot devices is used for the transition work to achieve the transition between the target states.
The blocks are arranged in association with the robotic apparatus used for the corresponding transition work.
The information presenting device according to claim 3. The third screen further includes a third reception unit that accepts a change in the robot device used for the corresponding transition work by an operation of changing the robot device associated with the block.
The information presenting device according to claim 4. The task is performed by a plurality of robot devices.
On the second screen, the second reception unit further accepts selection of at least one robot device used for the transition work for achieving the transition between the target states from the plurality of robot devices.
The information presenting device according to any one of claims 1 to 5. The robot device is a manipulator and
The plurality of objects include the end effector of the manipulator and the parts of the product to be moved by the manipulator.
The information presenting device according to any one of claims 1 to 6. The computer
It is a step of acquiring state transition information indicating a series of target states that transition in the process from the start to the end of a task executed by the robot device, and each target state included in the series is the robot device in the task. The steps and the steps, which are defined by the relative and physical relationships between the objects that are the objects of the operation of
Based on the acquired state transition information, it is a state transition diagram representing a series of the target states, and includes a plurality of nodes representing each of the target states and an edge representing the transition between the target states. The step of displaying the first screen showing the figure on the display device, and
In the first screen, a step of accepting a selection of any of the plurality of nodes and a modification regarding the transition of the target state, and
The display shows a second screen showing the relative and physical relationships between the plurality of objects defined in the target state corresponding to the selected node according to the selection of any of the plurality of nodes. Steps to display on the device and
In the second screen, a step of accepting a modification regarding the relative and physical relationship between the plurality of objects in the displayed target state, and
To execute,
Information presentation method. On the computer
It is a step of acquiring state transition information indicating a series of target states that transition in the process from the start to the end of a task executed by the robot device, and each target state included in the series is the robot device in the task. The steps and the steps, which are defined by the relative and physical relationships between the objects that are the objects of the operation of
Based on the acquired state transition information, it is a state transition diagram representing a series of the target states, and includes a plurality of nodes representing each of the target states and an edge representing the transition between the target states. The step of displaying the first screen showing the figure on the display device, and
In the first screen, a step of accepting a selection of any of the plurality of nodes and a modification regarding the transition of the target state, and
The display shows a second screen showing the relative and physical relationships between the plurality of objects defined in the target state corresponding to the selected node according to the selection of any of the plurality of nodes. Steps to display on the device and
In the second screen, a step of accepting a modification regarding the relative and physical relationship between the plurality of objects in the displayed target state, and
To execute,
Information presentation program.

Images