JP7347169B2 - Information presentation device, information presentation method, and information presentation program - Google Patents

Description

The present invention relates to an information presentation device, an information presentation method, and an information presentation program.

Various types of robot devices such as manipulators are used in production lines that produce products. The components of a robot device, such as the manipulator mechanism, end effector, and workpiece, have many variations depending on the tasks to be performed. It is difficult to teach a robot device a target task. For this reason, conventionally, after determining the types of components such as mechanisms, end effectors, and workpieces, the robot device is manually moved and the postures of the series of movements to be performed are recorded while being directly taught the task to be performed. method has been adopted.

However, in this method, each time a component such as a mechanism, an end effector, a workpiece, etc. is changed, the robot device is taught the task to be performed. Therefore, it costs too much to teach the robot device which tasks to perform. Therefore, in recent years, research has been conducted to improve the efficiency of methods for making robot devices learn tasks to be performed. For example, Non-Patent Document 1 proposes a method of causing a robot device to learn the motion of grasping an object based on image data obtained from a camera using reinforcement learning. According to this method, it is possible to automate at least a portion of a series of processes for teaching a robot device how to grasp an object. Therefore, the cost of teaching tasks to the robot device can be reduced.

Dmitry Kalashnikov, et al. "QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation" arXiv preprint arXiv:1806.10293, 2018.

The inventors of the present invention have found that the conventional control method for a robot device as described above has the following problems. In conventional control methods, time-series control commands to be given to a robot device for tasks to be performed are learned. In other words, the learned time-series control commands are directly associated with the task. If at least one of the environment and object in which a task is performed changes even slightly, the contents of the task will also change, making it difficult to appropriately perform the task using the associated time-series control commands. Become. Therefore, unless new time-series control commands are learned for the changed task, the robot device may not be able to properly perform the task.

For example, assume that the task taught to the robot device is to transport an object C, which is present at a point A and has a posture B, to a point D. In this case, if the object C is accurately placed at the point A with the posture B, the robot device can appropriately perform the task based on the learning result. However, if object C is placed slightly offset from point A, or placed at point A but tilted from posture B, the position and posture at which the robot device grips object C will change. The content of the task to be performed changes due to reasons such as: That is, in this case, the task to be accomplished is to "transport object C, which has shifted or tilted from point A, to point D," and "transport object C, which is present at point A with attitude B," This task is different from the original task of "carry it to point D." In this case, even if the robot device operates based on the learning results, it may not be able to properly perform this task. Therefore, with the conventional control method, there is a problem in that the ability to accomplish the task that has been learned is not versatile enough. Due to this problem, in order to operate the robot device in a general-purpose manner, control commands must be learned for each different task, and the cost of teaching the robot device a task is still high.

In order to solve this problem, the inventors of the present invention control the operation of a robot device based on the relative physical relationship (relative relationship amount) of multiple objects existing in the environment in which a task is performed. We propose a control method to In this control method, a series of relative relationships between a plurality of objects in the process of a robot device performing a task is defined, and a control command to the robot device is associated with an amount of change in the relative relationships. This makes it possible to teach time-series control commands to be given to the robot device for changing relative relationships, without depending on the content of the task. In other words, as long as the task can be accomplished with the same change in the amount of relative relationship, even if the content of the task has changed somewhat (in the above example, even if the position of object C is shifted or its posture is tilted), ), allowing the robotic device to perform those tasks appropriately. Therefore, according to the control method proposed by the inventors, it is possible to increase the versatility of the ability to perform the tasks that the robot device is to learn, and it is possible to reduce the cost required to teach the robot device the tasks. . However, it is assumed that setting the relative and physical relationships of a plurality of objects in the process of a robot device performing a task is time-consuming.

One aspect of the present invention has been made in view of the above circumstances, and its purpose is to set the relative and physical relationships of a plurality of objects in the process of a robot device performing a task. The aim is to provide technology to increase the efficiency of

The present invention adopts the following configuration in order to solve the above-mentioned problems.

That is, an information presentation device according to one aspect of the present invention is an information acquisition unit that acquires state transition information indicating a series of target states that transition in the process from the start to the end of a task performed by a robot device, Each target state included in the series is defined by a relative and physical relationship between a plurality of objects that are the targets of operation of the robot device in the task, and an information acquisition unit and the acquired state transition. Based on the information, a first state transition diagram representing a series of the target states, the state transition diagram including a plurality of nodes representing each of the target states, and edges representing transitions between the target states; a first display unit that displays a screen on a display device; a first reception unit that accepts a selection of one of the plurality of nodes and a modification regarding transition of the target state on the first screen; In response to the selection of any one of the nodes, a second screen is displayed on the display device showing the relative and physical relationships between the plurality of objects defined in the target state corresponding to the selected node. and a second reception unit that receives, on the second screen, a correction regarding the relative and physical relationship between the plurality of objects in the displayed target state.

According to this configuration, in order to set the relative and physical relationships of a plurality of objects in the process of the robot device performing a task, the first screen showing the state transition diagram and the second screen showing the relative relationships are displayed. Two types of screens are available. According to the first screen, for the process in which the robot device performs a task, a series of target states that define the relative and physical relationships to be achieved between multiple objects involved in the task are appropriately set. You can check whether the Furthermore, if there is a deficiency in the series of target states, it is possible to correct the transition between the target states. On the other hand, according to the second screen, it is possible to check whether the relative and physical relationships of the plurality of objects are appropriately defined for each target state. Furthermore, if there is a deficiency in the relationship, the relationship can be corrected. Therefore, the first screen and the second screen provided allow the user to check and set the overall flow of the task execution process and individual goal states. Thereby, it is possible to improve the efficiency of setting the relative and physical relationships of a plurality of objects in the process of the robot device performing a task.

In the information presentation device according to the above aspect, the information acquisition unit may further acquire environmental information that defines an initial state of the robot device and each of the objects. Furthermore, the information presentation device according to the above aspect includes a simulator unit that simulates a process in which the robot device performs the task based on the acquired state transition information and the environment information; The display device may further include a third display unit that displays a third screen shown on the display device. According to this configuration, it is possible to confirm whether or not the target task can be appropriately performed based on the set series of target states, based on the simulation results. By determining each target state while checking this, the relative and physical relationships of multiple objects in the process of the robot device performing the target task can be appropriately set according to the target task. You will be able to do this.

In the information presentation device according to the above aspect, the third screen includes a timeline representing elapsed time since the start of execution of the task, and a transition for achieving a transition between the target states in the timeline. The method may include a block representing a time period in which the robot device executes a work on the simulation. According to this configuration, visibility of the movement of the robot device and changes in the relative and physical relationships between the objects can be improved on the third screen. Thereby, it is possible to easily confirm whether or not the target task can be appropriately performed based on the set series of target states.

In the information presentation device according to the above aspect, the task may be performed by a plurality of robot devices, and at least one of the plurality of robot devices is used for the transition work to achieve the transition between the target states. The blocks may be arranged in association with a robotic device used for a corresponding transition task. According to this configuration, the blocks representing the time intervals are arranged in association with the robot devices used for the corresponding transition tasks, thereby increasing the visibility of the robot devices that perform each transition task on the third screen. be able to.

The information presentation device according to the above aspect further includes a third reception unit that accepts, on the third screen, a change in the robot device used for the corresponding transition work by an operation to change the robot device with which the block is associated. Good too. According to this configuration, it is possible to select a robot device appropriate for performing each transition task using the simulation results. Thereby, for a task in which a plurality of robot devices cooperate, it is possible to allocate the work of each robot device so that the task can be appropriately performed.

In the information presentation device according to the above aspect, the task may be performed by a plurality of robot devices, and the second reception unit is configured to receive a transition between the target states from the plurality of robot devices on the second screen. The method may further accept selection of at least one robotic device to be used for the transition task. According to this configuration, it is possible to select a robot device suitable for achieving each target state while checking each target state. Thereby, for a task in which a plurality of robot devices cooperate, it is possible to allocate the work of each robot device so that the task can be appropriately performed.

In the information presentation device according to the above aspect, the robot device may be a manipulator, and the plurality of objects may include an end effector of the manipulator and a part of a product to be moved by the manipulator. According to this configuration, it is possible to improve the efficiency of work in which the manipulator sets the relative and physical relationships of a plurality of objects in the process of performing a task.

As another aspect of the information presentation device according to each of the above embodiments, one aspect of the present invention may be an information processing method that realizes each configuration of the above information presentation device, or may be a program. It may be a computer-readable storage medium that stores such a program. A computer-readable storage medium is a medium that stores information such as programs through electrical, magnetic, optical, mechanical, or chemical action.

For example, an information presentation method according to one aspect of the present invention includes a step in which a computer acquires state transition information indicating a series of target states that transition in the process from the start to the end of a task performed by a robot device. , each target state included in the series includes steps and acquired state transition information defined by relative and physical relationships between a plurality of objects that are the targets of operation of the robot device in the task. A first screen showing a state transition diagram representing a series of the target states, the state transition diagram including a plurality of nodes representing each of the target states, and edges representing transitions between the target states. a step of displaying on a display device, a step of accepting, on the first screen, a selection of any one of the plurality of nodes and a correction regarding the transition of the target state; displaying on the display device, in response to the selection, a second screen showing relative and physical relationships between a plurality of objects defined in the target state corresponding to the selected node; The information processing method includes the step of accepting, on a screen, a correction regarding the relative and physical relationship between the plurality of objects in the displayed target state.

For example, an information presentation program according to one aspect of the present invention includes a step of causing a computer to obtain state transition information indicating a series of target states that transition in the process from the start to the end of a task performed by a robot device. , each target state included in the series includes steps and acquired state transition information defined by relative and physical relationships between a plurality of objects that are the targets of operation of the robot device in the task. A first screen showing a state transition diagram representing a series of the target states, the state transition diagram including a plurality of nodes representing each of the target states, and edges representing transitions between the target states. a step of displaying on a display device, a step of accepting, on the first screen, a selection of any one of the plurality of nodes and a correction regarding the transition of the target state; displaying on the display device, in response to the selection, a second screen showing relative and physical relationships between a plurality of objects defined in the target state corresponding to the selected node; The program executes, on a screen, a step of accepting corrections regarding relative and physical relationships between the plurality of objects in the displayed target state.

According to the present invention, it is possible to improve the efficiency of setting the relative and physical relationships of a plurality of objects in the process of a robot device performing a task.

FIG. 1 schematically shows an example of a scene to which the present invention is applied. FIG. 2A schematically shows an example of the relative relationship amount according to the embodiment. FIG. 2B schematically shows an example of the relative relationship amount according to the embodiment. FIG. 3 schematically shows an example of the hardware configuration of the information presentation device according to the embodiment. FIG. 4 schematically shows an example of the software configuration of the information presentation device according to the embodiment. FIG. 5 is a flowchart illustrating an example of a processing procedure of the information presentation device according to the embodiment. FIG. 6 schematically shows an example of a state transition screen according to the embodiment. FIG. 7A schematically shows an example of a status screen according to the embodiment. FIG. 7B schematically shows an example of the status screen according to the embodiment. FIG. 8 schematically shows an example of a simulation screen according to the embodiment. FIG. 9 schematically shows an example of a sequence screen according to the embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment (hereinafter also referred to as "this embodiment") according to one aspect of the present invention will be described below based on the drawings. However, this embodiment described below is merely an illustration of the present invention in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of the invention. In implementing the present invention, specific configurations depending on the embodiments may be adopted as appropriate. Although the data that appears in this embodiment is explained using natural language, more specifically, it is specified using pseudo language, commands, parameters, machine language, etc. that can be recognized by a computer.

§1 Application Example FIG. 1 schematically illustrates an example of an application scene of the present invention. The information presentation device 1 according to the present embodiment is a computer configured to present information regarding the process of performing a task by the robot device 2 and to accept corrections to a target state that transitions in the process of performing the task. The information presentation device 1 according to the present embodiment acquires state transition information 120 indicating a series of target states to which the robot device 2 transitions during revision from the start to the end of a task performed by the robot device 2. Each target state included in the series is defined by a relative and physical relationship (hereinafter also simply referred to as "relative relationship") between a plurality of objects that are the targets of operation of the robot device 2 in a task. In this embodiment, tasks are divided into transition tasks that accomplish transitions between goal states.

In the example of FIG. 1, two robot devices 2 are used to perform a task, and each robot device 2 is a manipulator. However, the number of robot devices 2 is not limited to two, and may be one, or three or more. Further, the type of the robot device 2 is not particularly limited as long as it can manipulate the object, and may be appropriately selected depending on the embodiment. The robot device 2 may include, for example, an industrial robot such as the manipulator described above, an automatically movable moving body, and the like. Industrial robots may include, for example, vertical articulated robots, SCARA robots, parallel link robots, Cartesian robots, collaborative robots, and the like. Further, the automatically movable mobile object may include, for example, a drone, a vehicle configured to be self-driving, a mobile robot, and the like.

In this embodiment, the end effector T and each work (W, V) are examples of objects. The end effector T may be, for example, a gripper, an aspirator, a driver, etc. Each work (W, V) may be a component of a product such as a connector, peg, socket, hole, gear, bolt, or nut. The product may be, for example, a household appliance, a computer device, an automobile, etc. In this manner, the plurality of objects may include the end effector of the manipulator and the parts of the product that are to be moved by the manipulator. However, as long as the object can be related to the operation of the robot device 2, the type thereof is not limited to this example, and may be selected as appropriate depending on the task to be performed, etc. . The target object may include, for example, an obstacle in addition to the end effector and the workpiece. Obstacles may include, for example, workbenches, trays, fixed sensors, and the like.

The type of task is not particularly limited as long as it is work to be performed by the robot device 2, and may be appropriately selected depending on the embodiment. The tasks may be, for example, transporting parts, fitting parts, driving screws, etc. The task may be a simple job such as grasping a workpiece, releasing a workpiece, etc., for example.

The relative physical relationship may be related to, for example, position, force, size (volume), and/or condition. Relative and physical relationships may be defined by relative quantities. The amount of relative relationship indicates the physical amount of an attribute related to the relative relationship between multiple objects. The amount of relative relationship is, for example, relative coordinates (relative position, relative posture), force acting between objects (e.g., load, etc.), relative dimensions, state between objects (e.g., whether they are connected or not), or It may be configured by a combination of these.

FIGS. 2A and 2B schematically illustrate an example of the amount of relative relationship constituted by relative coordinates. Relative coordinates are coordinates when one object is viewed from another object. Either of the two objects may be selected as the reference for relative coordinates. The relative coordinates may include at least one of a relative position and a relative orientation. In a three-dimensional space, a relative position may be expressed by three axes: front and back, left and right, and up and down, and a posture may be expressed by rotation (roll, pitch, yaw) about each axis. The relative coordinates may be expressed by a three-dimensional relative position and a three-dimensional relative orientation. However, the number of dimensions of the relative coordinates does not need to be limited to such an example, and may be determined as appropriate depending on the embodiment. For example, the number of dimensions of the relative position and relative orientation may be reduced as appropriate. Further, for example, the relative orientation may be expressed by a quaternion or a rotation matrix.

FIG. 2A schematically illustrates a situation in which the end effector T attempts to hold the first workpiece W. As in this case, while the end effector is not holding the target work, the state of the task may be defined by the amount of relative relationship between the end effector and the work. In the example of FIG. 2A, the state of the task may be defined by the relative coordinates RC1 between the end effector T and the first work W. The relative coordinate RC1 represents a local coordinate system CW having the reference point W0 of the first workpiece W as the origin, as seen from the local coordinate system CT having the reference point T0 of the end effector T as the origin.

On the other hand, FIG. 2B schematically illustrates a scene in which the first work W held by the end effector T is transported toward the second work V. As in this scene, while the end effector is holding the target work, the state of the task may be defined by the amount of relative relationship between the target work and the object to be transported (for example, another work). . In the example of FIG. 2B, the state of the task may be defined by the relative coordinates RC2 between the first work W and the second work V. Relative coordinates RC2 represent a local coordinate system CV whose origin is the reference point V0 of the second workpiece V, as seen from a local coordinate system CW whose origin is the reference point W0 of the first workpiece W.

Note that each reference point (T0, W0, V0) may be set arbitrarily. Furthermore, the method of giving relative coordinates does not need to be limited to the example described above, and may be determined as appropriate depending on the embodiment. For example, the relative coordinates RC1 may be configured to represent the local coordinate system CT whose origin is the reference point T0 of the end effector T as seen from the local coordinate system CW whose origin is the reference point W0 of the first workpiece W. , the relationship between each relative coordinate (RC1, RC2) may be reversed. Similar to the above-mentioned relative coordinates, relative relationship quantities other than relative coordinates may be appropriately set according to the respective physical quantities.

Returning to FIG. 1, the information presentation device 1 according to the present embodiment creates a state transition diagram representing a series of target states based on the acquired state transition information 120, and a plurality of nodes each representing each target state. A state transition screen 30 showing a state transition diagram including edges 302 and edges 305 representing transitions between target states is displayed on the display device 15. The state transition screen 30 is an example of a first screen. In this embodiment, the state transition diagram further includes a node 301 representing the initial state of the task. Node 301 is connected via edge 305 to node 302 representing an initial goal state (first goal state). The transition relationship expressed by the edge 305 may be, for example, any one of permutation, branching, combination, and repetition. In the example of FIG. 1, n+1 goal states are set (n is a natural number), the n+1-th goal state is the final goal state, and the nodes 302 of each goal state are connected in sequence. Each target state and transition relationship may be given as appropriate. The information presentation device 1 according to the present embodiment accepts selection of any one of the plurality of nodes 302 and correction regarding the transition of the target state on the state transition screen 30. Modifications regarding target state transitions may include, for example, adding or deleting a target state, changing a transition source or a transition destination, or a combination thereof.

In addition, in response to the selection of any one of the plurality of nodes 302, the information presentation device 1 according to the present embodiment displays the relative relationship between the plurality of objects defined in the target state corresponding to the selected node 302. In addition, a status screen 32 showing the physical relationship is displayed on the display device 15. The status screen 32 is an example of a second screen. The information presentation device 1 according to the present embodiment receives, on the status screen 32, corrections regarding the relative and physical relationships between the plurality of objects in the displayed target state. The correction regarding the relative relationship consists of at least one of directly correcting the relative relationship amount and indirectly correcting the relative relationship amount by changing the physical attribute value of each object. good. The physical attribute values of each object may relate to, for example, position, posture, force (eg, weight, acting force, etc.), dimensions, state (eg, presence or absence of connection), and the like.

As described above, the information presentation device 1 according to the present embodiment displays the state transition screen 30 showing the state transition diagram and the state screen 32 showing the relative relationships in each target state on the display device 15. According to the state transition screen 30, in the process of the robot device 2 performing a task, it is possible to determine whether or not a series of target states that define the relative relationships to be achieved among a plurality of objects involved in the task are appropriately set. You can check whether Furthermore, if there is a deficiency in the series of target states, the transition between the target states can be corrected. On the other hand, according to the status screen 32, it is possible to check whether the relative relationships of a plurality of objects are appropriately defined for each target status. Furthermore, if there is a defect in the relative relationship, it is possible to correct the relative relationship. Therefore, according to the displayed state transition screen 30 and state screen 32, it is possible to confirm and set the overall flow of the task execution process and each individual goal state. Thereby, it is possible to improve the efficiency of the work of setting the relative relationships of a plurality of objects in the process in which the robot device 2 performs a task.

§2 Configuration example [Hardware configuration]
FIG. 3 schematically illustrates an example of the hardware configuration of the information presentation device 1 according to this embodiment. As shown in FIG. 3, the information presentation device 1 according to the present embodiment is a computer to which a control section 11, a storage section 12, an external interface 13, an input device 14, a display device 15, and a drive 16 are electrically connected. be. In addition, in FIG. 3, the external interface is described as "external I/F."

The control unit 11 includes a CPU (Central Processing Unit), which is a hardware processor, a RAM (Random Access Memory), a ROM (Read Only Memory), etc., and is configured to execute information processing based on programs and various data. Ru. The storage unit 12 is an example of a memory, and includes, for example, a hard disk drive, a solid state drive, or the like. In this embodiment, the storage unit 12 stores various information such as an information presentation program 81, state transition information 120, CAD (computer-aided design) information 123, and environment information 125.

The information presentation program 81 is a program for causing the information presentation device 1 to execute information processing (FIG. 5), which will be described later, regarding the presentation of information regarding the process of performing a task by the robot device 2. The information presentation program 81 includes a series of instructions for the information processing. The state transition information 120 indicates a series of target states to which the robot device 2 transitions during revision from the start to the end of the task performed by the robot device 2. The CAD information 123 includes configuration information indicating the geometric configuration of a model (for example, a three-dimensional model) of the robot device 2 and each target object (end effector T, first work W, second work V). The environment information 125 defines the initial state of the robot device 2 and each target object. In addition to these, the environment information 125 may further include information indicating the environment in which the task is performed.

The external interface 13 is, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like, and is an interface for connecting to an external device. The type and number of external interfaces 13 may be selected as appropriate depending on the type and number of external devices to be connected. The information presentation device 1 may be connected to each robot device 2 via an external interface 13.

The input device 14 is, for example, a device for performing input such as a mouse or a keyboard. Further, the display device 15 is an example of an output device, and is, for example, a display. An operator can operate the information presentation device 1 via the input device 14 and the display device 15. Note that the display device 15 may be a touch panel display. In this case, the input device 14 may be omitted.

The drive 16 is, for example, a CD drive, a DVD drive, etc., and is a drive device for reading various information such as programs stored in the storage medium 91. The storage medium 91 stores information such as programs through electrical, magnetic, optical, mechanical, or chemical action so that computers, other devices, machines, etc. can read various information such as stored programs. It is a medium that accumulates by At least one of the information presentation program 81, state transition information 120, CAD information 123, and environment information 125 may be stored in the storage medium 91. The information presentation device 1 may acquire at least one of the information presentation program 81, state transition information 120, CAD information 123, and environment information 125 from the storage medium 91. Note that in FIG. 3, a disk-type storage medium such as a CD or a DVD is illustrated as an example of the storage medium 91. However, the type of storage medium 91 is not limited to the disk type, and may be other than the disk type. An example of a storage medium other than a disk type is a semiconductor memory such as a flash memory. The type of drive 16 may be arbitrarily selected depending on the type of storage medium 91.

Note that regarding the specific hardware configuration of the information presentation device 1, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA (field-programmable gate array), DSP (digital signal processor), or the like. The storage unit 12 may be configured by a RAM and a ROM included in the control unit 11. At least one of the external interface 13, the input device 14, the display device 15, and the drive 16 may be omitted. The information presentation device 1 may further include a communication interface so as to be able to perform data communication via a network. The information presentation device 1 may further include an output device other than the display device 15. The information presentation device 1 may be composed of multiple computers. In this case, the hardware configurations of the computers may or may not match. Further, the information presentation device 1 may be a general-purpose server device, a desktop PC (Personal Computer), a tablet PC, a PLC (programmable logic controller), etc., in addition to an information processing device designed exclusively for the provided service. .

[Software configuration]
FIG. 4 schematically illustrates an example of the software configuration of the information presentation device 1 according to this embodiment. The control unit 11 of the information presentation device 1 loads the information presentation program 81 stored in the storage unit 12 into the RAM. The control unit 11 then uses the CPU to interpret and execute instructions included in the information presentation program 81 loaded in the RAM, thereby controlling each component. As a result, as shown in FIG. 4, the information presentation device 1 according to the present embodiment includes an information acquisition section 171, a first display section 172, a first reception section 173, a second display section 174, a second reception section 175, It operates as a computer including a simulator section 176, a third display section 177, a third reception section 178, and an operation control section 179 as software modules. That is, in this embodiment, each software module of the information presentation device 1 is realized by the control unit 11 (CPU).

The information acquisition unit 171 acquires state transition information 120 indicating a series of target states to which the robot device 2 transitions during revision from the start to the end of the task performed by the robot device 2 . The first display unit 172 displays a state transition diagram representing a series of target states based on the acquired state transition information 120, and displays a plurality of nodes 302 representing each target state and transitions between the target states. A state transition screen 30 showing a state transition diagram including edges 305 to be expressed is displayed on the display device 15. The first accepting unit 173 accepts the selection of any one of the plurality of nodes 302 and the modification regarding the transition of the target state on the state transition screen 30.

The second display unit 174 displays, in response to the selection of any one of the plurality of nodes 302, the relative and physical relationships between the plurality of objects defined in the target state corresponding to the selected node 302. A status screen 32 shown is displayed on the display device 15. The second receiving unit 175 receives corrections regarding the relative and physical relationships between the plurality of objects in the displayed target state on the state screen 32.

The information acquisition unit 171 may further acquire environment information 125 that defines the initial state of the robot device 2 and each target object. The simulator unit 176 simulates the process in which the robot device 2 performs a task based on the acquired state transition information 120 and environment information 125. The third display unit 177 displays a sequence screen 36 showing the simulation results on the display device 15. The sequence screen 36 is an example of a third screen.

A task may be performed by a plurality of robot devices 2, and at least one of the plurality of robot devices 2 may be used for transition work to achieve a transition between target states. The third reception unit 178 receives, on the sequence screen 36, a change in the robot device 2 used for the corresponding transition work. The operation control unit 179 controls the operation of the robot device 2 based on the state transition information 120 and the environment information 125 so as to sequentially achieve each target state from the initial state to the final target state.

Each software module of the information presentation device 1 will be explained in detail in the operation example described later. Note that in this embodiment, an example in which each software module of the information presentation device 1 is implemented by a general-purpose CPU is described. However, some or all of the software modules described above may be implemented by one or more dedicated processors. Further, regarding the software configuration of the information presentation device 1, software modules may be omitted, replaced, or added as appropriate depending on the embodiment.

§3 Operation example Next, an operation example of the information presentation device 1 will be described using FIG. 5. FIG. 5 is a flowchart showing an example of the processing procedure of the information presentation device 1 according to the present embodiment. The processing procedure described below is an example of an information presentation method. However, the processing procedure described below is only an example, and each step may be changed as much as possible. Furthermore, steps may be omitted, replaced, or added as appropriate in the processing procedure described below, depending on the embodiment.

(Step S101)
In step S101, the control unit 11 operates as the information acquisition unit 171 and acquires the state transition information 120.

The relative relationship in each target state indicated by the state transition information 120 may be determined by any method. Known methods such as operator input and direct teaching may be used to determine the relative relationship. The state transition information 120 can be created by appropriately determining each goal state and the relative relationships in each goal state from the start to the end of performing the task. When determining a target state and a relative relationship in the target state by using a state transition screen 30 and a state screen 32, which will be described later, initially, the state transition information 120 may be provided by a template or may be empty. It may be information. The template may be determined as appropriate to provide a typical series of goal states and relative relationships in each goal state, depending on the task.

The state transition information 120 may be generated by the information presentation device 1 or by another computer other than the information presentation device 1. When the information presentation device 1 generates the state transition information 120, the control unit 11 determines each target state and the relative relationship in each target state automatically or by input from the operator via the input device 14. State transition information 120 can be generated. On the other hand, when the state transition information 120 is generated by another computer, the control unit 11 acquires the state transition information 120 generated by the other computer, for example, via the network, the storage medium 91, or the like.

Moreover, in this embodiment, the control unit 11 may further acquire the CAD information 123 and the environment information 125 in step S101. The CAD information 123 may be generated by known software. The initial state of the robot device 2 and each target object indicated by the environment information 125 is defined by physical attributes such as the position and posture of the robot device 2, for example. This initial state may be determined by any method similar to each of the target states described above. Similar to the state transition information 120, the CAD information 123 and the environment information 125 may be generated by the information presentation device 1, or may be generated by another computer other than the information presentation device 1. Further, when determining the attribute values of the robot device 2 and each object in the initial state using the status screen 32 or simulation screen 34 described later, the environment information 125 may be initially given by a template. However, it may also be empty information. After acquiring the various information, the control unit 11 advances the process to the next step S102.

(Step S102 to Step S105)
In step S102, the control unit 11 displays a corresponding screen on the display device 15 in response to the operation of the input device 14 by the operator. In this embodiment, the screen displayed on the display device 15 in step S102 is one of the state transition screen 30, the state screen 32, the simulation screen 34, and the sequence screen 36 shown in FIGS. 6 to 9, which will be described later. In step S103, the control unit 11 receives an operator's operation on the screen displayed on the display device 15. In this embodiment, in step S103, the operator edits the state of the robot device 2 and each object in the process of performing a task by operating the display screen.

In step S104, the control unit 11 determines whether the operation received on the displayed screen involves a screen transition. If the received operation involves a screen transition, the control unit 11 returns the process to step S102 and displays the corresponding screen on the display device 15. On the other hand, if this is not the case, the control unit 11 advances the process to the next step S105. In step S105, the control unit 11 determines whether the received operation is related to the end of editing work. If the received operation is related to the end of editing work, the control unit 11 ends the processing procedure according to this operation example. On the other hand, if this is not the case, the control unit 11 returns the process to step S103 and further accepts the operator's operation on the screen displayed on the display device 15.

FIG. 6 schematically illustrates an example of the state transition screen 30 according to this embodiment. In step S102, the control unit 11 operates as the first display unit 172, and displays the state transition screen 30 illustrated in FIG. 6 as the first display screen on the display device 15 based on the acquired state transition information 120. May be displayed. The state transition screen 30 shows a state transition diagram representing a series of target states. The state transition diagram includes a node 301 representing the initial state of a task, a plurality of nodes 302 representing each goal state, and edges 305 representing transitions between the goal states. A thumbnail representing the robot device 2 and each target object in each state or a description of each state may be drawn in each node (301, 302). Node 301 may be omitted.

After displaying the state transition screen 30, in step S103, the control unit 11 operates as the first reception unit 173, and on the state transition screen 30, selects one of the plurality of nodes 302 and selects the target state. Accept corrections regarding transitions. In this embodiment, the state transition screen 30 is provided with six buttons 311 to 316. The control unit 11 accepts various operations including selection of the node 302 and correction regarding transition of the target state using the buttons 311 to 316.

Button 311 and button 312 are used to accept corrections regarding transition of the target state. After receiving the designation of the node 302 in the state transition diagram, the control unit 11 receives the deletion of the designated node 302 and the corresponding target state in response to the operation of the button 311. Further, the control unit 11 accepts addition of a new target state and the corresponding node 302 in response to the operation of the button 312. However, the method of accepting deletion and addition of target states does not have to be limited to this example. As an example of another method, either deletion or addition of a target state may be accepted directly on the state transition diagram without going through the buttons (311, 312).

Further, the control unit 11 accepts deletion and addition of edges 305 representing transitions between target states on the state transition diagram as corrections regarding transitions of target states. Deletion and addition of edges 305 may be accepted by arbitrary operations. As an example, the control unit 11 may delete the specified edge 305 in response to an operation for specifying the edge 305. Further, the control unit 11 may add an edge 305 between the specified target states in response to an operation of specifying the nodes 302 of the transition source and transition destination target states. Deleting and adding the edge 305 corresponds to changing the transition source or transition destination of the target state. In addition, the control unit 11 may accept changes in the nodes 302 that connect the edges 305.

The button 313 is used to instruct the robot device 2 (actual device) to execute an operation based on the series of target states shown in the state transition diagram. The control unit 11 operates as an operation control unit 179 in response to the operation of the button 313, and based on the state transition information 120 and the environment information 125, so as to sequentially achieve each target state from the initial state to the final target state. Controls the operation of the robot device 2. The control command given to the robot device 2 may be composed of, for example, a target control amount, an operation amount, and the like. The control command may be determined as appropriate to achieve transition between each target state. Any method may be used to determine the control command, and for example, known methods such as operator input, direct teaching, etc. may be used. The control unit 11 may directly control the operation of the robot device 2. Alternatively, the control unit 11 may indirectly control the operation of the robot device 2 by giving a control command to a control device such as a PLC.

Button 314 is used to accept execution of a simulation. The button 315 is used to accept a transition to the sequence screen 36 showing the simulation results. The button 315 becomes operable after the simulation is executed.

Button 316 is used to accept selection of any one of the plurality of nodes 302. After receiving the designation of the node 302 in the state transition diagram, the control unit 11 recognizes that the designated node 302 has been selected in response to the operation of the button 316. That is, in this embodiment, selecting any one of the plurality of nodes 302 corresponds to specifying the node 302 in the state transition diagram and operating the button 316. However, the method of accepting the selection of the node 302 does not have to be limited to this example. As an example of another method, selection of node 302 may be accepted directly in the state transition diagram, rather than via button 316.

7A and 7B schematically illustrate an example of the status screen 32 according to this embodiment. In step S104, the control unit 11 determines that a transition operation to the state screen 32 has been received in response to the operation of the button 316 on the state transition screen 30 (that is, selection of any one of the plurality of nodes 302). . Then, in step S102, the control unit 11 operates as the second display unit 174 and displays the status screen 32 illustrated in FIGS. 7A and 7B on the display device 15.

The state screen 32 shows the relative relationships between a plurality of objects defined in the target state corresponding to the selected node 302. The state transition information 120 includes information that allows the amount of relative relationship in each target state to be reproduced. The information that can reproduce the relative relationship amount may be configured by the relative relationship amount itself, or may be configured by the physical attribute value of each object, for example. The control unit 11 specifies the relative relationship defined in the target state corresponding to the selected node 302 by referring to the state transition information 120. Any method may be used to display the relative relationship. In the present embodiment, the control unit 11 uses the CAD information 123 to draw a model of each object on the state screen 32 so as to express the target state corresponding to the selected node 302. Displays the relative relationships between multiple objects defined in the target state.

In the example of FIG. 7A, it is assumed that the relative relationship between the motion of holding the first workpiece W by the end effector T is corrected, and the drawing of the model of the second workpiece V that is unrelated to this motion is omitted. ing. In this way, drawing of objects unrelated to the amount of relative relationship to be corrected may be omitted. On the other hand, in the example of FIG. 7B, it is assumed that the first workpiece W held by the end effector T is transported toward the second workpiece V, and the end effector T, the first workpiece W, and the second workpiece V are transported. Each model is drawn on the status screen 32. The object whose relative relationship amount is to be corrected may be selected as appropriate.

After displaying the status screen 32, in step S103, the control unit 11 operates as the second reception unit 175, and receives corrections regarding the relative relationships between the plurality of objects in the displayed target status on the status screen 32. . The method for accepting modifications may be determined as appropriate depending on the embodiment. In this embodiment, boxes 321 are provided on the status screen 32 according to modifiable attributes regarding relative relationships.

Each box 321 is filled with the position (X, Y, Z) and posture (R, P, Y) is configured to accept changes. In the example of FIG. 7A, two boxes 321 are displayed that accept changes in the positions and postures of the end effector T and the first workpiece W. In the example of FIG. 7B, two boxes 321 for accepting changes in the positions and postures of the first workpiece W and the second workpiece V are displayed. In at least one of the boxes 321, by changing at least one of the values of the position and orientation of the corresponding object, the relative coordinates regarding the object can be modified. Further, the control unit 11 may receive changes in at least one of the values of the position and orientation of each object by operating the model of each object. In this case, the control unit 11 may modify the position and orientation values of each object displayed in each box 321 to match the current values of the model in accordance with the operation of the model of each object.

However, the attributes that can be changed by each box 321 are not limited to this example. Each box 321 may be configured to accept changes in physical attribute values other than position and orientation, such as force, dimension, state, etc., instead of or in addition to position and orientation. Furthermore, the control unit 11 determines at least one of the position and orientation of the reference point of the local coordinate system of each object as an attribute of each object in each box 321 or on the model drawn on the status screen 32. Corrections may be accepted. In the example of FIGS. 7A and 7B, the control unit 11 determines the reference points (T0, W0, V0) of the local coordinate systems (CT, CW, CV) of the end effector T, the first work W, and the second work V. Correction of at least one of the position and orientation may be accepted. Further, the object to be changed by each box 321 is not limited to the physical attribute value of each object. As another example, each box 321 may be configured to directly accept changes in the amount of relative relationship.

Note that, as illustrated in FIG. 7B, when two or more objects are integrated to form one aggregate, such as when the end effector T holds the first workpiece W, the status screen 32 displays this information. Modification of the amount of relative relationship between the aggregate and other objects may be accepted. In this case, the attribute of one object included in the aggregate may be treated as the attribute of the aggregate. That is, the status screen 32 may accept a modification of the amount of relative relationship between one object representing the aggregate and other objects.

One object representing the collection may be selected as appropriate. For example, in the example of FIG. 7B, it is assumed that the first workpiece W is transported toward the second workpiece V, so there is a The target object is the first workpiece W. Therefore, in the status screen 32 of FIG. 7B, the first workpiece W represents the aggregate. In this way, the main object for the operation of the relative relationship quantity to be corrected on the status screen 32 may be selected as the object representing the aggregate. Other objects may also be aggregates.

The amount of relative relationship between each object in the aggregate may be set as appropriate. For example, the control unit 11 may further accept setting or modification of the amount of relative relationship between each target object in the aggregate on the status screen 32 for modifying the amount of relative relationship with other objects. Alternatively, the control unit 11 may receive the setting or modification of the amount of relative relationship between the objects in the assembly on another status screen 32. If the amount of relative relationship between each object in the aggregate is set, the modification of the amount of relative relationship above changes the attributes of one object representing the aggregate, and the remaining objects The attributes of may be changed to satisfy the amount of relative relationship between each object set within the aggregate. For example, in the example of FIG. 7B, in response to the change in the position or orientation of the first workpiece W, the position or orientation of the end effector T is changed so that the end effector T holds the first workpiece W. May be changed. The control unit 11 may appropriately perform calculations to reflect such changes in the attributes of some objects on other objects in accordance with the operator's operations.

Furthermore, in this embodiment, the status screen 32 is provided with four buttons (323 to 325, 331). Two of these buttons (323, 324) are used to limit the possible range of the relative relationship amount. The button 323 is used to specify the touchable range of the object. Any method may be used to specify the contactable range. For example, the contactable range of each object may be specified by specifying an area on the model of each object drawn on the status screen 32. In the example of FIG. 7A, this contactable range is specified in order to determine the range in which the end effector T is allowed to come into contact with the first workpiece W when the end effector T holds the first workpiece W. It's fine. Further, in the example of FIG. 7B, the contactable range may be specified in order to determine the range in which the first workpiece W is allowed to be placed in the second workpiece V. Note that the method of specifying the contactable range is not limited to such a method of directly specifying. The accessible range may be specified indirectly, for example, by specifying the non-accessible range.

On the other hand, the button 324 is used to specify the allowable range of the amount of relative relationship. Any method may be used to specify the allowable range of the relative relationship amount. For example, in response to the operation of the button 324, the control unit 11 may display a slide bar according to the amount of relative relationship for which the specification is accepted, and may accept the specification of the maximum value and the minimum value on the slide bar. Thereby, the control unit 11 may accept the designation of the allowable range of the amount of relative relationship. In the example of FIG. 7A, the specification of this tolerance range is performed to determine the range of the amount of relative relationship between the end effector T and the first workpiece W that is allowed when the end effector T holds the first workpiece W. It's fine. In addition, in the example of FIG. 7B, the specification of this allowable range determines the range of the amount of relative relationship between the first work W and the second work V that is allowed when the first work W is placed on the second work V. It may be done in order to

Note that, at any timing, such as when the amount of relative relationship between any of the objects is changed by operating each box 321, the control unit 11 changes each range specified by each button (323, 324). It may be determined whether the conditions are satisfied or not. If any of the ranges is not satisfied, the control unit 11 may display a warning on the status screen 32 to notify this fact.

For example, in the scene shown in FIG. 7A, after the contactable range is specified on the first workpiece W, the attributes of the end effector T are set such that the end effector T holds a location other than the contactable range on the first workpiece W. Assume that the is changed. In this case, the control unit 11 may display a warning on the status screen 32 to notify that the end effector T is holding a portion of the first workpiece W that is not allowed to touch.

For example, in the scene of FIG. 7A, after the range of relative coordinates between the end effector T and the first work W that are allowed when the end effector T holds the first work W is specified, this range may not be satisfied. Assume that the attributes of the end effector T are changed as follows. In this case, the control unit 11 may display a warning on the status screen 32 to notify that the end effector T is holding the first workpiece W at a relative coordinate that is not permitted.

Furthermore, when the amount of relative relationship between each target object does not satisfy the conditions of each specified range, the control unit 11 may modify the amount of relative relationship between each object as appropriate. For example, the control unit 11 may change the amount of relative relationship between the objects randomly or according to a predetermined rule. The predetermined rule for changing the relative relationship amount may be given by an operator's designation, a setting in a program, or the like. In this case, the control unit 11 calculates a plurality of candidates that satisfy the conditions of the contactable range and the allowable range, and among the calculated candidates, the control unit 11 calculates the candidates that have the least amount of change (i.e., the relative relationship amount specified by the operator). The closest candidate) may be presented on the status screen 32. Then, the control unit 11 may receive a selection on the status screen 32 as to whether or not to adopt this candidate.

Further, the control unit 11 may determine at any timing whether or not physical interference occurs between the objects based on the amount of relative relationship specified on the status screen 32. In the present embodiment, the control unit 11 determines, based on the CAD information 123, that the model of each object is in a state that is physically impossible (for example, a model of a certain object is buried in a model of another object). It may also be determined whether the For example, in the scene of FIG. 7A, the control unit 11 may determine whether interference occurs between the end effector T and the first workpiece W. Furthermore, in the scene of FIG. 7B, the control unit 11 determines whether interference occurs between at least one of the end effector T and the second workpiece V and between the first workpiece W and the second workpiece V. You may.

Furthermore, the control unit 11 takes into consideration the transition of the target state specified in the state transition diagram, and calculates the relationship between each object and the target state for which the relative relationship amount is specified in the target state of the target displayed on the state screen 32. It may be determined whether physical interference will occur with each object for which the amount of relative relationship is specified in a target state to which the target state will transition later (for example, a target state to which the target state will transition next). For example, assume that the goal state of FIG. 7B is the next goal state to transition to the goal state of FIG. 7A. In this case, in the state transition information 120, the amount of relative relationship between the end effector T and the second workpiece V is specified via the first workpiece W. The control unit 11 determines whether interference will occur between the end effector T and the second work V based on the state transition information 120 and the CAD information 123 while the state screen 32 of FIG. 7A is displayed. You may judge. Thereby, it is possible to adjust the amount of relative relationship in the target target state, taking into account the target state to which the target state will transition later.

If it is determined that interference will occur in each scene, the control unit 11 may indicate the object causing the interference on the status screen 32 and display a warning to notify the occurrence of the interference. Further, similarly to the above, the control unit 11 may appropriately modify the amount of relative relationship between the objects that cause interference. In this case, the control unit 11 may calculate a plurality of candidates for relative relationship amounts that do not cause interference, and present on the status screen 32 the candidate with the least amount of change among the calculated candidates. Then, the control unit 11 may receive a selection on the status screen 32 as to whether or not to adopt this candidate.

The button 325 is used to select the robot device 2. A task may be performed by multiple robotic devices 2. In this case, the control unit 11 may further receive, on the state screen 32, a selection of at least one robot device 2 to be used for a transition operation to achieve a transition between target states from among the plurality of robot devices 2. The button 325 is used to select at least one robot device 2 to be used for this transition work.

Specifically, the control unit 11 can accept, in response to the operation of the button 325, the selection of at least one robot device 2 to be used for the transition work to achieve the displayed target state. The method for selecting the robot device 2 to be used may be determined as appropriate depending on the embodiment. As an example, the control unit 11 may display, in response to the operation of the button 325, a list of robot devices 2 that can be used for transition work to achieve the displayed target state. The control unit 11 may then accept selection of at least one robot device 2 from the displayed list.

If the robot device 2 to be used for the transition work has not been determined, this operation allows the robot device 2 to be used for the transition work to be specified. Further, if the robot device 2 to be used for the transition work is determined in advance, the robot device 2 to be used for the transition work can be modified (changed) by this operation. On this status screen 32, the operator can select the robot device 2 appropriate for achieving each target status while checking each target status. As a result, for a task in which a plurality of robot devices 2 cooperate, it is possible to allocate the work of each robot device 2 so that the task can be appropriately performed. In addition, in accordance with the modification of the robot device 2 to be used, the control unit 11 changes the model drawn on the status screen 32 from the object related to the robot device 2 before the change to the object related to the robot device 2 after the change. Good too.

Button 331 is used to return to state transition screen 30. In step S<b>104 described above, the control unit 11 determines that the transition operation to the state transition screen 30 has been accepted in response to the operation of the button 331 . Then, in step S102, the control unit 11 displays the state transition screen 30 illustrated in FIG. 6 again on the display device 15, and executes the processes from step S103 onwards.

Note that the node 301 expressing the initial state may also be handled in the same way as the node 302 expressing each target state. That is, the control unit 11 may accept the selection of the node 301 on the state transition screen 30. Then, in response to the selection of the node 301, the control unit 11 may display on the display device 15 a status screen 32 showing the relative relationships between the plurality of objects in the initial state. Then, on the status screen 32, corrections regarding the relative relationships between the plurality of objects in the initial state may be accepted. In addition to the buttons (323 to 325, 331), the status screen 32 may also include a button for selecting an object whose relative relationship amount is to be corrected. For example, in the example shown in FIG. 7A, the model of the second work V may appear on the status screen 32 in response to the operation of this button, and correction of the amount of relative relationship with the second work V may be accepted.

FIG. 8 schematically illustrates an example of the simulation screen 34 according to this embodiment. In step S<b>104 described above, the control unit 11 determines that the transition operation to the simulation screen 34 has been accepted in response to the operation of the button 314 on the state transition screen 30 . Then, in step S102, the control unit 11 displays the simulation screen 34 illustrated in FIG. 8 on the display device 15. The configuration of the simulation screen 34 may be arbitrary. In this embodiment, the control unit 11 draws a model of the robot device 2 and each object in an initial state on the simulation screen 34 based on the CAD information 123 and the environment information 125. Similar to the state transition information 120, the environment information 125 includes information that can reproduce the initial state, such as attribute values of the robot device 2 and attribute values of each object.

The simulation screen 34 is provided with three buttons (341, 343, 351). The button 341 is used to set the attributes of the robot device 2 and each object in the initial state. The attributes of the robot device 2 may be related to, for example, the type of robot, the initial angle of the joints, the installation position, the position where the end effector is attached, and the like. The attributes of each object may be related to, for example, an installation position, an installation angle (posture), and the like. The initial state setting method may be determined as appropriate depending on the embodiment. As an example, in response to the operation of the button 341, the control unit 11 displays boxes corresponding to attributes that can be changed in the same way as the status screen 32, and by inputting to the displayed box, the robot device 2 in the initial state is And the specification of the attribute value of each object may be accepted. Alternatively, the control unit 11 may accept the designation of the attribute values of the robot device 2 and each object in the initial state in accordance with the operation of the models of the robot device 2 and each object on the simulation screen 34. If the initial state has not been determined in advance, this operation allows the attributes of the robot device 2 and each object in the initial state to be specified. On the other hand, if the initial state is determined in advance, this operation allows the attributes of the robot device 2 and each object in the initial state to be modified.

Button 343 is used to instruct execution of simulation. The control unit 11 operates as a simulator unit 176 in response to the operation of the button 343, and simulates the process in which the robot device 2 performs a task based on the state transition information 120 and the environment information 125. The control unit 11 simulates the operation of the robot device 2 so as to sequentially achieve each target state from the initial state to the final target state, similar to the operation of the operation control unit 179 described above. Simulations may be performed in any manner. Known software may be used for the simulation. Note that the simulation may also be executed when controlling the operation of the actual machine by operating the button 313 described above.

The button 351, like the button 331 described above, is used to return to the state transition screen 30. In step S<b>104 described above, the control unit 11 determines that the transition operation to the state transition screen 30 has been accepted in response to the operation of the button 351 . Then, in step S102, the control unit 11 displays the state transition screen 30 illustrated in FIG. 6 again on the display device 15, and executes the processes from step S103 onwards. At this time, the control unit 11 enables the button 315 described above.

FIG. 9 schematically illustrates an example of the sequence screen 36 according to this embodiment. In step S<b>104 described above, the control unit 11 determines that the transition operation to the sequence screen 36 has been accepted in response to the operation of the button 315 on the state transition screen 30 . Then, in step S102, the control unit 11 operates as the third display unit 177 and displays the sequence screen 36 illustrated in FIG. 9 on the display device 15. Note that the timing for displaying the sequence screen 36 does not have to be limited to this example. As another example, the control unit 11 may automatically display the sequence screen 36 on the display device 15 in response to completion of simulation execution by operating the button 343.

The sequence screen 36 according to this embodiment includes a timeline 361 and blocks (3631 to 3634, 3641 to 3643). The timeline 361 represents the elapsed time since the start of the task execution. Each block (3631 to 3634, 3641 to 3643) represents a time interval in the timeline 361 in which the robot device 2 executes a transition work to achieve a transition between target states on a simulation. As mentioned above, a task may be performed by a plurality of robot devices 2, and at least one of the plurality of robot devices 2 may be used for the transition work. Accordingly, each block (3631-3634, 3641-3643) may be arranged in association with the robot device 2 used for the corresponding transition work. Thereby, the visibility of the robot device 2 that performs each transition work can be improved.

In the example shown in FIG. 9, it is assumed that two robot devices 2 are used to accomplish a task. For convenience of explanation, they will be referred to as a first robot and a second robot, respectively. In addition, as shown in Figure 6, four target states are set for the task, and the second robot is in charge of the transition work from the second target state to the third target state, and the remaining transition work is performed by the second robot. This assumes a situation in which the first robot is in charge. As shown in FIG. 9, for convenience of explanation, the operations of each transition work are sequentially referred to as "first action", "second action", etc.

Each block 3631 to 3634 is associated with the first robot and arranged in the upper stage. On the other hand, each block 3641 to 3643 is associated with the second robot and arranged at the lower stage. Blocks (3631, 3632, 3643) indicate that the first robot is performing each operation related to each transition task. Correspondingly, blocks (3641, 3643) indicate that the second robot is waiting while the first robot is performing each operation. Similarly, block 3642 indicates that the second robot is performing a third operation related to a transition task from the second target state to the third target state. Correspondingly, block 3633 indicates that the first robot is waiting while the second robot is performing the third operation. In the example of FIG. 9, one robot device 2 is in charge of each operation. However, the number of robot devices 2 used for each operation does not need to be limited to this example, and may be two or more.

After displaying this sequence screen 36, in step S103, the control unit 11 operates as the third reception unit 178, and in the sequence screen 36, the control unit 11 selects the robot device 2 to associate each block (3631 to 3634, 3641 to 3643). Depending on the changing operation, a change in the robot device 2 used for the corresponding transition work may be accepted. As an example, assume that the control unit 11 receives an operation from the operator via the input device 14 to move the block 3642 from the lower stage to the upper stage or to move the block 3633 from the upper stage to the lower stage. In this case, the control unit 11 changes the robot device 2 used for the third operation from the second robot to the first robot. Thereby, the robot device 2 used for the corresponding transition work can be modified. On this sequence screen 36, the operator can select the robot device 2 appropriate for performing each transition task while checking the simulation results. As a result, for a task in which a plurality of robot devices 2 cooperate, it is possible to allocate the work of each robot device 2 so that the task can be appropriately performed.

An indicator 362 is provided on the timeline 361 according to this embodiment. Furthermore, the sequence screen 36 according to this embodiment is provided with two display areas (367, 368). The operator can specify any time on the timeline 361 by operating the indicator 362. In the display area 367, the states of the robot device 2 and each object reproduced through simulation are displayed at the time specified by the indicator 362. On the other hand, the display area 368 displays the state of the relative relationship of each object reproduced in the simulation at the time specified by the indicator 362.

Furthermore, a button 371 is provided on the sequence screen 36 according to this embodiment. The button 371 is used to return to the state transition screen 30, similar to the button 331 and the like described above. In step S<b>104 described above, the control unit 11 determines that the transition operation to the state transition screen 30 has been accepted in response to the operation of the button 371 . Then, in step S102, the control unit 11 displays the state transition screen 30 illustrated in FIG. 6 again on the display device 15, and executes the processes from step S103 onwards.

The operator performs the task by operating each screen (30, 32, 34, 36) via the input device 14 while checking the content displayed on each screen (30, 32, 34, 36). It is possible to edit the state of the robot device 2 and each object in the process of doing so. The modification regarding the initial state accepted in this editing is reflected in the environment information 125 at an arbitrary timing after the modification is received. Similarly, corrections related to transitions of target states, corrections related to relative relationships between multiple objects in each target state, and corrections to the robot device 2 to be used can be made in the state transition information 12 at any timing after the corrections are received. reflected in Reflecting a modification may consist of generating new data or updating existing data. The state transition information 120 and the environment information 125 may be structured using a language such as XML (Extensible Markup Language). In this case, the control unit 11 can acquire the state transition diagram and information on each state by parsing the structured state transition information 120 and the environment information 125.

Note that the configuration of each screen (30, 32, 34, 36) does not need to be limited to the above example. Regarding the configuration of each screen (30, 32, 34, 36), components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, each screen (30, 32, 34, 36) may further be provided with a button (not shown) for ending the editing work. In response to the operation of this button, the control unit 11 may determine to end the editing work in step S105, and may end the processing procedure according to the present operation example.

[Features]
As described above, in the present embodiment, two types of screens are displayed on the display device 15 in step S102: the state transition screen 30 showing a state transition diagram and the state screen 32 showing relative relationships in each target state. According to the state transition screen 30, in the process of the robot device 2 performing a task, it is possible to determine whether or not a series of target states that define the relative relationships to be achieved among a plurality of objects involved in the task are appropriately set. You can check whether Furthermore, if there is a deficiency in the series of target states, the transition between the target states can be corrected in step S103. On the other hand, according to the status screen 32, it is possible to check whether the relative relationships of a plurality of objects are appropriately defined for each target status. Furthermore, if there is a defect in the relative relationship, the relative relationship can be corrected in step S103. Therefore, according to the displayed state transition screen 30 and state screen 32, it is possible to confirm and set the overall flow of the task execution process and each individual goal state. Thereby, it is possible to improve the efficiency of the work of setting the relative relationships of a plurality of objects in the process in which the robot device 2 performs a task.

Further, in this embodiment, the sequence screen 36 showing the results of the simulation is displayed on the display device 15 in step S102. According to the sequence screen 36, it is possible to confirm whether or not the target task can be appropriately performed based on the set target state series based on the simulation results. In addition, by determining each target state on the state transition screen 30 and state screen 32 based on this confirmation result, the relative relationship between multiple objects in the process in which the robot device 2 performs the target task is determined. Relationships can now be set appropriately depending on the target task. Further, in this embodiment, the simulation results are expressed on the sequence screen 36 by a timeline 361 and each block (3631 to 3634, 3641 to 3643). This makes it possible to increase the visibility of the movement of the robot device 2 and changes in the relative relationships between objects, and to check whether the target task can be performed appropriately based on the series of set target states. It can be made easier.

§4 Modifications Although the embodiments of the present invention have been described in detail above, the above descriptions are merely illustrative of the present invention in all respects. It goes without saying that various improvements or modifications can be made without departing from the scope of the invention. For example, the following changes are possible. In addition, below, the same code|symbol is used regarding the same component as the said embodiment, and description is abbreviate|omitted suitably about the same point as the said embodiment. The following modified examples can be combined as appropriate.

<4.1>
In the embodiment described above, the control unit 11 displays each screen (30, 32, 34, 36) on the display device 15 included in the information presentation device 1. However, the display device that displays each screen (30, 32, 34, 36) does not have to be limited to this example. The control unit 11 may display each screen (30, 32, 34, 36) on a display device included in another computer.

<4.2>
In the above embodiment, each operation on each screen (30, 32, 34, 36) may be omitted as appropriate. The method of each operation may be changed as appropriate depending on the embodiment. For example, the operation to change the robot device 2 to be used on the status screen 32 may be omitted. In this case, the button 325 may be omitted from the status screen 32. Furthermore, for example, on the status screen 32, at least one of the two buttons (323, 324) may be omitted. Further, for example, the operation to change the robot device 2 to be used on the sequence screen 36 may be omitted. In this case, the third reception unit 178 may be omitted from the software configuration of the information presentation device 1.

In the sequence screen 36, the arrangement of each block (3631 to 3634, 3641 to 3643) may not be associated with the robot device 2 used for the corresponding transition work. In this case, the correspondence between each block (3631 to 3634, 3641 to 3643) and the robot device 2 may be expressed using other methods such as the color of the block. As long as the simulation results can be checked, the sequence screen 36 may show the simulation results by a method other than the method using the timeline 361 and each (3631 to 3634, 3641 to 3643).

In the above embodiments, execution of simulation may be omitted. In this case, the display of the simulation screen 34 and the sequence screen 36 may be omitted. In the state transition screen 30, each button (314, 315) may be omitted. The simulator section 176 and the third display section 177 may be omitted from the software configuration of the information presentation device 1. Furthermore, in the embodiments described above, execution of the operation of the actual device may be omitted. In this case, the button 313 may be omitted on the state transition screen 30. The operation control unit 179 may be omitted from the software configuration of the information presentation device 1.

1... Information presentation device,
11...control unit, 12...storage unit, 13...external interface,
14...Input device, 15...Display device, 16...Drive,
91...Storage medium, 81...Information presentation program,
120... State transition information, 123... CAD information,
125...Environmental information,
171...Information acquisition department,
172...first display section, 173...first reception section,
174...Second display section, 175...Second display section,
176...Simulator section,
177...Third display section, 178...Third reception section,
179...Operation control unit,
2... Robot device,
T...end effector (object),
T0...Reference point, CT...Local coordinate system,
W...first work (object),
W0...Reference point, CW...Local coordinate system,
V...Second work (object),
V0...Reference point, CV...Local coordinate system,
RC1/RC2...Relative coordinates,
30...State transition screen,
301, 302...node, 305...edge,
311-316...buttons,
32...Status screen (second screen),
321...box,
325/331...button,
34...Simulation screen,
341, 343, 351...button,
36...Sequence screen (third screen),
361...Timeline, 362...Indicator,
3631-3634...Block,
3641-3643...Block,
367/368...display area,
371...button

Claims (9)

An information acquisition unit that acquires state transition information indicating a series of target states that transition in a process from the start to the end of a task performed by a robot device, wherein each target state included in the series is an information acquisition unit defined by a relative and physical relationship between a plurality of objects to be operated by the robot device;
Based on the obtained state transition information, a state transition diagram representing a series of the target states, the state transition diagram including a plurality of nodes representing each of the target states, and edges representing transitions between the target states. a first display unit that displays a first screen showing a diagram on a display device;
a first reception unit that accepts a selection of any one of the plurality of nodes and a modification regarding the transition of the target state on the first screen;
In response to the selection of any one of the plurality of nodes, the second screen showing the relative and physical relationship between the plurality of objects defined in the target state corresponding to the selected node is displayed. a second display section for displaying on the device;
on the second screen, a second reception unit that accepts corrections regarding the relative and physical relationships between the plurality of objects in the displayed target state;
Equipped with
Information presentation device. The information acquisition unit further acquires environmental information that defines an initial state of the robot device and each object,
The information presentation device includes:
a simulator unit that simulates a process in which the robot device performs the task based on the acquired state transition information and the environment information;
a third display unit that displays a third screen showing the results of the simulation on the display device;
further comprising;
The information presentation device according to claim 1. The third screen includes a timeline representing the elapsed time since the start of the execution of the task, and a timeline in which the robot device performs a transition work on the simulation to achieve a transition between the target states. Contains a block representing the time interval to be executed,
The information presentation device according to claim 2. the task is performed by a plurality of robotic devices;
At least one of the plurality of robot devices is used for the transition work to achieve the transition between the target states,
the block is arranged in association with a robotic device used for a corresponding transition task;
The information presentation device according to claim 3. further comprising a third reception unit that accepts a change in the robot device used for the corresponding transition work by an operation of changing the robot device with which the block is associated on the third screen;
The information presentation device according to claim 4. the task is performed by a plurality of robotic devices;
The second reception unit further receives, on the second screen, a selection of at least one robot device to be used for a transition work to achieve a transition between the target states from among the plurality of robot devices.
The information presentation device according to any one of claims 1 to 5. The robot device is a manipulator,
The plurality of objects include an end effector of the manipulator and parts of a product to be moved by the manipulator.
The information presentation device according to any one of claims 1 to 6. The computer is
A step of acquiring state transition information indicating a series of target states that transition in a process from the start to the end of a task performed by a robot device, wherein each target state included in the series the steps defined by the relative physical relationships between the objects that are the objects of the operations;
Based on the obtained state transition information, a state transition diagram representing a series of the target states, the state transition diagram including a plurality of nodes representing each of the target states, and edges representing transitions between the target states. displaying a first screen showing a diagram on a display device;
on the first screen, accepting a selection of any one of the plurality of nodes and a modification regarding the transition of the target state;
In response to the selection of any one of the plurality of nodes, the second screen showing the relative and physical relationship between the plurality of objects defined in the target state corresponding to the selected node is displayed. steps to display on the device;
accepting, on the second screen, a modification regarding the relative and physical relationship between the plurality of objects in the displayed target state;
execute,
Information presentation method. to the computer,
A step of acquiring state transition information indicating a series of target states that transition in a process from the start to the end of a task performed by a robot device, wherein each target state included in the series the steps defined by the relative physical relationships between the objects that are the objects of the operations;
Based on the obtained state transition information, a state transition diagram representing a series of the target states, the state transition diagram including a plurality of nodes representing each of the target states, and edges representing transitions between the target states. displaying a first screen showing a diagram on a display device;
on the first screen, accepting a selection of any one of the plurality of nodes and a modification regarding the transition of the target state;
In response to the selection of any one of the plurality of nodes, the second screen showing the relative and physical relationship between the plurality of objects defined in the target state corresponding to the selected node is displayed. steps to display on the device;
accepting, on the second screen, a modification regarding the relative and physical relationship between the plurality of objects in the displayed target state;
In order to execute
Information presentation program.