JP7456552B2 - Information processing device, information processing method, and program - Google Patents

Description

The present disclosure relates to an information processing device, a modification system, an information processing method, and a non-transitory computer-readable medium for performing processing for modifying a robot's motion plan.

When a robot's motion along the optimal path is reproduced on an actual machine, it may exhibit behavior that differs from what a human would originally expect. For example, if an optimal path is generated in which the tip of the arm avoids multiple obstacles, parts of the arm other than the tip of the arm may collide, or an overload may be applied due to the magnitude of changes in the arm's posture. is assumed. Further, when the arm performs an operation while gripping an object, a situation may occur in which the gripped object collides with surrounding people, animals, or objects. After the robot's optimal motion plan is generated in this way, there is a need for a function that allows humans to visualize the motion and modify it.

Patent Document 1 discloses a variable modification method for modifying position variables of a robot control program created by offline programming. Patent Document 2 discloses a robot system, such as a palletizing system, in which a robot is driven and controlled by a program selected and input, and the robot handles a predetermined product (hereinafter referred to as a work). Patent Document 3 discloses a robot motion simulation method in which programming is performed by simulating the motion of an industrial robot using a robot simulator. Patent Document 4 discloses a robot motion simulation method in which programming is performed by simulating the motion of an industrial robot using a robot simulator.

Japanese Unexamined Patent Publication No. 62-106507 Japanese Patent Application Publication No. 07-214485 Japanese Patent Application Publication No. 08-328632 Japanese Patent Application Publication No. 2013-136123

However, the techniques according to Patent Documents 1 to 4 cannot appropriately acquire attributes regarding objects within the robot's operation space when modifying the robot's operation sequence. These attributes can serve as constraints on the robot's operation, but some of the attributes related to these objects may be difficult to accurately determine even with various sensors such as cameras.

The present disclosure has been made to solve such problems, and provides an information processing device, a correction system, and an information processing device capable of acquiring the attributes of objects in the robot's operation space when modifying the robot's operation sequence. One of the purposes is to provide a processing method, etc.

The information processing device according to the first aspect of the present disclosure includes:
an input reception unit that receives input for modifying the motion sequence for the robot;
an object information acquisition unit that acquires object information representing information about an object or a virtual object in the operating space of the robot;
an attribute information acquisition unit that acquires attribute information regarding the object or the virtual object based on the input via the input reception unit;
Equipped with

The correction system according to the second aspect of the present disclosure includes:
a sequence display section that displays the robot's operation sequence;
an input reception unit that receives input for modifying the operation sequence for the robot in response to the display result;
a control signal processing unit that transmits a control signal for the operation sequence to the robot based on the input;
an object information acquisition unit that acquires object information representing information about an object or a virtual object in the operating space of the robot;
an attribute information acquisition unit that acquires attribute information regarding the object or the virtual object based on the input via the input reception unit;
an attribute signal generation unit that combines the object information and the attribute information to generate an attribute signal that is a combination of the object information and the attribute information;
Receive the attribute signal generated by the attribute signal generation unit and having information on a combination of an object and an attribute, and generate an abstract state indicating an operation space of the robot based on the attribute signal and stored stored information. It has an attribute information processing unit that generates correction information to be corrected.

The information processing method according to the third aspect of the present disclosure includes:
accepts input for modifying the motion sequence for the robot;
Obtaining object information representing information about an object or a virtual object in the operation space of the robot,
Attribute information regarding the object or the virtual object is acquired based on the input.

A non-transitory computer-readable medium storing a program according to a fourth aspect of the present disclosure includes a process in which the program receives an input for modifying an operation sequence for the robot;
a process of acquiring object information representing information about an object or a virtual object in the operating space of the robot;
a process of acquiring attribute information regarding the object or the virtual object based on the input;
have the computer execute it.

An information processing device according to a fifth aspect of the present disclosure displays a space in which a robot operates and a plurality of attribute candidates for objects or virtual objects in the space,
A selected attribute candidate among the plurality of attribute candidates is set as an attribute of the object or virtual object.

According to the present disclosure, it is possible to provide an information processing device, a correction system, an information processing method, and the like that can acquire attributes of objects in a robot's operation space.

1 shows a functional block diagram of an information processing device according to a first embodiment. FIG. 4 is a flowchart showing an information processing method according to the first embodiment. FIG. 13 shows a functional block diagram of a correction device according to a second embodiment. 7 is a flowchart showing a correction method according to a second embodiment. The configuration of the robot control system is shown. The hardware configuration of the robot controller is shown. The hardware configuration of the sequence processing device is shown. An example of the data structure of application information is shown. An example of a functional block diagram of a robot controller is shown. An example of a functional block diagram of a sequence processing device is shown. An example of an overhead view of the operating space is shown. An example of a plan display screen before sequence processing in a third embodiment is shown. An example of a plan display screen during sequence processing in the third embodiment is shown. An example of a plan display screen after sequence processing in the third embodiment is shown. 13 shows an example of a plan display screen after sequence processing in the third embodiment. An example of a block of the target logical formula generation unit is shown. 12 is an example of a flowchart showing an overview of correction processing executed by the sequence processing device in the third embodiment. An example of a plan display screen during sequence processing in the fourth embodiment is shown. An example of a plan display screen during sequence processing in the fifth embodiment is shown.

Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and for clarity of explanation, redundant explanation will be omitted as necessary.

<First embodiment>
FIG. 1 shows a functional block diagram of an information processing apparatus according to a first embodiment.
The information processing device 10 is realized by a computer including a processor, memory, and the like. The information processing device 10 can be used to obtain attribute information when a user modifies an operation sequence for a robot.

The information processing device 10 includes an input reception section 72, an object information acquisition section 73, and an attribute information acquisition section 74. The input accepting unit 72 accepts input from the user for modifying the operation sequence for the robot. The input reception unit 72 can receive input from a user via an input device such as a mouse, keyboard, touch panel, stylus pen, or microphone.

The object information acquisition unit 73 acquires information regarding objects or virtual objects within the robot's operating space. As used herein, an object refers to a real object (eg, a real obstacle, a plastic bottle, a door). A virtual object refers to a virtual object (e.g., a virtual obstacle) set (e.g., drawn) by a user in the operating space of the robot. The object information acquisition unit 73 may acquire object information within the robot's operating space based on input from the user via the input receiving unit 72, or may acquire object information within the robot's operating space using various sensors such as a camera. object information may be obtained. For example, the object information acquisition unit 73 can use image recognition technology to acquire object information (for example, position information, shape, and type of object) in the robot's operating space from an image captured by a camera. In other embodiments, for example, object information regarding an object or a virtual object is obtained by the user selecting the object or virtual object displayed on the display (i.e., via the input receiving unit 72 that accepts the user input). , may be acquired from a storage unit that stores information regarding the object or virtual object.

The attribute information acquisition unit 74 acquires attribute information regarding the object based on input from the user via the input reception unit 72. Attribute information is information indicating attributes of an object, and more specifically, may be information that depends on the relationship between the object and the robot. In some embodiments, if the robot is unable to touch or pass through a virtual object drawn by a user, the virtual object is given an attribute of "obstacle" and the robot If the virtual object can be passed through as a waypoint, the attribute of "waypoint" may be given to the virtual object. In other embodiments, if the robot is unable to perform a task related to a lid that is an example of an object (e.g., open), the lid is given an attribute "Closed" and the robot is assigned a task related to the lid (e.g., open). , Open), the attribute "Open" may be given to the lid. Furthermore, in other embodiments, for example, if the robot cannot move the object, the object is given attribute information "obstacle", and if the robot is able to move the object, the object is In some cases, attribute information such as "object" is assigned. That is, the attribute information defines constraints for the robot with respect to the object. Some of this attribute information may include information that cannot be accurately determined even by various sensors such as cameras.

A plurality of selectable pieces of attribute information associated with information regarding the object or virtual object acquired by the object information acquisition unit 73 are presented to the user, and one attribute is acquired according to the user's selection.

The information processing device 10 may include a storage unit that stores various object information and a plurality of selectable attribute information associated with each object, or may be connected to such a storage unit via a network. Good too.

FIG. 2 is a flowchart showing the information processing method according to the first embodiment.
The input accepting unit 72 accepts input from the user for modifying the operation sequence for the robot (step S1a). The object information acquisition unit 73 acquires information about an object or a virtual object within the robot's operation space (step S2a). The attribute information acquisition unit 74 acquires attribute information regarding the object or virtual object based on input from the user (step S3a).

According to the first embodiment described above, when modifying the motion sequence for the robot, it is possible to appropriately acquire attributes regarding objects within the motion space of the robot.

<Second embodiment>
FIG. 3 shows a functional block diagram of the correction device 3 according to the second embodiment. The modification device 3 is realized by a computer including a processor, memory, and the like. The modification device 3 can be used by a user to modify the motion sequence for the displayed robot. The modification device 3 can be used in cooperation with a robot controller described below. As shown in FIG. 3, the correction device 3 includes a control signal processing section 71, an input reception section 72, an object information acquisition section 73, an attribute information acquisition section 74, an attribute signal generation section 75, and a sequence display section 76. The modification device 3 is an example of the information processing device 10 according to the first embodiment.

When the control signal processing unit 71 receives a control signal from the robot controller, it generates a signal for displaying a subtask sequence plan and supplies it to the sequence display unit 76 . Further, when the control signal processing unit 71 receives a control signal from the robot controller, it generates an input acceptance signal for accepting input from the user, and supplies it to the input acceptance unit. Further, upon receiving a signal for operating the robot from the input receiving unit 72, the control signal processing unit 71 transmits a control signal indicating a subtask sequence to the robot.

The sequence display section 76 displays the subtask sequence based on the control signal received from the robot controller.

The input accepting unit 72 accepts input from a user via an input device. The input receiving unit 72 receives as user input operations necessary for changing the robot operation sequence, such as drawing a virtual object in the action space, selecting an object or virtual object, and changing attributes of the object and the drawn virtual object.

When the user uses an input device to select a desired object or virtual object from among the objects or virtual objects in the displayed action space, the object information acquisition unit 73 acquires object information regarding the object or virtual object. Object information regarding an object or a virtual object is stored in advance inside the modification device 3 or in a storage unit connected to the modification device 3.

When the user uses an input device to assign an attribute to the selected object, the attribute information acquisition unit 74 acquires attribute information regarding the selected object. For example, as described above, when a user uses an input device to select a desired object or virtual object from objects or virtual objects in the displayed operation space, multiple objects associated with the desired object or virtual object are The attributes of may be selectably displayed on a display device (eg, a display). Thereafter, when the user selects one attribute from the plurality of attributes using the input device, the attribute information acquisition unit 74 acquires the attribute information.

The attribute signal generation unit 75 generates an attribute signal indicating information obtained by combining the acquired object information and the acquired attribute information based on the object information and attribute information described above, and supplies the generated attribute signal to the robot controller.

FIG. 4 is a flowchart showing a correction method according to the second embodiment.
The sequence display unit 76 displays an operation sequence (subtask sequence) for the robot based on the control signal received from the robot controller (step S1b). The input accepting unit 72 accepts input from the user via the input device for the displayed operation sequence (step S2b). When the user uses an input device to select a desired object or virtual object from among the objects or virtual objects in the displayed operation space, the object information acquisition unit 73 acquires object information regarding the object or virtual object ( Step S3b). When the user assigns an attribute to the selected object using the input device, the attribute information acquisition unit 74 acquires attribute information regarding the object (step S4b). The attribute signal generation unit 75 combines the acquired object information and the acquired attribute information based on the object information and attribute information described above (step S5b). The attribute signal generation unit 75 generates an attribute signal indicating the combined information, and supplies it to the robot controller. As a result, the operation sequence is corrected on the robot controller side (step S6b).

According to the second embodiment described above, it is possible to modify the motion sequence for the robot based on the attribute information about the object etc. given by the user.

<Third embodiment>
(1) System Configuration FIG. 5 shows the configuration of a robot control system 100 according to the third embodiment.
The robot control system 100 mainly includes a robot controller 1, an input device 2, a sequence processing device 3, a storage device 4, a robot 5, and a measuring device 6.

When a task to be executed by the robot 5 (also referred to as a "target task") is specified, the robot controller 1 executes the target task at each time step (time step) of a simple task that the robot 5 can accept. It is converted into a sequence, and the robot 5 is controlled based on the sequence. The robot controller may also be referred to as an information processing device in this specification. Hereinafter, detailed tasks (commands) obtained by breaking down the target task that the robot 5 can accept will also be referred to as "subtasks."

The robot controller 1 is electrically connected to an input device 2, a sequence processing device 3, a storage device 4, a robot 5, and a measuring device 6. For example, the robot controller 1 receives an input signal “S1” from the input device 2 for specifying a target task. Further, the robot controller 1 transmits a display signal "S2" to the input device 2 for displaying a task to be executed by the robot 5. Furthermore, the robot controller 1 transmits a control signal "S3" to the robot 5 regarding control of the robot 5. For example, the robot controller 1 uses the control signal S3 as a sequence of subtasks (also called a "subtask sequence") to be executed by each robot. ) is transmitted to the sequence processing device 3. Further, the robot controller 1 receives an output signal “S6” from the measuring device 6. The robot controller 1 also receives an attribute signal "S5" from the sequence processing device 3 regarding attribute information regarding a specific object or virtual object within the robot's operation space.

The input device 2 is an interface that receives input regarding a target task specified by the user, and includes, for example, a touch panel, buttons, a keyboard, a voice input device (eg, a microphone), a personal computer, and the like. The input device 2 transmits an input signal S1 generated based on user input to the robot controller 1.

Based on the control signals received from the robot controller 1, the sequence processing device 3 processes the robot operation sequence, such as displaying a subtask sequence, drawing a virtual object in the movement space, and changing the attributes of the object and the drawn virtual object. This is a device that allows the user to perform the operations necessary to change the information on the screen. The sequence processing device 3 is also called a modification device, and is an example of the modification device of the second embodiment. The sequence processing device 3 displays the subtask sequence based on the control signal S3 supplied from the robot controller 1, and transmits the control signal S3 to the robot 5 after displaying the subtask sequence. Further, the sequence processing device 3 transmits an attribute signal S5 representing an attribute of an object in the robot's operation space to the robot controller 1. The input device 2 may be a tablet terminal including an input section and a display section, or may be a stationary personal computer.

The storage device 4 has an application information storage section 41 . The application information storage unit 41 stores application information necessary for generating a sequence of subtasks from a target task. Details of the application information will be described later. The storage device 4 may be an external storage medium such as a hard disk connected to or built into the robot controller 1, or may be a storage medium such as a flash memory. Further, the storage device 4 may be a server device that performs data communication with the robot controller 1. In this case, the storage device 4 may be composed of a plurality of server devices.

The robot 5 performs operations related to the target task based on the control signal S3 transmitted from the robot controller 1. The robot 5 is, for example, an assembly robot used at a manufacturing site or a robot that picks packages at a logistics site. In the case of a robot arm, it may have a single arm or two or more arms. In addition to the robot arm, it may also be a mobile robot or a robot that combines a mobile robot and a robot arm.

The measurement device 6 is one or more sensors, which may be a camera, a range sensor, a sonar, or a combination thereof, that measure the state of the robot's operating space in order to accomplish the target task. In this embodiment, the measurement device 6 includes at least one camera that images the operating space. The measuring device 6 supplies the generated measurement signal S6 to the robot controller 1. The measurement signal S6 includes at least image data captured within the operating space. The measuring device 6 does not need to remain in a stopped state, and may be a sensor attached to the robot 5 in operation, a self-propelled mobile robot, or a drone in flight. The measurement device 6 may also include a sensor (eg, a microphone) that detects sound within the operating space. The measuring device 6 is a sensor (for example, a CCD (charge coupled device) or a CMOS (complementary metal oxide semiconductor) image sensor) that captures an image of the operating space, and is a sensor that is attached to any location including outside the operating space. May include.

Note that the configuration of the robot control system 100 shown in FIG. 5 is an example, and various changes may be made to the configuration. For example, a plurality of robots 5 may exist. Further, the robot 5 may include only one or two or more control objects, such as a plurality of robot arms. Even in these cases, the robot controller 1 generates a subtask sequence to be executed for each robot 5 or a controlled object provided in the robot 5 based on the target task, and sends a control signal S3 indicating the subtask sequence to the control signal S3. It is sent to the robot 5 that has the target. Further, the measuring device 6 may be a part of the robot 5. Further, the input device 2 and the sequence processing device 3 may be treated as the same device by being built into the robot controller 1 or the like. Furthermore, the robot controller 1 may be composed of a plurality of devices. In this case, the plurality of devices constituting the robot controller 1 exchange information necessary for executing pre-allocated processing. Further, the robot controller 1 and the robot 5 may be integrally configured. Note that all or part of the robot control system may also be referred to as a modification system, since it can be used by a user to modify the robot's motion sequence.

(2) Hardware Configuration FIG. 6 shows the hardware configuration of the robot controller 1. The robot controller 1 includes a processor 11, a memory 12, and an interface 13 as hardware. Processor 11, memory 12, and interface 13 are connected via data bus 15.

The processor 11 functions as a controller (computing device) that performs overall control of the robot controller 1 by executing a program stored in the memory 12. The processor 11 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors.

The memory 12 includes various types of memories such as RAM (Random Access Memory) and ROM (Read Only Memory). Further, the memory 12 stores a program for the robot controller 1 to execute a specific process. Further, the memory 12 is used as a working memory and temporarily stores information etc. acquired from the storage device 4. Further, part of the information stored in the memory 12 may be stored in one or more external storage media that can communicate with the robot controller 1. Further, the information may be stored in a storage medium that is detachable from the robot controller 1.

The interface 13 is an interface for electrically connecting the robot controller 1 and other devices. These interfaces may be wireless interfaces for transmitting and receiving data with other devices through wireless communication, or may be hardware interfaces for making wired connections with other devices using cables, etc. .

Note that the hardware configuration of the robot controller 1 is not limited to the configuration shown in FIG. 6. For example, the robot controller 1 may be connected to or built in an input device 2, a sequence processing device 3, a storage device 4, and a sound output device such as a speaker or earphone. In these cases, the robot controller 1 may be a tablet terminal or the like having an input/output function and a storage function.

FIG. 7 shows the hardware configuration of the sequence processing device 3. The sequence processing device 3 includes, as hardware, a processor 21, a memory 22, an interface 23, an input section 24a, a display section 24b, and an output section 24c. Processor 21, memory 22, and interface 23 are connected via data bus 25. Further, the interface 23 is connected to an input section 24a, a display section 24b, and an output section 24c.

The processor 21 executes a predetermined process by executing a program stored in the memory 22. The processor 21 is, for example, a processor such as a CPU, GPU, or TPU. The processor 21 receives the signal generated by the input unit 24a via the interface 23, performs a process of converting the acquired attribute information into an attribute signal S5, and transmits the attribute signal S5 to the robot controller 1 via the interface 23. Send. Further, the processor 21 can acquire attribute information by controlling the display section 24b or the output section 24c via the interface 23 based on the control signal S3 received from the robot controller 1 via the interface 23.

The memory 22 is composed of various types of memories such as RAM and ROM. The memory 22 also stores programs for executing processes executed by the sequence processing device. Furthermore, the memory 22 temporarily stores the control signal S3 received from the robot controller 1.

The interface 23 is an interface for electrically connecting the sequence processing device 3 to other devices. These interfaces may be wireless interfaces for wirelessly transmitting and receiving data to other devices, or may be hardware interfaces for wired connection to other devices using cables or the like. The interface 23 also performs interface operations for the input unit 24a, the display unit 24b, and the output unit 24c. The input unit 24a is an interface that accepts user input, and corresponds to, for example, a touch panel, a button, a keyboard, a sound input device (for example, a microphone), etc. The display unit 24b is, for example, a display, a projector, etc., and displays based on the control of the processor 21. The output unit 24c is, for example, a speaker, and outputs sound based on the control of the processor 21.

Note that the hardware configuration of the sequence processing device 3 is not limited to the configuration shown in FIG. For example, at least one of the input section 24a, the display section 24b, and the output section 24c may be configured as a separate device electrically connected to the sequence processing device 3. Further, the sequence processing device 3 may be connected to a measuring device such as a camera, or may have a built-in measurement device.

(3) Application Information Next, the data structure of the application information stored in the application information storage section 41 will be explained.

FIG. 8 shows an example of the data structure of application information stored in the application information storage unit 41. As shown in FIG. 8, the application information storage unit 41 stores abstract state specification information I1, constraint information I2, operation limit information I3, subtask information I4, abstract model information I5, object model information I6, Attribute information I7.

The abstract state designation information I1 is information that designates an abstract state that needs to be defined when generating a subtask sequence. This abstract state is an abstract state of an object in the motion space, and is defined as a proposition used in a target logical formula described later. For example, the abstract state specification information I1 specifies an abstract state that needs to be defined for each type of target task. Note that the target task may be various types of tasks, such as pick-and-place, picking up an object, and rotating an object.

The constraint condition information I2 is information indicating the constraint conditions when executing the target task. For example, when the target task is pick and place, the constraint information I2 includes a constraint that the robot 5 (robot arm) must not come into contact with an obstacle, and a constraint that the robots 5 (robot arms) must not come into contact with each other. Indicates constraints, etc. Note that the constraint information I2 may be information in which constraint conditions suitable for each type of target task are recorded.

The operational limit information I3 indicates information regarding the operational limits of the robot 5 controlled by the robot controller 1. The operational limit information I3 is, for example, information that specifies upper or lower limits of the speed, acceleration, angular velocity, etc. of the robot 5 shown in FIG.

Subtask information I4 indicates information on subtasks that the robot 5 can accept. For example, when the target task is pick and place, the subtask information I4 can define reaching, which is movement of the robot arm of the robot 5, and grasping, which is gripping by the robot arm, as subtasks. The subtask information I4 may indicate information on usable subtasks for each type of target task.

The abstract model information I5 is information regarding an abstract model that abstracts the dynamics in the motion space. As will be described later, the abstract model is represented by a model in which the dynamics of reality are abstracted by a hybrid system. Abstract model information I5 includes information indicating conditions for switching dynamics in the above-described hybrid system. The conditions for switching include, for example, in the case of pick-and-place, in which the robot 5 grabs an object and moves it to a predetermined position, the object cannot be moved unless it is gripped by the robot 5. The abstract model information I5 has information regarding an abstract model suitable for each type of target task.

The object model information I6 is information on the object model of each object in the operating space to be recognized from the measurement signal S6 generated by the measuring device 6. The above-mentioned objects include, for example, the robot 5, obstacles, tools and other objects handled by the robot 5, and moving bodies other than the robot 5. The object model information I6 includes, for example, information necessary for the robot controller 1 to recognize the type, position, posture, and operation being performed of each of the above-mentioned objects, and three-dimensional shape information such as CAD (computer-aided design) data for recognizing the three-dimensional shape of each object. The former information includes parameters of an inference device obtained by learning a learning model in machine learning such as a neural network. For example, when an image is input, this inference device is trained in advance to output the type, position, posture, and other information of the object that is the subject of the image.

The attribute information I7 is information indicating an attribute of an object or a virtual object (for example, an immovable obstacle or a movable object), and is information for adding internal processing in the robot controller 1. Specifically, the attribute information I7 depends on the relationship between the object or virtual object and the robot, and is a constraint on the robot with respect to the object or virtual object. When the robot controller 1 receives the attribute signal S5 generated by the sequence processing device 3, it can perform internal processing according to the acquired attribute. For example, when generating a new virtual obstacle, the sequence processing device 3 executes processing to update the value of the position vector of the obstacle drawn by the user and the information on the number of obstacles after drawing. , the robot can be controlled in a new operating space in which obstacles are newly placed. In another example of changing the object in the robot's operation space from an obstacle to a target object, this can be done by changing the identification label in the object model information I6 from an immovable obstacle to a movable target object. It becomes possible to treat objects that were previously obstacles as targets, and to perform actions such as pick-and-place on objects that are considered targets.

The above attribute information (obstacle or target object) is information indicating whether or not the robot can move the object. That is, this attribute information is based on the relationship between the robot and the object. In other embodiments, various attributes may be used, as described below.

In this way, various attribute information I7 is set in advance and stored in the application information storage section 41.

In addition to the above-mentioned information, the application information storage unit 41 may also store various information regarding the subtask sequence generation process and the control signal S3.

(4) Functional block diagram (4-1) Functional block diagram of robot controller FIG. 9 shows an example of a functional block diagram of the robot controller 1. The processor 11 of the robot controller 1 functionally includes an abstract state setting section 31, a target logical formula generating section 32, a time step logical formula generating section 33, an abstract model generating section 34, and a control input generating section 35. , a subtask sequence generation section 36, and an attribute information processing section 37. Note that although FIG. 8 shows an example of data exchanged between blocks, the present invention is not limited to this. The same applies to other functional block diagrams to be described later.

The abstract state setting unit 31 generates information indicating the measurement results in the operating space (also called "measurement information Im") based on the output signal S6 supplied from the measurement device 6. Specifically, when the abstract state setting unit 31 receives the output signal S6, it refers to the object model information I6, etc., recognizes the type and position of each object in the operating space related to the execution of the target task (robot 5, obstacles, tools and other objects handled by the robot 5, moving objects other than the robot 5, etc.), and generates the recognition result as measurement information Im. In addition, when the abstract state setting unit 31 receives a signal supplied from the attribute information processing unit 37, it updates the aforementioned measurement information Im and newly generates information indicating the measurement results in the operating space taking the attributes into consideration. The abstract state setting unit 31 supplies the generated measurement information Im to the abstract model generation unit 34.

Further, the abstract state setting unit 31 sets an abstract state in the operation space when executing the target task based on the above-mentioned measurement information Im and the abstract state designation information I1 acquired from the application information storage unit 41. In this case, the abstract state setting unit 31 defines a proposition to be expressed by a logical formula for each abstract state. The abstract state setting section 31 supplies information indicating the set abstract state (also referred to as "abstract state setting information Is") to the target logical formula generation section 32.

When receiving the input signal S1 regarding the target task from the input device 2, the target logical formula generation unit 32 converts the target task indicated by the input signal S1 into a time phase representing the final achievement state based on the abstract state setting information Is. It is converted into a logical formula (also called "target logical formula Ltag"). In this case, the target logical formula generation unit 32 adds the constraint conditions to be satisfied in the execution of the target task to the target logical formula Ltag by referring to the constraint information I2 from the application information storage unit 41. Then, the target logical formula generation unit 32 supplies the generated target logical formula Ltag to the time step logical formula generation unit 33. Further, the target logical formula generation unit 32 generates a display signal S2 for displaying a task input screen that receives input necessary for executing the target task, and supplies the display signal S2 to the input device 2.

The time step logical formula generation unit 33 converts the target logical formula Ltag supplied from the target logical formula generation unit 32 into a logical formula (also referred to as “time step logical formula Lts”) representing the state at each time step. do. The time step logical formula generation unit 33 then supplies the generated time step logical formula Lts to the control input generation unit 35.

The abstract model generation unit 34 generates a model that abstracts the actual dynamics in the motion space based on the measurement information Im and the abstract model information I5 stored in the application information storage unit 41.

と示す。 Note that the abstract model is

It shows.

In this case, the abstract model generation unit 34 regards the target dynamics as a hybrid system in which continuous dynamics and discrete dynamics are mixed, and generates an abstract model based on the hybrid system. The method for generating the abstract model will be described later. The abstract model generation unit 34 supplies the generated abstract model to the control input generation unit 35.

The control input generation section 35 satisfies the time step logical formula Lts supplied from the time step logical formula generation section 33 and the abstract model supplied from the abstract model generation section 34, and generates a value for each time step that optimizes the evaluation function. Determine the control input to the robot 5. Then, the control input generation unit 35 supplies information indicating the control input to the robot 5 at each time step (also referred to as “control input information Ic”) to the subtask sequence generation unit 36.

The subtask sequence generation unit 36 generates a subtask sequence based on the control input information Ic supplied from the control input generation unit 35 and the subtask information I4 stored in the application information storage unit 41, and generates a control signal S3 indicating the subtask sequence. is supplied to the sequence processing section 3.

The attribute information processing unit 37 calculates information on a specific object or virtual object and information on a specific attribute based on the attribute signal S5 supplied from the sequence processing device 3 and the attribute information I7 stored in the application information storage unit 41. Modification information Ir for modifying the above-mentioned abstract state is generated according to the combination. The attribute information processing section 37 supplies the modification information Ir to the abstract state setting section 31.

(4-2) Functional block diagram of sequence processing device FIG. 10 is an example of a functional block diagram of the sequence processing device 3. The processor 11 of the sequence processing device 3 functionally includes a control signal processing unit 71, an input reception unit 72, an object information acquisition unit 73, an attribute information acquisition unit 74, an attribute signal generation unit 75, and a sequence display It has a section 76. Note that although FIG. 10 shows an example of data exchanged between blocks, the present invention is not limited to this. The same applies to other functional block diagrams to be described later.

When the control signal processing unit 71 receives the control signal S3 from the robot controller 1, it generates a signal Ss for displaying the subtask sequence plan and supplies it to the sequence display unit 76. Further, when the control signal processing unit 71 receives a control signal from the robot controller 1, it generates an input acceptance signal Si for accepting input from the user, and supplies it to the input acceptance unit 72. Further, upon receiving the signal Sa for operating the robot 5 from the input receiving unit 72, the control signal processing unit 71 transmits a control signal S3 indicating a subtask sequence to the robot 5.

When the input reception unit 72 is supplied with the input reception signal Si from the control signal processing unit 71, it becomes possible for the user to perform operations on the screen. The input reception unit 72 then generates an input display signal Sr for displaying the content input by the user on the screen in real time, and transmits it to the sequence display unit 76. Furthermore, when the user performs an operation to select an object or virtual object on the screen, the input reception unit 72 generates an object selection signal So indicating that the object or virtual object has been selected on the screen. , is supplied to the object information acquisition section 73. Further, when the user performs an operation of selecting an attribute on the screen, the input reception unit 72 generates an attribute selection signal Sp indicating that the attribute has been selected on the screen, and the attribute information acquisition unit 74. Further, the input reception unit 72 generates a signal Sa for operating the robot 5 and supplies it to the control signal processing unit 71.

When the object selection signal So is supplied from the input reception unit 72, the object information acquisition unit 73 acquires object selection information Io representing object information (for example, position vector, object type) corresponding to the object selection signal So. and is supplied to the attribute signal generation section 75. For example, among the object information, the position vector may be measured by the measuring device 6. For example, among the object information, the type of the object can be obtained by recognizing an image taken by the measuring device 6 using an image recognition technique. Information about the virtual object drawn by the user can be obtained from a storage unit that stores drawing information.

When supplied with the object selection signal So from the input reception section 72, the attribute information acquisition section 74 acquires attribute selection information Ip corresponding to the object selection signal So, and supplies it to the attribute signal generation section 75. As will be described later, the attribute selection information is stored in the storage unit as a plurality of selectable attribute information corresponding to various objects.

The attribute signal generation unit 75 generates an attribute signal S5 indicating information obtained by combining the acquired object information and the acquired attribute information, based on the object selection information Io and the attribute selection information Ip. By being supplied to the attribute information processing unit 37 of the robot controller 1, the attribute signal S5 transmits information indicating that "a specific object or virtual object selected by the user has been given a specific attribute" to the robot controller. 1 can be notified.

(5) Details of processing for each block of robot controller Next, details of processing for each functional block of the robot controller 1 shown in FIG. 9 will be explained using a specific example.

(5-1) Abstract state setting unit The abstract state setting unit 31 refers to the object model information I6, analyzes the output signal S6 supplied from the measuring device 6 based on the technology for recognizing the motion space, and Measurement information Im indicating measurement results (type, position, etc.) of each object is generated. Further, the abstract state setting unit 31 generates measurement information Im and sets an abstract state in the motion space. In this case, the abstract state setting unit 31 refers to the abstract state designation information I1 and recognizes the abstract state to be set within the motion space. Note that the abstract state to be set in the action space differs depending on the type of target task. Therefore, when the abstract state to be set for each type of target task is specified in the abstract state designation information I1, the abstract state setting unit 31 sets the abstract state designation corresponding to the target task specified by the input signal S1. The abstract state to be set is recognized by referring to the information I1.

FIG. 11 shows an example of a bird's-eye view of the operation space when the target task is pick-and-place. In the operation space shown in FIG. 11, two robot arms 52a and 52b, four objects 61a to 61d, and an obstacle 62a exist.

In this case, the abstract state setting unit 31 of the robot controller 1 first analyzes the output signal S6 received from the measuring device 6 using the object model information I6, etc. to determine the state of the object 61 and the presence of the obstacle 62a. The range, the existence range of the area G set as the goal point, etc. are recognized. Here, the abstract state setting unit 31 recognizes the position vectors "x1" to "x4" of the center of each of the objects 61a to 61d as the positions of the objects 61a to 61d. Further, the abstract state setting unit 31 recognizes the position vector "xr1" of the robot hand 53a that grips the object 61 and the position vector "xr2" of the robot hand 53b as the positions of the robot arm 52a and the robot arm 52b. Similarly, the abstract state setting unit 31 recognizes the postures of the objects 61a to 61d (unnecessary in the example of FIG. 11 because the objects are spherical), the existence range of the obstacle 62a, the existence range of the region G, and the like. Note that, for example, when the obstacle 62a is regarded as a rectangular parallelepiped and the region G is regarded as a rectangle, the abstract state setting unit 31 recognizes the position vector of each vertex of the obstacle 62a and the region G. Then, the abstract state setting unit 31 generates these recognition results based on the output signal S6 as measurement information Im.

Further, the abstract state setting unit 31 determines the abstract state to be defined in the target task by referring to the abstract state designation information I1. In this case, the abstract state setting unit 31 recognizes the objects and regions existing in the motion space based on the measurement information Im, and the recognition results regarding the objects and regions (for example, the number of each type of objects and regions) and the constraint conditions. Based on the information I2, a proposition indicating an abstract state is determined.

In the example of FIG. 11, the abstract state setting unit 31 assigns identification labels "1" to "4" to the objects 61a to 61d specified by the measurement information Im, respectively. Further, the abstract state setting unit 31 generates a proposition " _g ” is defined. Further, the abstract state setting unit 31 attaches an identification label "O" to the obstacle 62 specified by the measurement information Im, and defines a proposition "o _i " that the object i is interfering with the obstacle O. do. Further, the abstract state setting unit 31 defines a proposition "h" that the robot arms 52 interfere with each other.

FIG. 14 shows an overhead view of the operating space after it has been modified by the sequence processing device 3. In the operation space shown in FIG. 14 , there are two robot arms 52a and 52b, four objects 61a to 61d, an obstacle 62a, and a virtual obstacle 62b. The difference from FIG. 11 is the presence or absence of a virtual obstacle 62b, and this virtual obstacle 62b shown in FIG. A virtual obstacle drawn by the user when it is generated. In this case, when the abstract state setting unit 31 is supplied with the correction information Ir, it assumes that the entity of the virtual obstacle 62b actually exists in the motion space, and a new virtual obstacle 62b is placed in the motion space. As if, I recognized it again.

In this case, in addition to the recognition in the motion space before modification, the abstract state setting section 31 is supplied with modification information Ir that reflects the attributes of the virtual obstacle 62b generated by the attribute information processing section. The existence range of the obstacle 62b is recognized. Then, the abstract state setting unit 31 generates the output signal S6 and the recognition results based on the correction information Ir as measurement information Im.

In the example of FIG. 14, the abstract state setting unit 31 assigns identification labels "1" to "4" to the objects 61a to 61d specified by the measurement information Im, respectively. The abstract state setting unit 31 also generates a proposition “g” that the object “i” (i=1 to 4) exists in the area G (see broken line frame 63) which is the target point where the object “i” (i=1 to 4) should be finally placed. _i ' is defined. Further, the abstract state setting unit 31 attaches an identification label "O" to the obstacle 62a specified by the measurement information Im, and defines a proposition "o _i " that the object i is interfering with the obstacle O. do. Further, the abstract state setting unit 31 attaches an identification label "Ov" to the virtual obstacle 62b specified by the correction information Ir, and creates a proposition "ov _i " that the object i interferes with the virtual obstacle Ov. Define. Furthermore, the abstract state setting unit 31 defines a proposition "h" that the robot arms 52 interfere with each other.

In this way, the abstract state setting unit 31 recognizes the abstract state to be defined by referring to the abstract state specification information I1, and sets the proposition representing the abstract state (in the above example, g _i , o _i , ov _i , h) are respectively defined according to the number of objects 61, the number of robot arms 52, the number of obstacles 62, etc. Then, the abstract state setting section 31 supplies information indicating a proposition representing the abstract state to the target logical formula generation section 32 as abstract state setting information Is.

(5-2) Target Logical Formula Generation Unit FIG. 16 is a functional block diagram of the target logical formula generation unit 32. As shown in FIG. 16, the target logical formula generation section 32 functionally includes an input reception section 321, a logical formula conversion section 322, a constraint information acquisition section 323, and a constraint addition section 324.

The input receiving unit 321 receives an input signal S1 specifying the type of target task and the final state of the object to be worked by the robot. In addition, the input accepting unit 321 transmits a display signal S2 of a task input screen that accepts these inputs to the target logical formula generating unit 32 of the robot controller 1.

The logical formula conversion unit 322 converts the target task specified by the input signal S1 into a logical formula using temporal logic. Note that there are various techniques for converting tasks expressed in natural language into logical expressions. For example, in the example of FIG. 11, it is assumed that a target task of "finally the target object (i=2) exists in area G" is given. In this case, the target logical formula generation unit 32 defines the target task as an operator “◇” corresponding to “eventually” in a linear temporary logic (LTL) and a proposition “g” defined by the abstract state setting unit 31. _i ” to generate the logical expression “◇g ₂ ”. In addition, the target logical expression generation unit 32 generates an arbitrary temporal logic operator other than the operator “◇” (logical product “∧”, logical sum “∨”, negation “￢”, logical inclusion “⇒”, always A logical expression may be expressed using symbols such as "□", next "〇", until "U", etc.). Furthermore, the logical formula may be expressed using not only linear temporal logic but also any temporal logic such as MTL (Metric Temporal Logic) or STL (Signal Temporal Logic).

The constraint information acquisition unit 323 acquires the constraint information I2 from the application information storage unit 41. Note that when the constraint information I2 is stored in the application information storage unit 41 for each type of task, the constraint information acquisition unit 323 acquires the constraint information corresponding to the type of target task specified by the input signal S1. I2 is acquired from the application information storage unit 41.

The constraint addition unit 324 generates the target logical formula Ltag by adding the constraint indicated by the constraint information I2 acquired by the constraint information acquisition unit 323 to the logical formula generated by the logical formula conversion unit 322.

For example, if the constraint information I2 includes two constraints corresponding to pick-and-place: "robot arms 52 do not interfere with each other" and "object i does not interfere with obstacle O", the constraint condition addition unit 324 converts these constraints into logical expressions. Specifically, the constraint addition unit 324 uses the proposition “o _i ” and the proposition “h” defined by the abstract state setting unit 31 in the above explanation to apply the above two constraint conditions to the following, respectively. Convert to logical expression.
□￢h
∧ _i □￢o _i

Therefore, in this case, the constraint addition unit 324 adds these constraints to the logical expression "◇g ₂ " corresponding to the objective task "Finally, the object (i=2) exists in the area G". By adding the logical formula, the following target logical formula Ltag is generated.
(◇ _g2 )∧(□￢h)∧(∧ _i □￢o _i )

In reality, the constraint conditions corresponding to pick-and-place are not limited to the two mentioned above, but may include constraints such as "the robot arms 52 do not grasp the same object" and "the objects do not come into contact with each other." exist. Further, the constraint conditions in the modified motion space as shown in FIG. 14 include a constraint condition such as "object i does not interfere with virtual obstacle Ov". Such constraints are similarly stored in the constraint information I2 and reflected in the target logical formula Ltag.

(5-3) Time step logical formula generation unit The time step logical formula generation unit 33 determines the number of time steps (also referred to as “target time step number”) to complete the target task, and formulates a target logical formula using the target number of time steps. A combination of propositions representing the state at each time step that satisfies Ltag is determined. Since there are usually a plurality of combinations, the time step logical formula generation unit 33 generates a logical formula in which these combinations are combined by a logical sum as the time step logical formula Lts. The above-mentioned combination becomes a candidate for a logical expression representing an operation sequence to be commanded to the robot 5, and will also be referred to as a "candidate φ" hereinafter.

Here, we will explain the processing of the time step logical formula generation unit 33 (FIG. 9) when the objective task "finally the target object (i=2) exists in area G" is set as illustrated in the explanation of FIG. 11. A specific example will be explained.

In this case, the time step logical formula generation unit 33 receives “(◇g ₂ )∧(□￢h)∧(∧ _i □￢o _i )” from the target logical formula generation unit 32 as the target logical formula Ltag. Ru. In this case, the time step logical formula generation unit 33 uses a proposition "g _{i ,k} " which is an extension of the proposition "g i " to include the concept of a time step. Here, the proposition "g _i,k " is the proposition "object i exists in region G at time step k". Here, when the target number of time steps is "3", the target logical formula Ltag is rewritten as follows.
(◇g _2,3 )∧(∧ _k=1,2,3 □￢h _k )∧(∧ _i,k = _1,2,3 □￢o _i )

Moreover, ◇g _{2, 3} can be rewritten as shown in the following formula.

At this time, the target logical formula Ltag described above is expressed by the logical sum (φ ₁ ∨φ ₂ ∨φ ₃ ∨φ ₄ ) of the four candidates “φ ₁ ” to “φ ₄ ” shown below.

Therefore, the time step logical formula generation unit 33 determines the logical sum of the four candidates φ ₁ to φ ₄ as the time step logical formula Lts. In this case, the time step logical formula Lts becomes true when at least one of the four candidates φ ₁ to φ ₄ is true.

Preferably, the time step logical formula generation unit 33 determines the feasibility of the generated candidates by referring to the operation limit information I3, and excludes candidates determined to be unfeasible. For example, the time step logical formula generation unit 33 recognizes the distance that the robot hand can move per one time step based on the operation limit information I3. Further, the time step logical formula generation unit 33 recognizes the distance between the object to be moved (i=2) and the robot hand based on the position vectors of each object and the robot hand indicated by the measurement information Im.

For example, when it is determined that the distance between the robot hand 53a and the robot hand 53b is longer than the movable distance per time step for both the robot hand 53a and the robot hand 53b, the time step logical formula generation unit 33 Candidates φ ₃ and φ ₄ are determined to be unfeasible. In this case, the time step logical formula generation unit 33 excludes candidate φ ₃ and candidate φ ₄ from the time step logical formula Lts. In this case, the time step logical formula Lts is the logical sum (φ ₁ ∨φ ₂ ) of the candidate φ ₁ and the candidate φ ₂ .

In this way, the time step logical formula generation unit 33 refers to the operation limit information I3 and excludes unfeasible candidates from the time step logical formula Lts, thereby suitably reducing the processing load on the subsequent processing unit. be able to.

Next, a supplementary explanation will be given of the method for setting the target time step number.

The time step logical formula generating unit 33 determines the target number of time steps based on, for example, the expected time of the task specified by user input. In this case, the time step logical formula generating unit 33 calculates the target number of time steps from the expected time based on the information on the time width per time step stored in the memory 12 or the storage device 4. In another example, the time step logical formula generating unit 33 stores in advance in the memory 12 or the storage device 4 information that associates the target number of time steps appropriate for each type of target task, and determines the target number of time steps according to the type of target task to be executed by referring to the information.

Preferably, the time step logical formula generation unit 33 sets the target time step number to a predetermined initial value. Then, the time step logical formula generation unit 33 gradually increases the target time step number until the time step logical formula Lts with which the control input generation unit 35 can determine the control input is generated. In this case, if the control input generation unit 35 performs optimization processing using the set target number of time steps and is unable to derive an optimal solution, the time step logical formula generation unit 33 sets the target number of time steps to a predetermined number. Add only a number (an integer greater than or equal to 1).

At this time, the time step logical formula generation unit 33 preferably sets the initial value of the target number of time steps to a value smaller than the number of time steps corresponding to the work time of the target task expected by the user. Thereby, the time step logical formula generation unit 33 suitably suppresses setting an unnecessarily large target time step number.

A supplementary explanation will be given of the effect of the method for setting the target time step number described above. In general, the larger the target time step number, the higher the possibility that an optimal solution will exist in the optimization process by the control input generation unit 35, but the processing load such as the minimization process and the robot 5 required to achieve the target task increase. It takes longer. Taking the above into consideration, the time step logical formula generation unit 33 sets the initial value of the target time step number to a small value, and gradually increases the target time step number until a solution exists in the optimization process of the control input generation unit 35. . Thereby, the time step logical formula generation unit 33 can set the target number of time steps as small as possible within the range where a solution exists in the optimization process of the control input generation unit 35. Therefore, in this case, it is possible to reduce the processing load in the optimization process and shorten the time required for the robot 5 to accomplish the target task.

(5-4) Abstract Model Generation Unit The abstract model generation unit 34 generates an abstract model based on the measurement information Im and the abstract model information I5. Here, information necessary for generating an abstract model is recorded in the abstract model information I5 for each type of target task. For example, if the target task is pick and place, a general-purpose abstraction that does not specify the position and number of objects, the position of the area where the objects are placed, the number of robots 5 (or the number of robot arms 52), etc. The model is recorded in abstract model information I5. Then, the abstract model generation unit 34 generates the position and number of objects indicated by the measurement information Im, the position of the area where the objects are placed, and the robot 5's By reflecting the number of machines, etc., an abstract model Σ is generated.

Here, when the robot 5 is working on a target task, the dynamics within the motion space change frequently. For example, in pick-and-place, when the robot arm 52 is grasping the object i, the object i moves, but when the robot arm 52 is not grasping the object i, the object i does not move.

Taking the above into consideration, in the present embodiment, in the case of pick-and-place, the action of grabbing an object i is abstractly expressed using a logical variable "δ _i ". In this case, for example, the abstract model generation unit 34 can determine the abstract model to be set for the motion space shown in FIG. 1 using the following equation (1).

Here, "u _j " indicates a control input for controlling robot hand j ("j=1" is robot hand 53a, "j=2" is robot hand 53b), and "I" is a unit matrix. show. Note that although the control input is assumed to be speed as an example here, it may be acceleration. Moreover, "δ _j,i " is a logical variable that becomes "1" when the robot hand j grasps the object i, and becomes "0" in other cases. Furthermore, “x _r1 ” and “x _r2 ” represent the position vectors of the robot hand j, and “x ₁ ” to “x ₄ ” represent the position vectors of the object i. Furthermore, “h(x)” is a variable that satisfies “h(x)≧0” when the robot hand is close enough to the object to be able to grasp the object, and the relationship between it and the logical variable δ is as follows: satisfies the relationship.
δ=1 ⇔ h(x)≧0

Here, equation (1) is a difference equation showing the relationship between the state of the object at time step k and the state of the object at time step k+1. In Equation (1) above, the grasp state is expressed by a logical variable with a discrete value, and the movement of the object is expressed by a continuous value, so Equation (1) indicates a hybrid system. .

Equation (1) does not take into account the detailed dynamics of the entire robot 5, but only the dynamics of the robot hand, which is the tip of the robot 5 that actually grips the object. Thereby, the amount of calculation for optimization processing can be suitably reduced by the control input generation unit 35.

In addition, the abstract model information I5 includes information for deriving the difference equation of equation (1) from the logical variables and measurement information Im that correspond to the action of switching the dynamics (the action of grabbing the object i in the case of pick-and-place). Information is recorded. Therefore, the abstract model generation unit 34 generates the abstract model information I5 and the measurement information even when the position and number of objects, the area where the objects are placed (region G in FIG. 11), the number of robots 5, etc. change. By combining with Im, it is possible to determine an abstract model that suits the environment of the target motion space.

Note that instead of the model shown in equation (1), the abstract model generation unit 34 generates a model of a mixed logical dynamical (MLD) system or a hybrid system that combines Petri nets, automatons, etc. Good too.

(5-5) Control Input Generation Unit The control input generation unit 35 generates an optimal Determine the control input for each time step to the robot 5 for each time step. In this case, the control input generation unit 35 defines an evaluation function for the target task, and solves an optimization problem for minimizing the evaluation function using the abstract model and the time step logical formula Lts as constraints. The evaluation function is determined in advance for each type of target task, for example, and is stored in the memory 12 or the storage device 4.

For example, when pick and place is the target task, the control input generation unit 35 determines whether the distance “d _k ” between the object to be carried and the target point to which the object is carried and the control input “u _k ” are the minimum. The evaluation function is determined so that the energy consumed by the robot 5 is minimized. The above-mentioned distance _dk corresponds to the distance between the object (i=2) and the region G in the case of the objective task of "finally the object (i=2) exists in the region G".

For example, the control input generation unit 35 determines the sum of the square of the distance d _k and the square of the control input u _k in all time steps as the evaluation function, and sets the abstract model and the time step logical formula Lts (that is, the candidate φ _i Solve the constrained mixed integer optimization problem shown in Equation (2) below with the constraint condition ``OR''.

Here, "T" is the number of time steps to be optimized, and may be the target number of time steps, or may be a predetermined number smaller than the target number of time steps, as will be described later. In this case, the control input generation unit 35 preferably approximates the logical variable to a continuous value (sets it as a continuous relaxation problem). Thereby, the control input generation unit 35 can suitably reduce the amount of calculation. Note that if STL is used instead of linear logic equations (LTL), it is possible to describe the problem as a nonlinear optimization problem.

Furthermore, when the target number of time steps is long (e.g., larger than a predetermined threshold), the control input generating unit 35 may set the number of time steps used for optimization to a value smaller than the target number of time steps (e.g., the above-mentioned threshold). In this case, the control input generating unit 35 sequentially determines the control input u _k by solving the above-mentioned optimization problem, for example, every time a predetermined number of time steps elapses.

Preferably, the control input generation unit 35 may solve the above-mentioned optimization problem and determine the control input u _k to be used for each predetermined event corresponding to an intermediate state with respect to the achievement state of the target task. In this case, the control input generation unit 35 sets the number of time steps until the next event occurs to the number of time steps used for optimization. The above-mentioned event is, for example, an event in which the dynamics in the motion space are switched. For example, if pick and place is the objective task, events such as the robot 5 grabbing an object or the robot 5 finishing transporting one of the multiple objects to the destination point are defined as events. It will be done. The event is determined in advance for each type of target task, for example, and information specifying the event for each type of target task is stored in the storage device 4.

(5-6) Subtask sequence generation unit The subtask sequence generation unit 36 generates a subtask sequence Sr based on the control input information Ic supplied from the control input generation unit 35 and the subtask information I4 stored in the application information storage unit 41. generate. In this case, the subtask sequence generation unit 36 refers to the subtask information I4 to recognize subtasks that can be accepted by the robot 5, and converts the control input for each time step indicated by the control input information Ic into a subtask.

For example, in the subtask information I4, functions indicating two subtasks, namely, moving (reaching) of the robot hand and grasping (grasping) of the robot hand, are defined as subtasks that the robot 5 can accept when the target task is pick-and-place. In this case, the function "Move" representing reaching is a function that takes as arguments, for example, the initial state of the robot 5 before the execution of the function, the final state of the robot 5 after the execution of the function, and the time required to execute the function. Furthermore, the function "Grasp" representing grasping is a function that takes as arguments, for example, the state of the robot 5 before the execution of the function, the state of the object to be grasped before the execution of the function, and the logical variable δ. Here, the function "Grasp" represents a grasping action when the logical variable δ is "1", and represents a releasing action when the logical variable δ is "0". In this case, the subtask sequence generation unit 36 determines the function "Move" based on the trajectory of the robot hand determined by the control input for each time step indicated by the control input information Ic, and determines the function "Grasp" based on the transition of the logical variable δ for each time step indicated by the control input information Ic.

Then, the subtask sequence generation unit 36 generates a control signal S3 indicating a subtask sequence composed of the function "Move" and the function "Grasp". For example, if the objective task is "Finally, the target object (i=2) exists in area G", the subtask sequence generation unit 36 generates the function " A subtask sequence of the function "Move", the function "Grasp", the function "Move", and the function "Grasp" is generated. In this case, the robot hand closest to the object (i = 2) moves to the position of the object (i = 2) by the first function "Move", and moves to the position of the object (i = 2) by the first function "Grasp". =2), move it to area G using the second function "Move", and place the object (i=2) in area G using the second function "Grasp".

Further, the subtask sequence generation section 36 transmits a control signal S3 to the sequence display section 76 of the sequence processing device 3. This is done by transmitting the control signal S3 to the robot 5 and before actually moving it, after transmitting it to the sequence display section 76 of the sequence processing device 3, a subtask sequence by a robot model of the same type as the robot 5 that has been set in advance is displayed with human eyes. This is for confirmation. When the control signal S3 is supplied to the sequence display section 76 of the sequence processing device 3, a robot model of the same type as the robot 5 displayed on the sequence display section 76 of the sequence processing device 3 operates according to the generated subtask sequence. This operation can be confirmed repeatedly. Furthermore, it is possible to rotate and translate the viewpoint in three dimensions during operation, and it is possible to confirm the operation of the robot in the operation space according to the subtask sequence from a desired viewpoint.

(5-7) Attribute information processing unit The attribute information processing unit 37 is based on the attribute signal S5 generated by the attribute signal generation unit 75 of the sequence processing device 3 and the attribute information I7 stored in the application information storage unit 41. , generates modification information Ir. In this case, the attribute information processing unit 37 recognizes the combination of the object selected by the user and the attribute selected by the user in the sequence processing device 3 by referring to the attribute information I7, and Accordingly, modification information Ir for modifying the abstract state is generated.

For example, in the example of FIG. 14, in the task of moving the object 61d to the area G, by newly generating the virtual obstacle 62b, a new plan for moving the object 61d to the area G while avoiding the virtual obstacle 62b is created. It becomes possible to generate.

In the example shown in FIG. 14, the attribute information processing unit 37 generates an attribute that has information from the attribute signal generation unit 75 of the sequence processing device 3 that “the object Ov drawn by the user has been given the attribute of an obstacle.” A signal S5 is provided. Further, by referring to the attribute information I7, the correction information Ir that "newly generate a virtual obstacle 62b at a specific position in the motion space" is generated based on the information of the attribute signal S5. By supplying the modification information Ir to the abstract state setting unit 31, it becomes possible to newly set an abstract state in which "the virtual obstacle 62b is placed in the motion space from the beginning."

In addition, in the example of Figure 15, in the task of moving the object 61d to area G, by generating a new via point 62c, it is possible to generate a new plan to move to area G via via point 62c.

Furthermore, in the example shown in FIG. 15, the attribute information processing unit 37 receives information from the attribute signal generation unit 75 of the sequence processing device 3 that “the object Ov drawn by the user has been given the attribute of way point.” An attribute signal S5 having the same attribute is supplied. Furthermore, by referring to the attribute information I7, correction information Ir that "a new way point 62c is generated at a specific position in the motion space" is generated based on the information of the attribute signal S5. By supplying the modification information Ir to the abstract state setting unit 31, it becomes possible to newly set an abstract state in which "the way point 62c is set in the motion space from the beginning."

(6) Details of processing for each block of sequence processing device Next, details of processing for each functional block of the sequence processing device 3 shown in FIG. 10 will be described using a specific example.

(6-1) Control signal processing unit When the control signal processing unit 71 receives the control signal S3 indicating the subtask sequence from the robot controller 1, it generates a plan display signal Ss for displaying the plan of the subtask sequence, The signal is supplied to the sequence display section 76. Further, when the control signal processing unit 71 receives a control signal from the robot controller 1, it generates an input acceptance signal Si for accepting input from the user, and supplies it to the input acceptance unit 72. The input acceptance signal Si also includes information such as the position vectors and shapes of robots, obstacles, objects, and other objects constructing the operating space in the operating space. Further, the control signal processing unit 71 can store the received control signal S3 without returning it. Furthermore, when the control signal processing unit 71 receives the signal Sa for operating the robot 5 from the input receiving unit 72, it sends the stored control signal S3 as it is to the robot 5, thereby making the robot follow the subtask sequence. It becomes possible to perform operations.

(6-2) Input Acceptance Unit When the input acceptance unit 72 is supplied with the input acceptance signal Si from the control signal processing unit 71, it becomes possible for the user to perform operations on the screen. Specifically, the input reception signal Si includes information such as the position vectors and shapes of robots, obstacles, objects, and other objects that construct the operation space in the operation space, and the input reception signal By performing the conversion process, it becomes possible to perform operations that reflect this information.

For example, FIG. 13 shows a screen during sequence processing in the third embodiment. At this time, the user can use an input device such as a mouse to draw the virtual object Ov on the screen at a position where it will not interfere with other objects. After that, an icon (an inverted triangle icon in Figure 13) that allows you to select an attribute is displayed near the virtual object Ov, and when you select the icon on the screen, you can select two attributes: "obstacle" or "way point". can do. If "obstacle" is selected at this time, the virtual object Ov is regarded as a newly generated obstacle. On the other hand, when "way point" is selected, the virtual object Ov is regarded as a newly generated way point. If the virtual object Ov is considered to be a waypoint, the robot is constrained to travel within the area of the virtual object Ov.

The input reception unit 72 then generates an input display signal Sr for displaying the content input by the user on the screen in real time, and transmits it to the sequence display unit 76. As a result, operations such as drawing a virtual object on the screen, selecting an object or virtual object, and selecting an attribute are displayed in real time.

Furthermore, when the user performs an operation such as drawing an object on the screen or selecting an object, the input reception unit 72 generates an object selection signal So indicating that the object has been selected on the screen. and supplies it to the object information acquisition section 73.

Further, when the user performs an operation of selecting an attribute on the screen, the input reception unit 72 generates an attribute selection signal Sp indicating that the attribute has been selected on the screen, and the attribute information acquisition unit 74.

Further, the input reception unit 72 generates an operation signal Sa for operating the robot 5 according to a user's instruction, and supplies it to the control signal processing unit 71. Specifically, options for determining whether or not to operate the robot according to the subtask sequence are displayed on the screen, and the selection by the user determines whether or not to perform the robot operation. When the input receiving unit 72 receives an input from the user to perform a robot action, the robot action is performed by generating an action signal Sa and supplying it to the control signal processing unit 71. On the other hand, when receiving an input indicating that the robot will not perform robot motion, the input receiving unit 72 accepts input regarding sequence processing, such as drawing of virtual objects in the motion space, selection of objects, and settings for each object. Operations such as selecting attributes become possible.

(6-3) Object Information Acquisition Unit When the object information acquisition unit 73 is supplied with the object selection signal So from the input reception unit 72, the object information acquisition unit 73 supplies the application information storage unit 41 with information about the object selected on the screen by the user. From there, object selection information Io, which is object model information I6 corresponding to the selected object, is acquired and supplied to the attribute signal generation section 75. This object selection information Io includes the position vector (for example, x, y coordinates) of the selected object, the shape of the object (for example, rectangle, circle, cylinder, sphere), the type of object (the type of object, , or virtual objects). For example, in the example of FIG. 13, when the virtual object Ov is selected, information such as "value of the position vector of the vertex", "rectangle", and "drawn object" of Ov is acquired. In some embodiments, the object information acquisition unit 73 may perform an image recognition technique on an image taken by a camera to recognize the object and acquire object information about the object.

(6-4) Attribute information acquisition unit When the attribute selection signal Sp is supplied from the input reception unit 72, the attribute information acquisition unit 74 acquires information on the attribute selected on the screen by the user from the application information storage unit 41. Obtain the attribute selection information Ip to represent. The acquired attribute selection information Ip is supplied to the attribute signal generation section 75. For example, in the example of FIG. 13, the attribute in which either "obstacle" or "way point" is selected is acquired.

(6-5) Attribute signal generation unit The attribute signal generation unit 75 generates an attribute signal indicating information obtained by combining the acquired object information and the acquired attribute information based on the object selection information Io and the attribute selection information Ip. Generate S5. By being supplied to the attribute information processing section 37, the attribute signal S5 can notify the robot controller 1 of information that "a specific object selected by the user has been given a specific attribute." For example, in the example of FIG. 13, the attribute signal generation unit 75 generates an attribute signal S5 indicating that "the virtual object Ov existing at the drawn position has been given the attribute of an obstacle."

(7) Details of Display Screen Information output to the sequence display section 76 of the sequence processing device 3 will be explained.

FIG. 12 is a display example of a plan before sequence processing in the third embodiment. This display example is the first display displayed when the display signal Ss is supplied to the sequence display section 76 from the control signal processing section 71. In FIG. 12, a plan 64d, which is a subtask sequence plan, is displayed in the work overhead view of FIG. After the user confirms the plan 64d on the screen, the input reception unit 72 displays an input on the screen for determining whether or not to supply the control signal processing unit 71 with the operation signal Sa for performing the robot operation. done above.

FIG. 13 is a display example of a plan during sequence processing in the third embodiment. This display example is a display example when the object 62b is drawn by the user after the input reception unit 72 receives an input indicating that the robot does not perform an action. At this time, when the drawn object 62b is selected, an icon 66 that allows attribute selection is displayed near the object 62b, and when the icon 66 is selected, a window 67 showing attributes is displayed. In the first embodiment, a plurality of attributes such as "obstacle" and "way point" are displayed in the window 67, and when one of the attributes is selected on the screen, the attribute information acquisition unit 74 selects the selected attribute. The information of the specified attributes is obtained.

( 8 ) Processing Flow FIG. 17 is an example of a flowchart showing an overview of subtask sequence processing executed by the sequence processing device 3 and the robot controller 1 in the third embodiment.

First, the abstract state setting unit 31 of the robot controller 1 generates measurement information Im indicating the measurement result of an object in the motion space and sets an abstract state based on the output signal S6 supplied from the measurement device 6 (step S11). Next, the control input S3 of the subtask sequence is generated by the processing of the target logical formula generation unit 32, time step logical formula 33, abstract model generation unit 34, control input generation unit 35, and subtask sequence generation unit 36 of the robot controller 1. (Step S12).

Next, the control signal processing unit 71 of the sequence processing device 3 generates a signal Ss for displaying the subtask sequence, and the sequence display unit displays the plan of the subtask sequence on the screen (step S13). If the plan of the subtask sequence is a plan desired by the user, when the user gives an instruction using the input device, the input reception unit 72 generates a signal Sa for performing robot operation, and the control signal processing unit 71 is supplied to After that, the control signal S3 is supplied to the robot 5, and the robot operates (step S14; Yes).

On the other hand, if the user determines that a subtask sequence modification processing operation is necessary, and the user gives an instruction using the input device, the input reception unit 72 determines that the sequence modification processing operation by the user is performed in step S14; (Step S15).

Next, when a specific object is selected by the user on the screen, the object information acquisition unit 73 receives the object selection signal So and acquires object information such as the position vector, shape, and type of the selected object. Object selection information Io corresponding to the object selection signal So expressed is acquired (step S16). Further, when an attribute is selected by the user on the screen, the attribute information acquisition unit 74 receives an attribute selection signal Sp and acquires attribute information Ip corresponding to the attribute selection signal Sp representing information on the selected attribute. (Step S17). Based on the object selection information Io and the attribute selection information Ip, the attribute signal generation unit 75 generates an attribute signal S5 indicating information obtained by combining the acquired object information and the acquired attribute information (step S18 ). The generated attribute signal S5 is supplied to the robot controller 1.

Then, the attribute information processing unit 37 of the robot controller 1 generates an abstract state based on the attribute signal S5 generated by the attribute signal generation unit 75 of the sequence processing device 3 and the attribute information I7 stored in the application information storage unit 41. is newly set, and correction information Ir necessary for recognizing the state of the operation space after correction is generated (step S19). After that, the process in the flowchart returns to step S11.

The abstract state setting unit 31 generates measurement information Im indicating the measurement result of the object in the motion space based on the correction information Ir supplied from the attribute information processing unit 37 and the output signal S6 supplied from the measurement device 6. The settings of the abstract state are updated (step S11). Thereafter, as described above, each process of steps S12, S13, and S14 is executed.

Although the flowchart of FIG. 17 shows a specific order of execution, the order of execution may be different from that depicted. For example, the order of execution of two or more steps may be permuted relative to the order shown. Also, two or more steps shown sequentially in FIG. 17 may be performed simultaneously or partially simultaneously. Additionally, in some embodiments, one or more steps shown in FIG. 17 may be skipped or omitted.

As described above, the sequence processing device (correction device) 3 and the robot controller 1 cooperate to control the operation of the robot 1. Therefore, the functional blocks of the robot controller and the functional blocks of the sequence processing device described above are merely examples. In other embodiments, some or all of the functional blocks of the sequence processing device (correction device) 3 may be included as functions of the robot controller 1, or vice versa.

<Fourth embodiment>
FIG. 18 is an example of a screen during sequence modification by the sequence processing device 3 in the fourth embodiment. The task displayed in FIG. 18 is a pick-and-place task in which the robot arm 80 “transports a plastic bottle 81 to area G.” In this example, by adding attributes related to the state of the target object, The goal is to accomplish the following tasks. Specifically, by assigning the attributes "Open" and "Closed" to the PET bottle 81 with a lid attached, the pick-and-place task is executed after changing the state according to the selected attribute. . Even with the measuring device 6, it is not possible to determine whether the lid of the plastic bottle is open or closed. Therefore, by assigning attributes such as "Open" and "Closed" by the user, it is possible to appropriately modify the robot's operation sequence and achieve a desired task by the robot. The attribute "Open" indicates information that the plastic bottle has a lid, but the lid is not completely sealed because the lid has been opened by a user or the like. That is, the robot is capable of performing lid-related tasks (eg, opening). On the other hand, the attribute "Closed" indicates that the plastic bottle has a lid attached in a completely sealed state, so that the robot cannot perform a task related to the lid (eg, opening). The attribute information according to this embodiment may also depend on the relationship between the robot and the object.

For example, the input receiving unit 72 of the sequence processing device 3 receives an input when a plastic bottle 81 is selected on the screen by the user. Thereafter, an icon 82 that allows attribute selection and a window 84 that visually shows the status of the plastic bottle 81 are displayed near the plastic bottle 81. Furthermore, when the user selects an icon on the screen, he or she can select two attributes: "Open" and "Closed." At this time, if "Open" is selected, a new task to "open the lid of the plastic bottle 81" can be generated for the robot. That is, it is possible to generate a plan for performing the task of "carrying the plastic bottle 81 to area G" in stages after the task of "opening the lid of the plastic bottle 81". On the other hand, when "Closed" is selected, if the lid of the plastic bottle 81 is attached, the task related to the lid is not performed, and only the task of "transporting the plastic bottle 81 to area G" is performed.

For example, in the example shown in FIG. 18, if the initial state of the PET bottle is "with the lid open", if you select "Closed", the task "Close the lid of PET bottle 81" is followed by "PET bottle 81". 81 to area G. On the other hand, when "Open" is selected, the task related to the lid is not performed, and only the task "transport the plastic bottle 81 to area G" is performed.

In the fourth embodiment, a task of opening and closing the lid of a plastic bottle is taken up as an example of realizing a desired task by assigning attributes related to the state of the object. As another example, in the case of a door, if the state of the object can be visually determined, such as "open" or "closed", it is also included in this embodiment. shall be taken as a thing.

<Fifth embodiment>
FIG. 19 is an example of a screen during sequence modification by the sequence processing device 3 in the fifth embodiment. The task displayed in FIG. 19 is a pick-and-place task in which the robot arm 90 “transports the object 92 to area G.” In this example, the purpose is to achieve a desired task by assigning attributes related to the types of objects in the action space. The situation in the example of FIG. 19 is such that when performing the pick-and-place task of the object 92, there is a possibility that the object 92 will come into contact with a large obstacle 91 located nearby. Until then, the tip of the robot arm 90 cannot be moved. Therefore, after performing the task of "moving the obstacle 91," an attribute is given to the obstacle 91 in order to perform the task of "transporting the object 92 to the area G." Specifically, by assigning an attribute of "obstacle" or "object" to the obstacle 91, the pick-and-place task is executed after the obstacle 91 is changed to a type according to the selected attribute. The attribute "obstacle" indicates that the object cannot be moved, and the attribute "object" indicates that the object is movable. The attribute information according to this embodiment may also depend on the relationship between the robot and the object.

For example, the input receiving unit 72 of the sequence processing device 3 receives input when an obstacle 91 is selected on the screen by the user. After that, an icon 94 for selecting an attribute and a window 95 showing the attributes of the obstacle 91 are displayed near the obstacle 91, and when the icon is selected on the screen, two attributes, "obstacle" or "object", can be selected. At this time, when "object" is selected, the obstacle 91 can be regarded as the object 91. This makes it possible to generate a new task of "moving the object 91". In other words, a plan can be generated to perform the task of "moving the object 91" in stages, followed by the task of "carrying the object 92 to area G". On the other hand, when "obstacle" is selected, the type of the object of the obstacle 91 remains unchanged as "obstacle", and the result is that the task of "carrying the object 92 to area G" cannot be performed.

In the various embodiments described above, by the user assigning attribute information regarding the object, attributes that cannot be specified even by a measuring device or the like can be assigned to the object. Therefore, a more flexible plan can be generated for the robot to accomplish the desired task.

In some embodiments, the information processing device or the information processing method displays a space in which the robot operates and a plurality of attribute candidates for objects or virtual objects in the space, and selects one of the plurality of attribute candidates. The attribute candidates are set as attributes of the object or virtual object.

<Other embodiments>
In addition, although the above-mentioned embodiment was explained as a hardware configuration, it is not limited to this. The present disclosure can also realize arbitrary processing by causing the CPU to execute a computer program.

In the examples above, the program may be stored and provided to the computer using various types of non-transitory computer readable media. Non-transitory computer-readable media includes various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, Includes CD-R/W, DVD (Digital Versatile Disc), and semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (Random Access Memory)). The program may also be provided to the computer on various types of transitory computer readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can provide the program to the computer via wired communication channels, such as electrical wires and fiber optics, or wireless communication channels.

Note that the present disclosure is not limited to the above embodiments, and can be modified as appropriate without departing from the spirit. Further, the present disclosure may be implemented by appropriately combining the respective embodiments.

Although the present invention has been described above with reference to the embodiments (and examples), the present invention is not limited to the above embodiments (and examples). The configuration and details of the present invention can be modified in various ways that can be understood by those skilled in the art within the scope of the present invention.

Part or all of the above embodiments may be described as in the following additional notes, but are not limited to the following.
(Additional note 1)
an input reception unit that receives input for modifying the motion sequence for the robot;
an object information acquisition unit that acquires object information representing information about an object or a virtual object in the operating space of the robot;
an attribute information acquisition unit that acquires attribute information regarding the object or the virtual object based on the input via the input reception unit;
An information processing device comprising:
(Additional note 2)
a sequence display unit that displays an operation sequence of the robot;
the input reception unit that receives an input signal for performing an operation of the robot in response to the display result;
The information processing device according to supplementary note 1, further comprising a control signal processing unit that transmits a control signal for the operation sequence to the robot based on the input.
(Additional note 3)
The information processing device according to appendix 1 or 2, further comprising an attribute signal generation unit that combines the object information and the attribute information to generate an attribute signal that is a combination of the object information and the attribute information.
(Additional note 4)
The information processing device according to any one of Supplementary Notes 1 to 3, wherein the object information acquisition unit acquires information about the object or virtual object based on the input via the input reception unit.
(Appendix 5)
The information processing device according to appendix 2, wherein the control signal processing unit converts a control signal for the operation sequence of the robot into a display signal for displaying the operation sequence, and supplies the signal to the sequence display unit.
(Appendix 6)
Supplementary notes 1 to 3, wherein the input receiving unit receives the input, generates a signal for displaying an operation on the object or virtual object, or an attribute of the object or virtual object in real time, and supplies the signal to a sequence display unit. 5. The information processing device according to any one of 5.
(Appendix 7)
The information processing device according to appendix 6, wherein the sequence display unit selectably displays a plurality of attributes of the object or virtual object when the input reception unit receives the input regarding selection of the object or virtual object. .
(Appendix 8)
8. The information processing device according to any one of appendices 1 to 7, wherein the attribute information is based on a relationship between the robot and the object or virtual object.
(Appendix 9)
The information processing device according to appendix 8, wherein the attribute information includes information indicating that the robot can pass through the virtual object and information indicating that the robot cannot pass through the virtual object.
(Appendix 10)
The attribute information includes information indicating that the robot is capable of executing a task related to the object, and information indicating that the robot is unable to execute a task related to the object, according to appendix 8. Information processing device.
(Appendix 11)
The information processing device according to appendix 8, wherein the attribute information includes information indicating that the object is movable by the robot and information indicating that the object cannot be moved by the robot.
(Appendix 12)
a sequence display section that displays the robot's operation sequence;
an input reception unit that receives input for modifying the operation sequence for the robot in response to the display result;
a control signal processing unit that transmits a control signal for the operation sequence to the robot based on the input;
an object information acquisition unit that acquires object information representing information about an object or a virtual object in the operating space of the robot;
an attribute information acquisition unit that acquires attribute information regarding the object or the virtual object based on the input via the input reception unit;
an attribute signal generation unit that combines the object information and the attribute information to generate an attribute signal that is a combination of the object information and the attribute information;
Receive the attribute signal generated by the attribute signal generation unit and having information on a combination of an object and an attribute, and generate an abstract state indicating an operation space of the robot based on the attribute signal and stored stored information. A correction system that includes an attribute information processing unit that generates correction information to be corrected.
(Appendix 13)
It is further equipped with a measuring device that measures the operating space.
an abstract state setting unit that sets an abstract state in consideration of attributes based on correction information generated by the attribute information processing unit and measurement information indicating a measurement result of the robot's motion space measured by the measurement device. , the correction system described in Appendix 12.
(Appendix 14)
accepts input for modifying the motion sequence for the robot;
Obtaining object information representing information about an object or a virtual object in the operation space of the robot,
An information processing method that obtains attribute information regarding the object or the virtual object based on the input.
(Additional note 15)
a process of accepting input for modifying a motion sequence for the robot;
a process of acquiring object information representing information about an object or a virtual object in the operating space of the robot;
a process of acquiring attribute information regarding the object or the virtual object based on the input;
A non-transitory computer-readable medium that stores a program that causes a computer to execute.
(Appendix 16)
displaying a space in which the robot operates and a plurality of attribute candidates for objects or virtual objects in the space;
An information processing device that sets a selected attribute candidate among the plurality of attribute candidates as an attribute of the object or virtual object.

1 Robot controller 2 Input device 3 Sequence processing device (correction device)
4 Storage device 5 Robot 6 Measuring device 10 Information processing device 41 Application information storage section 71 Control signal processing section 72 Input reception section 73 Object information acquisition section 74 Attribute information acquisition section 75 Attribute signal generation section 76 Sequence display section 100 Robot control system ( correction system)

Claims (10)

an input receiving unit that receives an input for modifying an operation sequence for the robot;
an attribute information acquisition unit that acquires attribute information regarding an object or a virtual object in a motion space of the robot based on the input via the input reception unit;
An information processing device comprising: The information processing device according to claim 1, further comprising an object information acquisition unit that acquires object information representing information about an object or a virtual object in the operating space of the robot. a sequence display unit that displays an operation sequence of the robot;
the input reception unit that receives an input signal for performing an operation of the robot in response to the display result;
The information processing apparatus according to claim 1, further comprising a control signal processing unit that transmits a control signal for the operation sequence to the robot based on the input. The information processing device according to claim 2, further comprising an attribute signal generation unit that combines the object information and the attribute information to generate an attribute signal that is a combination of the object information and the attribute information. The information processing device according to claim 2 , wherein the object information acquisition unit acquires information about the object or virtual object based on the input via the input acceptance unit. 4. The information processing apparatus according to claim 3 , wherein the control signal processing section converts the control signal of the motion sequence of the robot into a display signal for displaying the motion sequence, and supplies the signal to the sequence display section. accepts input for modifying the motion sequence for the robot;
An information processing method that obtains attribute information regarding an object or a virtual object in an operation space of the robot based on the input. 8. The information processing method according to claim 7, wherein object information representing information about an object or a virtual object in the operating space of the robot is acquired. a process of accepting input for modifying a motion sequence for the robot;
A process of acquiring attribute information regarding an object or a virtual object in the operation space of the robot based on the input;
A program that causes a computer to execute. 10. The program according to claim 9 , which causes a computer to execute a process of acquiring object information representing information about an object or a virtual object in the operating space of the robot .

Images