WO2022080007A1 - Server, control method therefor, program, and system - Google Patents

Server, control method therefor, program, and system Download PDF

Info

Publication number
WO2022080007A1
WO2022080007A1 PCT/JP2021/030433 JP2021030433W WO2022080007A1 WO 2022080007 A1 WO2022080007 A1 WO 2022080007A1 JP 2021030433 W JP2021030433 W JP 2021030433W WO 2022080007 A1 WO2022080007 A1 WO 2022080007A1
Authority
WO
WIPO (PCT)
Prior art keywords
support
priority
operator
information
task
Prior art date
Application number
PCT/JP2021/030433
Other languages
French (fr)
Japanese (ja)
Inventor
義弘 坂本
亮輔 川西
Original Assignee
東京ロボティクス株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 東京ロボティクス株式会社 filed Critical 東京ロボティクス株式会社
Publication of WO2022080007A1 publication Critical patent/WO2022080007A1/en

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B19/00—Programme-control systems
    • G05B19/02—Programme-control systems electric
    • G05B19/418—Total factory control, i.e. centrally controlling a plurality of machines, e.g. direct or distributed numerical control [DNC], flexible manufacturing systems [FMS], integrated manufacturing systems [IMS], computer integrated manufacturing [CIM]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Definitions

  • the present invention relates to a semi-autonomous robot system that involves remote control by an operator.
  • Automation using robots is being promoted at production sites and distribution warehouses.
  • Document 1 discloses a robot control method for switching robot control from autonomous control to manual control when task execution fails.
  • the present invention has been made in view of the above-mentioned technical background, and an object of the present invention is to provide a technique in which an operator can easily deal with a support request even when a plurality of support requests are made by a robot. To provide.
  • the server according to the present invention is a server connected to a plurality of robot systems that execute predetermined tasks semi-autonomously, one or a plurality of operator terminals, and receives a support request from the robot system.
  • the support request receiving unit, and the priority determination unit that determines the priority of the support request when a plurality of the support requests are received, and the operator terminal, the information regarding the support request and the information regarding the priority. It is equipped with an information provision department, which provides information.
  • the support means that the semi-autonomous robot is operated or instructed for the purpose of assisting the execution of the target motion.
  • the support request means the information requesting the operator for such assistance.
  • the priority determination unit further calculates a first total value, which is the total value of the time required for the subsequent tasks of the task executed by the robot system, for each of the robot systems for which the support request is made.
  • a time calculation unit and a required time-based priority determination unit that determines the priority so that the larger the first total value is, the higher the priority may be.
  • the priority determination unit further sets the remaining required time of the task being executed to the total value of the required time of the task succeeding the task executed by the robot system for each of the robot systems for which the support request is made.
  • a second required time calculation unit that calculates the added second total value, and a second required time-based priority that determines the priority so that the larger the second total value is, the higher the priority is. It may be provided with a ranking determination unit.
  • the operator can deal with tasks that have a large impact on the entire process. Further, since the remaining required time of the task being executed is also taken into consideration, when there is a task or the like having a long execution time, the priority can be determined more precisely.
  • the priority determination unit adds the total time required for the task succeeding the task executed by the robot system to the total time required for the task being executed, and the remaining time required for the task being executed, for each robot system for which the support request is made.
  • the third total value is calculated by adding the required time of the required support operation, and the third required time calculation unit, and the larger the third total value, the higher the priority.
  • a third time-based priority determination unit which determines the above, may be provided.
  • the operator can deal with tasks that have a large impact on the entire process.
  • the remaining time required for the task being executed and the time required for the required support operation are also taken into consideration, so if there is a task with a long execution time or a task that requires time for support, prioritize it more precisely. Can be decided.
  • Any of the required times may be determined based on the actually measured value measured by operating the robot system.
  • the required time can be estimated accurately based on the measured value.
  • the server further includes, for each operator corresponding to each operator terminal, a support authority storage unit that stores support authority for the robot system, the task, and / or a part of operations constituting the task.
  • the information providing unit may provide information regarding the support request to the operator terminal based on the support authority.
  • support requests can be efficiently assigned to the operator terminal based on the presence or absence of support authority.
  • the server further calculates the sum of the first total value, the second total value, or the third total value for each combination of the support request and the operator corresponding to the operator terminal.
  • the information providing unit may further include a calculation unit, and further provide information regarding the support request to the operator terminal based on the combination that minimizes the total sum.
  • the system as a whole can be optimized and the downtime of the entire process can be minimized.
  • the server further includes, for each operator corresponding to each operator terminal, a support authority storage unit that stores support authority for the robot system, the task, and / or a part of operations constituting the task. For each combination of the support request and the operator corresponding to the operator terminal having the support authority for the support request, the sum of the first total value, the second total value, or the third total value is calculated.
  • the information providing unit may further provide information regarding the support request to the operator terminal based on the combination that minimizes the total.
  • the priority may be determined so that the older the time when the support request is received from the robot system, the higher the priority.
  • the server includes a priority information storage unit that stores predetermined priority information about the robot system, the task, and / or an operation constituting a part of the task, and the priority is the priority. It may be informed.
  • the operator terminal includes a list display unit that displays a list of the connected robot systems, and a highlighting unit that highlights the robot system making the support request based on the information regarding the support request. , May be provided.
  • the operator terminal may further include a sort display unit that sorts and displays the robot system requesting support according to the priority based on the information regarding the priority.
  • the server further manages account information, including support authority for the robot system, the task, and / or some operations constituting the task, which are set for each operator of the operator terminal.
  • the operator terminal may further include a login processing unit that provides a login process that provides collation with the account information to an operator who uses the operator terminal.
  • the same operator terminal can be used by different operators with their own accounts.
  • the operator terminal further displays, among the robot systems requesting support, the robot system for which the operator who has performed the login process has the support authority so as to be distinguishable from other robot systems. It may be provided with a display unit.
  • the logged-in operator can immediately grasp the robot system that he / she can support based on the support authority.
  • the robot system may include a mobile manipulator.
  • the priority can be determined for the support requests, so that the operator terminal can use the configuration.
  • the operator who is the operator, can easily deal with a plurality of support requests.
  • the present invention can also be thought of as a system. That is, the system according to the present invention is a system including a plurality of robot systems that semi-autonomously execute predetermined tasks, one or a plurality of operator terminals, the robot system, and a server connected to the operator terminals.
  • the server has a support request receiving unit that receives a support request from the robot system, and a priority determination unit that determines the priority of the support request when a plurality of the support requests are received.
  • the operator terminal is provided with an information providing unit that provides information on the support request and information on the priority.
  • the present invention can also be considered as a method. That is, the method according to the present invention is a control method for a plurality of robot systems that semi-autonomously execute predetermined tasks, one or a plurality of operator terminals, and a connected server, and the robot system requests support.
  • Information provision steps and information provision steps that provide information about rankings, It is equipped with.
  • the present invention can also be thought of as a computer program. That is, the program according to the present invention is a control program of a server connected to a plurality of robot systems that semi-autonomously execute predetermined tasks, one or a plurality of operator terminals, and a support request is made from the robot system.
  • a system or the like in which an operator can easily deal with a plurality of support requests even when a plurality of support requests are received from a robot system that executes a task semi-autonomously. can do.
  • FIG. 1 is an overall configuration diagram of the system.
  • FIG. 2 is a functional block diagram of the system.
  • FIG. 3 is a table showing the correspondence between the robot system ID and the task.
  • FIG. 4 is a table showing the correspondence between the task, the action which is a unit operation constituting the task, and the required time of each action.
  • FIG. 5 is a table showing the correspondence between the operator ID and the authority given to the operator.
  • FIG. 6 is a flowchart illustrating the operation of the robot system.
  • FIG. 7 is a flowchart relating to the information collection operation of the management server.
  • FIG. 8 is a flowchart relating to the process of providing information to the operator system.
  • FIG. 9 is a flowchart relating to the detailed operation of the priority determination process.
  • FIG. 10 is an arrow diagram relating to a series of tasks.
  • FIG. 11 is a display example of the operation status list screen.
  • FIG. 12 is an operation flowchart relating to the support process.
  • the robot system is applied to a distribution warehouse system in the present embodiment, it is not limited to such an application example, and can be applied to various other situations such as factories. Further, in the present embodiment, the robot system will be described as including the robot arm, but various hardware may be adopted in addition to the robot arm. The robot system does not have to be fixed, and for example, a mobile manipulator or the like can be adopted. Further, the task of the robot system is not limited to the above-mentioned one, and can be applied to various other tasks.
  • FIG. 1 is an overall configuration diagram of the system 200 according to the present embodiment.
  • the system 200 includes a plurality of robot systems 1, a management server 3, and an operator system 5, which are connected to each other via a LAN (local area network) by wire or wirelessly. ing.
  • LAN local area network
  • the devices are connected by LAN, but they may be connected via the Internet.
  • Each robot system 1 includes a robot-side information processing device 11 that performs various information processing described later, a sensor 15 connected to the robot-side information processing device 11, and an articulated robot arm connected to the robot-side information processing device 11. It is composed of 17.
  • the robot-side information processing device 11, the sensor 15, and the robot arm 17 are described as constituting one system while being physically separated from each other in the present embodiment, a part or all of them are described. May be integrally configured.
  • the robot-side information processing device 11 includes a control unit composed of a CPU and the like for executing various programs, and a storage unit composed of a ROM, RAM, a hard disk, and the like.
  • the sensor 15 is a three-dimensional camera fixed to an environment for acquiring image information that is the basis of recognition. For example, when performing an article selection task, a box containing a large number of target products and other workpieces. The inside is imaged, and the work on the conveyor is imaged when performing the boxing task. The information acquired from the sensor is transmitted to the information processing device 11 on the robot side.
  • the three-dimensional camera which is the sensor 15, is fixed in the environment, and as will be described later, according to the configuration shared for both recognition and remote control, the same viewpoint can be used, so that the configuration is simplified. At the same time, it is advantageous in terms of cost.
  • the method of attaching the sensor 15 is not limited to the example of fixing the sensor 15 in the environment in this way, and a hand-eye camera attached to the robot arm 17 or the like may be used, or an existing surveillance camera may be used. ..
  • the articulated robot arm 17 has a gripper at the tip and can move freely. For example, when performing an article sorting task, an operation of sandwiching a work and placing it on a belt conveyor or the like is performed. Further, in the boxing / tape closing task, the work placed on the belt conveyor is gripped and moved, and the tape closing operation is performed.
  • the management server 3 appropriately collects information on the robot system 1 and information on the operator system 5, as will be described later. Further, various information is provided to the robot system 1 and the operator system 5 as needed.
  • the operator system 5 includes an operator-side information processing device 51, a display device 52, and a haptic input device 53.
  • the operator-side information processing device 51 is an information processing device that controls the entire operator system 5, and is, for example, a PC (personal computer).
  • the display device 52 is a display device, and displays a list or the like of the robot system 1 that has issued a support request, as will be described later.
  • the haptic input device 53 is provided with an angle sensor for each joint, and can control the 6-axis posture of the hand of the robot arm 17 and the opening / closing of the gripper. As will be described later, the haptic input device 53 is operated by an operator to support task execution of the robot arm 17 in response to a support request.
  • the haptic input device 53 may be one in which force feedback is performed based on the sensor provided in the robot arm 17.
  • the display device and the input device are not limited to the examples of the present embodiment, and various known devices can be used.
  • the display device may be a head-mounted display
  • the input device may be an input device using a stick lever, a keyboard, or the like.
  • the operator-side information processing device 51, the display device 52, and the haptic input device 53 are described as being physically separated from each other to form one system, but a part thereof. Alternatively, the whole may be integrally configured.
  • FIG. 2 is a functional block diagram of the system 200. As is clear from the figure, the robot system 1 exchanges information with the management server 3 and the operator system 5 via the respective transmission / reception units.
  • the robot system 1 includes a robot-side information processing device 11, a sensor 15, and a robot arm 17.
  • the robot-side information processing device 11 includes a sensor information acquisition unit 111 that acquires detection information in the sensor 15, and a recognition processing unit 112 that performs predetermined recognition processing based on the sensor information.
  • the recognition result by the recognition processing unit 112 is provided to the determination unit 114.
  • the determination unit 114 determines whether or not to make a support request described later based on the recognition result, and provides the determination result to the motion generation unit 118.
  • the motion generation unit 118 and the motion command unit 119 issue motion commands to the robot arm 17.
  • the support process execution unit 121 When the support process execution unit 121 receives the support information via the transmission / reception unit 127, the support process execution unit 121 executes various support processes according to the support information. For example, the operation of the robot arm 17 is controlled.
  • the management server 3 includes a support request detection unit 32 and an information acquisition processing unit 33.
  • the support request detection unit 32 detects the support request transmitted from the robot system 1. Further, when a plurality of support requests are received, the priority determination processing unit 35 determines the priority of the support request.
  • the information provision processing unit 36 transmits information regarding support requests and information regarding priorities to the operator system 5 via the transmission / reception unit.
  • the information acquisition processing unit 33 acquires status information and the like of the robot system 1 and the operator system 5.
  • the acquired information is stored in a storage unit (not shown) by the storage processing unit 38.
  • the list control unit 37 manages predetermined list information described later based on the status information and the like.
  • table information stored in advance as setting information in the management server 3 will be described.
  • the present invention is not limited to such a configuration and may be set via the operator system 5 or the like during operation.
  • FIG. 3 is a table showing the correspondence between the robot system ID and the task stored in advance in the storage unit of the management server 3.
  • identification information is given to each robot system 1 such as "robot01” and "robot02", and the identification information of the task to be executed corresponding to the identification information is, for example, "TASK_A”.
  • FIG. 4 is a table showing the correspondence relationship between the task stored in advance in the storage unit of the management server 3, the action which is a unit operation constituting the task, and the required time ([s]) of each action. ..
  • “TASK_A” is composed of seven actions from “ACTION_01” to "ACTION_07”, and each action is associated with the required time shown in the figure.
  • FIG. 5 is a table showing the correspondence between the operator ID, which is the identification information of the operator, stored in advance in the storage unit of the management server 3 and the authority given to the operator.
  • the operator ID “Sakamoto” is given support authority for the two actions “TASK_A” and “TASK_B”. Further, the operator ID “Kawanishi” is given only the support authority for "TASK_B”. However, “Sakamoto” is not granted support authority for "robot03".
  • the support authority is defined in units of the task and the robot ID in this embodiment, it may be defined by another method. Therefore, the support authority may be defined by other means such as an action unit.
  • system configuration in this embodiment is an example, and the number and configuration of each device connected to the network can be freely changed.
  • FIG. 6 is a flowchart illustrating the operation of the robot system 1.
  • the robot system 1 executes a target motion or a task semi-autonomously, and sends a support request to the management server 3 when an appropriate recognition cannot be performed or the target motion fails.
  • the sensor information acquisition unit 111 acquires the sensor detection information from the sensor 15, and the image information obtained by photographing the inside of the box or the conveyor in the present embodiment.
  • the recognition processing unit 112 performs a process of recognizing a work or the like to be gripped by the robot arm 17 based on the sensor information (S2).
  • the recognition is performed based on the image information, but the present invention is not limited to such a configuration. Therefore, for example, the configuration may be such that the three-dimensional information acquired by the three-dimensional camera is used instead of the image or together with the image.
  • the determination unit 114 performs a process of determining the necessity of support (S4). More specifically, the determination unit 114 does not need support when recognition by the recognition unit 112 is normally performed, for example, when the work to be gripped can be recognized with sufficient recognition accuracy. On the other hand, if the recognition is not performed normally, it is determined that support is required.
  • the necessity of support is determined based on whether or not normal recognition is possible, but the present invention is not limited to such a configuration, and various determination methods can be adopted.
  • the determination process may be performed based on a criterion such as whether or not a gripping solution for the gripping object can be calculated.
  • the motion generation unit 118 generates a target motion (S6).
  • the target motion is an motion expected to be executed by the robot system 1 such as a task motion set for the robot system 1.
  • the motion command unit 119 issues an motion command to the robot arm 17 so as to execute the generated target motion, and executes the target motion (S7).
  • support means to perform operations and instructions to the semi-autonomous robot for the purpose of assisting the execution of the target movement.
  • the support request means the information requesting the operator for such assistance.
  • the support request from the robot system 1 may be referred to as an alert.
  • the support request transmission process is performed to the management server 3 (S11). After that, the information processing apparatus 11 on the robot side goes into a standby state until the support information is received (S12NO).
  • the support processing execution unit 121 When the support information is received in this state (S12YES), the support processing execution unit 121 performs the support execution processing according to the support information (S14). After that, a predetermined end determination is performed, and if the end signal is not detected, the reception standby state (S15NO) is set again and a series of processes are repeated. On the other hand, when the end signal is detected (S15YES), the above-mentioned end determination process (S9) is performed.
  • FIG. 7 is a flowchart relating to the information collection operation of the management server 3 that acquires various information from the robot system 1 and the operator system 5.
  • the information acquisition processing unit 33 has four status information (S21), a support request (S31), a support completion signal (S41), and an operator information (S51) of the robot system 1.
  • the processes for acquiring information are executed in parallel, and each process is in a standby state until the corresponding information is acquired (S22NO, S32NO, S42NO, S52NO).
  • the status information of the robot system 1 is information representing the current operating state of the robot system 1, and is detected by, for example, a task being executed, an action, an elapsed time from the start of the operation, joint angle information of the robot arm 17, and a sensor 15. Includes information such as sensor information.
  • the support request is information for requesting support by the operator transmitted from the robot system 1 by the above processing, and includes, for example, information such as the ID of the robot system 1 that made the support request and the time when the support request was generated.
  • the support completion signal is information transmitted from the operator system 5 when the support by the operator is completed, and includes, for example, the support completion time.
  • the operator information is information about an operator associated with the operator by logging in to the information processing apparatus 51 on the operator side, and includes, for example, information such as an operator ID.
  • the login process is performed by collating with the account information stored by the management server 3.
  • the same operator system 5 can be shared by different operators.
  • the storage processing unit 38 When the storage processing unit 38 receives the status information from the robot system 1 (S22YES), the storage processing unit 38 performs a process of storing the status information (S23), and is in the standby state again.
  • the list control unit 37 When the list control unit 37 receives the support request from the robot system 1 (S32YES), the list control unit 37 performs a process of adding the ID information of the robot system 1 that made the support request to the predetermined support request list (S33). At this time, the ID information of the robot system 1 for which the support is requested by the storage processing unit 38 is stored in the storage unit. After that, it goes into the standby state again.
  • the list control unit 37 when the list control unit 37 receives the support completion signal (S42YES), the list control unit 37 performs a process of deleting the ID information of the robot system 1 for which support has been completed from the predetermined support request list (S43). At this time, the ID information of the robot system 1 for which the support is requested by the storage processing unit 38 is deleted from the storage unit. After that, it goes into the standby state again.
  • the storage processing unit 38 receives the operator information from the operator system 5 (S52YES), the storage processing unit 38 performs a process of storing the operator information (S53), and is in the standby state again.
  • the management server 3 collects, acquires, stores, and manages various information transmitted from the robot system 1 and the operator system 5.
  • FIG. 8 is a flowchart relating to the process of providing information to the operator system 5.
  • the flowchart on the left side of the figure shows the operation of the operator system 5, and the flowchart on the right side of the figure shows the operation of the management server 3.
  • the display processing unit 57 of the information processing apparatus 51 on the operator side makes a display request to the management server 3 (S61). After this display request, the information processing apparatus 51 on the operator side goes into a standby state for predetermined information (S62NO). When the predetermined information is received (S62YES), the display process corresponding to the information is performed on the display unit 52 (S64). After that, the series of processes is repeated again (S61 to S64). That is, the information processing device 51 on the operator side is constantly performing update display processing.
  • the information providing processing unit 36 provides the operator system 5 with information for presenting the support request to the operator system 5. Processing is performed (S75).
  • the provided information may include any information collected by the management server 3.
  • the priority determination processing unit 35 performs a process (S76) for determining the priority among the support requests, and after this process, a predetermined information provision process together with the priority. (S78).
  • the provided information may include any information collected by the management server 3.
  • FIG. 9 is a flowchart relating to the detailed operation of the priority determination process. As is clear from the figure, when the process starts, the variable is initialized and the initialization process is performed with reference to the specific robot system 1 (S81).
  • a process of calculating the total required time of the task following the task performed by the referenced robot system 1 is performed (S83). More specifically, the total required time is calculated based on the above-mentioned table showing the correspondence between the robot system ID and the task and the table showing the correspondence between the task and its required time ([s]). calculate.
  • FIG. 10 is an arrow diagram relating to a series of tasks according to the present embodiment.
  • the article sorting task (task A) is executed by "robot01”, the sorted objects are placed and conveyed on different conveyors as three parallel processes. ..
  • the articles conveyed by the conveyor are boxed by the subsequent "robot02", “robot03” and “robot04", respectively, and the boxes are tape-closed.
  • each robot system 1 places the tape-closed box on a conveyor and conveys it (task B).
  • FIG. 11 is a display example of the operation status list screen displayed on the display unit 52 of the operator system 5 as a result of the display process (S64).
  • the operation status list display unit 80 is provided in the center of the screen, and the ID information "Sakamoto" of the operator currently using the operator system 5 is displayed in the upper right of the screen. The time is displayed.
  • the alert generation state is a state in which the robot system 1 is requesting support from the operator because the target operation cannot be executed or is unlikely to be executed.
  • “Trouble occurrence time” is information about the date and time when the support request occurred.
  • the “robot ID” is ID information which is identification information of the robot system 1 corresponding to the support request.
  • the "task” is a task assigned to each robot system 1, and in the example of the figure, the "article selection” task (task A) or the “boxing / tape closing” task (boxing / tape closing) for the robot (task A).
  • One of the tasks in task B) is assigned.
  • the “assistant” displays the ID of the operator who is providing support. For example, in the example of the figure, it can be seen that the operator "Kawanishi” provides support for "robot07".
  • “Assistance status” indicates whether or not support is being provided in response to a support request. For example, in the example of the figure, it is shown that the operator "Kawanishi” is responding to the support request from "robot07".
  • the support request is displayed as "robot01”, “robot04”, “robot07”, and “robot03" from the top in order from the highest priority of the response based on the calculated priority. Has been done. That is, the robot system 1 requesting support is sorted and displayed so as to be distinguishable from the robot system 1 not requesting support.
  • the method for displaying the robot system 1 requesting support in a distinctive manner is not limited to such an example, and various other display methods can be adopted. For example, only the robot system 1 requesting support may be displayed, or the display related to the robot system 1 requesting support may be colored to be highlighted.
  • FIG. 12 is an operation flowchart relating to the support process performed by the operator.
  • the flowchart on the left side of the figure is an operation flowchart of the operator system 5, and the flowchart on the right side of the figure is an operation flowchart of the management server 3.
  • the information processing apparatus 51 on the operator side performs a process of detecting whether or not an input for selecting one of the support requests displayed in the operation status list has been made (S101). If no input is detected as a result of this detection process (S102NO), the input detection process is repeated (S101, S102NO).
  • the management server 3 when the management server 3 has been in the standby state (S111NO) until the support request is received (S111YES), the management server 3 performs a process of setting the operator who sent the support request as the operator in charge in association with the support request. (S112). As a result, for example, in the "assistant” and “assistance status” columns of FIG. 11, the corresponding operator ID and "corresponding" are displayed. After that, the management server 3 transmits a notification indicating that the support request has been set to the operator system 5 (S114).
  • the operator system 5 When the operator system 5 receives the notification (S105YES), the operator system 5 performs a process of displaying the support screen.
  • FIG. 13 is a display example of a support screen that enables the operator to respond to a support request.
  • a status display unit 83 related to the selected support request being handled is provided in the upper center of the screen, and the ID information of the operator currently using the operator system 5 is provided in the upper right of the screen. "Sakamoto" and the current time are displayed.
  • a camera view display area 83 for displaying the camera view captured by the camera 15 of the robot system 1 is provided.
  • the image of the stacked work 92 in the box 91 which is the target of the article selection task, is displayed in the camera view display area 83.
  • the work 92 has a cylindrical shape, is gripped by the robot arm 17, and is sorted according to a predetermined standard.
  • the present invention is not limited to such a configuration. Therefore, for example, the configuration may be such that the three-dimensional information acquired by the three-dimensional camera is presented in place of the image or together with the image.
  • a CG image display unit that displays a CG (Computer Graphics) image 841 generated based on posture information obtained from a joint angle sensor (not shown) provided on the robot arm 17 in the current posture. 84 is provided.
  • the image display unit allows the operator to instantly grasp the current posture of the robot arm 17.
  • the work start button 85 for starting the support process is arranged.
  • the content of the support process is an operation of remotely controlling the robot arm 17 using the haptic input device 53 while checking the camera view display area 83 to change the execution condition of the target operation. ..
  • the operation of changing the execution condition of the target operation is, for example, an operation of changing the scene or the environmental condition, and more specifically, the robot arm 17 is used to shake the box or the inside of the box by remote control by the operator. This is an operation of changing the position and posture of the work 92 by gripping and moving the work 92.
  • the content of the support process is an operation of changing the execution condition of the target operation, but the content is not limited to this. Therefore, for example, the target operation itself may be performed as an alternative by remote control.
  • the support completion signal is transmitted to the management server 3, and then the input detection process (S101) repeats a series of processes.
  • the support completion signal is acquired by the management server 3 and erased from the corresponding list of support requests (S41 to S43).
  • the operator can deal with tasks having a large influence on the entire process based on the required time, so that the entire system can be operated efficiently.
  • the priority may be determined based on a value obtained by further adding the required time of the remaining actions of the referenced robot system 1 to the total required time. According to such a configuration, the time required for the remaining actions of the task being executed is also taken into consideration, so that even if there is an action having a long execution time, the priority can be determined more precisely.
  • the priority may be determined based on a value obtained by further adding the required time expected to be required for the support to the robot system 1 to the total required time. According to such a configuration, even if it takes a certain amount of time for support, the priority can be determined more precisely.
  • the operator is configured to display all the robot systems 1 requesting support, but the present invention is not limited to such a configuration. Therefore, for example, the combination of the robot system 1 for which the support is requested and the operator system 5 may be optimized.
  • the time required for the entire process is calculated for all combinations of the robot system 1 and the operator system 5 for which support is requested, and the combination with the shortest required time is recommended or provided to the operator. You may. With such a configuration, optimization can be achieved for the entire system, and downtime for the entire process can be minimized.
  • the required time of the entire process is calculated for all combinations of the robot system 1 that has made a support request and the operator system 5 that has the support authority for the support request, and the required time is the longest.
  • the minimum combination may be recommended or provided to the operator.
  • the priority is determined based on the required time of the succeeding task and the like, but it may be simply displayed on the display terminal 52 in chronological order in which the support request is generated. With such a configuration, it is possible to provide optimal support for a system in which processing in the order of occurrence is desirable.
  • the present invention can be used at least in an industry that manufactures robot systems and the like.
  • Robot system 11 Robot side information processing device 15 Sensor 17 Robot arm 3 Server 5 Operator system 51 Operator side information processing device 52 Display device 53 Haptic input device

Abstract

Provided is a server connecting a plurality of robot systems that semi-autonomously perform prescribed tasks and one or more operator terminals, said server comprising: a support request reception unit which receives support requests from the robot systems; a priority determination unit which determines the priority of the support requests when a plurality of support requests have been received; and an information provision unit which provides, to the one or more operator terminals, information pertaining to the support requests and information pertaining to the priority.

Description

サーバ、その制御方法及びプログラム並びにシステムServers, their control methods and programs, and systems
　この発明は、オペレータによる遠隔操作等を伴う半自律ロボットシステムに関する。 The present invention relates to a semi-autonomous robot system that involves remote control by an operator.
　生産現場や物流倉庫等において、ロボットを用いた自動化が推進されている。 Automation using robots is being promoted at production sites and distribution warehouses.
　この種の自動化システムにおいては、本来全てのロボットが自律的に動作することが望ましい。しかしながら、ハードウェア的な制約や想定外の事態への対応等により、現実には、すべての動作を自律的に行わせることは困難である。 In this kind of automation system, it is desirable that all robots operate autonomously. However, in reality, it is difficult to autonomously perform all operations due to hardware restrictions and response to unexpected situations.
　そこで、近年、大部分の動作を自動化しつつも、ロボットが対応できない場面のみ、オペレータが遠隔操作によりロボットの動作に一時的に介入して、ロボット動作をサポートする半自律システムが検討されている。 Therefore, in recent years, a semi-autonomous system that supports robot movements by the operator temporarily intervening in the robot movements by remote control is being studied only in situations where the robot cannot respond while automating most of the movements. ..
　例えば、文献１には、タスク実行に失敗した場合に、自律制御から手動制御にロボットの制御を切り替えるロボットの制御方法が開示されている。 For example, Document 1 discloses a robot control method for switching robot control from autonomous control to manual control when task execution fails.
特開２０１６－０６８１６１号公報Japanese Unexamined Patent Publication No. 2016-066161
　しかしながら、従前のこの種の技術において、オペレータによるサポートが行われる場合、オペレータとロボットとは１対１の関係を前提としており、オペレータとロボットとの関係が１：Ｎ又はＭ：Ｎの関係（Ｎ及びＭは２以上の自然数）の場合については十分な検討がなされていなかった。そのため、複数のロボットから同時に複数のサポート要請がなされた場合等には、オペレータによるロボットのサポートが困難となる虞があった。 However, in the conventional technique of this kind, when the support by the operator is performed, it is premised that the operator and the robot have a one-to-one relationship, and the relationship between the operator and the robot is a 1: N or M: N relationship ( N and M are natural numbers of 2 or more), but not enough studies have been made. Therefore, when a plurality of support requests are made from a plurality of robots at the same time, it may be difficult for the operator to support the robots.
　本発明は上述の技術的背景に鑑みてなされたものであり、その目的は、ロボットから複数のサポート要請がなされる場合であっても、オペレータが容易にサポート要請に対処することができる技術を提供することにある。 The present invention has been made in view of the above-mentioned technical background, and an object of the present invention is to provide a technique in which an operator can easily deal with a support request even when a plurality of support requests are made by a robot. To provide.
　上述の技術的課題は、以下の構成を有するサーバ等により解決することができる。 The above technical problem can be solved by a server or the like having the following configuration.
　すなわち、本発明に係るサーバは、所定のタスクを半自律的に実行する複数のロボットシステムと、１又は複数のオペレータ端末と、接続されたサーバであって、前記ロボットシステムからサポート要請を受信する、サポート要請受信部と、複数の前記サポート要請を受信した場合、前記サポート要請の優先順位を決定する、優先順位決定部と、前記オペレータ端末へと、前記サポート要請に関する情報と前記優先順位に関する情報を提供する、情報提供部と、を備えている。 That is, the server according to the present invention is a server connected to a plurality of robot systems that execute predetermined tasks semi-autonomously, one or a plurality of operator terminals, and receives a support request from the robot system. , The support request receiving unit, and the priority determination unit that determines the priority of the support request when a plurality of the support requests are received, and the operator terminal, the information regarding the support request and the information regarding the priority. It is equipped with an information provision department, which provides information.
　このような構成によれば、半自律的にタスクを実行するロボットシステムから複数のサポート要請を受信した場合であっても、サポート要請に対して優先順位を決定することができるので、オペレータは複数のサポート要請に対して容易に対処することができる。なお、ここで、サポートとは、半自律ロボットに対して目標動作の実行補助を目的として操作や指示等を行うことを意味する。また、サポート要請とは、そのような補助をオペレータに対して要求する情報を意味する。 With such a configuration, even when a plurality of support requests are received from a robot system that executes a task semi-autonomously, the priority can be determined for the support requests, so that a plurality of operators can be used. You can easily deal with the support request. Here, the support means that the semi-autonomous robot is operated or instructed for the purpose of assisting the execution of the target motion. Also, the support request means the information requesting the operator for such assistance.
　前記優先順位決定部は、さらに、前記サポート要請を行った前記ロボットシステム毎に、当該ロボットシステムが実行するタスクの後続のタスクの所要時間の合計値である第１の合計値を算出する、所要時間算出部と、前記第１の合計値が大きい程、前記優先順位が高くなるよう優先順位を決定する、所要時間ベース優先順位決定部と、を備えてもよい。 The priority determination unit further calculates a first total value, which is the total value of the time required for the subsequent tasks of the task executed by the robot system, for each of the robot systems for which the support request is made. A time calculation unit and a required time-based priority determination unit that determines the priority so that the larger the first total value is, the higher the priority may be.
　このような構成によれば、オペレータは、工程全体への影響の大きいタスクから対処することができる。 With such a configuration, the operator can deal with tasks that have a large impact on the entire process.
　前記優先順位決定部は、さらに、前記サポート要請を行った前記ロボットシステム毎に、当該ロボットシステムが実行するタスクの後続のタスクの所要時間の合計値に、実行中のタスクの残りの所要時間を加えた第２の合計値を算出する、第２の所要時間算出部と、前記第２の合計値が大きい程、前記優先順位が高くなるよう優先順位を決定する、第２の所要時間ベース優先順位決定部と、を備えてもよい。 The priority determination unit further sets the remaining required time of the task being executed to the total value of the required time of the task succeeding the task executed by the robot system for each of the robot systems for which the support request is made. A second required time calculation unit that calculates the added second total value, and a second required time-based priority that determines the priority so that the larger the second total value is, the higher the priority is. It may be provided with a ranking determination unit.
　このような構成によれば、オペレータは、工程全体への影響の大きいタスクから対処することができる。また、実行中のタスクの残りの所要時間も加味するので、実行時間の大きいタスク等が存在する場合には、より精緻に優先順位を決定することができる。 With such a configuration, the operator can deal with tasks that have a large impact on the entire process. Further, since the remaining required time of the task being executed is also taken into consideration, when there is a task or the like having a long execution time, the priority can be determined more precisely.
　前記優先順位決定部は、さらに、前記サポート要請を行った前記ロボットシステム毎に、当該ロボットシステムが実行するタスクの後続のタスクの所要時間の合計値に、実行中のタスクの残りの所要時間と、必要なサポート動作の所要時間と、を加えた第３の合計値を算出する、第３の所要時間算出部と、前記第３の合計値が大きい程、前記優先順位が高くなるよう優先順位を決定する、第３の所要時間ベース優先順位決定部と、を備えてもよい。 Further, the priority determination unit adds the total time required for the task succeeding the task executed by the robot system to the total time required for the task being executed, and the remaining time required for the task being executed, for each robot system for which the support request is made. , The third total value is calculated by adding the required time of the required support operation, and the third required time calculation unit, and the larger the third total value, the higher the priority. A third time-based priority determination unit, which determines the above, may be provided.
　このような構成によれば、オペレータは、工程全体への影響の大きいタスクから対処することができる。また、実行中のタスクの残りの所要時間と必要なサポート動作の所要時間も加味するので、実行時間の大きいタスクやサポートに時間を要するタスク等が存在する場合には、より精緻に優先順位を決定することができる。 With such a configuration, the operator can deal with tasks that have a large impact on the entire process. In addition, the remaining time required for the task being executed and the time required for the required support operation are also taken into consideration, so if there is a task with a long execution time or a task that requires time for support, prioritize it more precisely. Can be decided.
　いずれの前記所要時間も前記ロボットシステムを運用することにより計測された実測値に基づいて決定されるものであってもよい。 Any of the required times may be determined based on the actually measured value measured by operating the robot system.
　このような構成によれば、所要時間を実測値に基づいて正確に見積もることができる。 With such a configuration, the required time can be estimated accurately based on the measured value.
　前記サーバは、さらに、各前記オペレータ端末に対応するオペレータ毎に、前記ロボットシステム、前記タスク及び／又は前記タスクを構成する一部の動作に対するサポート権限を記憶する、サポート権限記憶部を備え、前記情報提供部は、前記サポート権限に基づいて、前記オペレータ端末へと前記サポート要請に関する情報を提供する、ものであってもよい。 The server further includes, for each operator corresponding to each operator terminal, a support authority storage unit that stores support authority for the robot system, the task, and / or a part of operations constituting the task. The information providing unit may provide information regarding the support request to the operator terminal based on the support authority.
　このような構成によれば、サポート権限の有無に基づき、オペレータ端末に対してサポート要請を効率よく割り当てることができる。 According to such a configuration, support requests can be efficiently assigned to the operator terminal based on the presence or absence of support authority.
　前記サーバは、さらに、前記サポート要請と前記オペレータ端末に対応するオペレータとの各組み合わせについて、前記第１の合計値、前記第２の合計値又は前記第３の合計値の総和を算出する、総和算出部を備え、前記情報提供部は、さらに、前記総和が最小となる組み合わせに基づいて、前記オペレータ端末へと前記サポート要請に関する情報を提供する、ものであってもよい。 The server further calculates the sum of the first total value, the second total value, or the third total value for each combination of the support request and the operator corresponding to the operator terminal. The information providing unit may further include a calculation unit, and further provide information regarding the support request to the operator terminal based on the combination that minimizes the total sum.
　このような構成によれば、システム全体として最適化を図ることができ、工程全体のダウンタイムを最小化することができる。 With such a configuration, the system as a whole can be optimized and the downtime of the entire process can be minimized.
　前記サーバは、さらに、各前記オペレータ端末に対応するオペレータ毎に、前記ロボットシステム、前記タスク及び／又は前記タスクを構成する一部の動作に対するサポート権限を記憶する、サポート権限記憶部を備え、前記サポート要請と、当該サポート要請に対する前記サポート権限を有する前記オペレータ端末に対応するオペレータとの各組み合わせについて、前記第１の合計値、前記第２の合計値又は前記第３の合計値の総和を算出する、総和算出部を備え、前記情報提供部は、さらに、前記総和が最小となる組み合わせに基づいて、前記オペレータ端末へと前記サポート要請に関する情報を提供する、ものであってもよい。 The server further includes, for each operator corresponding to each operator terminal, a support authority storage unit that stores support authority for the robot system, the task, and / or a part of operations constituting the task. For each combination of the support request and the operator corresponding to the operator terminal having the support authority for the support request, the sum of the first total value, the second total value, or the third total value is calculated. The information providing unit may further provide information regarding the support request to the operator terminal based on the combination that minimizes the total.
　このような構成によれば、ロボットシステムやオペレータの数が大きくなる場合であっても、組み合わせ爆発を防止しつつ、システム全体として最適化を図ることができ、工程全体のダウンタイムを最小化することができる。 With such a configuration, even when the number of robot systems and operators is large, it is possible to optimize the entire system while preventing combinatorial explosions, and minimize the downtime of the entire process. be able to.
　前記優先順位は、前記ロボットシステムからサポート要請を受信した時刻が古い程高くなるよう決定される、ものであってもよい。 The priority may be determined so that the older the time when the support request is received from the robot system, the higher the priority.
　このような構成によれば、時系列順の対処が適するシステムにおいて、適切に優先順位を決定することができる。 According to such a configuration, it is possible to appropriately determine the priority in a system in which measures in chronological order are suitable.
　前記サーバは、前記ロボットシステム、前記タスク及び／又は前記タスクの一部を構成する動作について予め定められた優先度情報を記憶した、優先度情報記憶部を備え、前記優先順位は、前記優先度情報に基づいて決定される、ものであってもよい。 The server includes a priority information storage unit that stores predetermined priority information about the robot system, the task, and / or an operation constituting a part of the task, and the priority is the priority. It may be informed.
　このような構成によれば、運用に際して重視したい又は重視しないタスク等を予め設定することができる。 According to such a configuration, it is possible to set in advance tasks that should be emphasized or not emphasized in operation.
　前記オペレータ端末は、接続された前記ロボットシステムの一覧を表示する、一覧表示部と、前記サポート要請に関する情報に基づいて、前記サポート要請を行っている前記ロボットシステムを強調表示する、強調表示部と、を備えるものであってもよい。 The operator terminal includes a list display unit that displays a list of the connected robot systems, and a highlighting unit that highlights the robot system making the support request based on the information regarding the support request. , May be provided.
　このような構成によれば、サポート要請を行っているロボットシステムを即座に把握することができる。 With such a configuration, it is possible to immediately grasp the robot system requesting support.
　前記オペレータ端末は、さらに、前記優先順位に関する情報に基づいて、前記サポート要請を行っている前記ロボットシステムを前記優先順位でソート表示する、ソート表示部を備える、ものであってもよい。 The operator terminal may further include a sort display unit that sorts and displays the robot system requesting support according to the priority based on the information regarding the priority.
　このような構成によれば、ソート表示により、サポートすべきロボットシステムを直感的に把握することができる。 With such a configuration, it is possible to intuitively grasp the robot system to be supported by the sort display.
　前記サーバは、さらに、前記オペレータ端末のオペレータ毎に設定された前記ロボットシステム、前記タスク及び／又は前記タスクを構成する一部の動作に対するサポート権限を含む、アカウント情報を管理する、アカウント情報管理部を備え、前記オペレータ端末は、さらに、前記オペレータ端末を利用するオペレータに対して前記アカウント情報との照合を提供するログイン処理を提供する、ログイン処理部と、を備えるものであってもよい。 The server further manages account information, including support authority for the robot system, the task, and / or some operations constituting the task, which are set for each operator of the operator terminal. The operator terminal may further include a login processing unit that provides a login process that provides collation with the account information to an operator who uses the operator terminal.
　このような構成によれば、同一のオペレータ端末を異なるオペレータが自己のアカウントで利用することができる。 According to such a configuration, the same operator terminal can be used by different operators with their own accounts.
　前記オペレータ端末は、さらに、前記サポート要請を行っている前記ロボットシステムのうち、ログイン処理を行った前記オペレータが前記サポート権限を有する前記ロボットシステムを他のロボットシステムと区別可能に表示する、権限別表示部を備える、ものであってもよい。 The operator terminal further displays, among the robot systems requesting support, the robot system for which the operator who has performed the login process has the support authority so as to be distinguishable from other robot systems. It may be provided with a display unit.
　このような構成によれば、ログインしたオペレータが、サポート権限に基づいて、自己がサポート可能なロボットシステムを即座に把握することができる。 With such a configuration, the logged-in operator can immediately grasp the robot system that he / she can support based on the support authority.
　前記ロボットシステムは、移動マニピュレータを含む、ものであってもよい。 The robot system may include a mobile manipulator.
　このような構成によれば、半自律的にタスクを実行する移動マニピュレータから複数のサポート要請を受信した場合であっても、サポート要請に対して優先順位を決定することができるので、オペレータ端末の操作者であるオペレータは複数のサポート要請に対して容易に対処することができる。 With such a configuration, even when a plurality of support requests are received from a mobile manipulator that executes a task semi-autonomously, the priority can be determined for the support requests, so that the operator terminal can use the configuration. The operator, who is the operator, can easily deal with a plurality of support requests.
　本発明はシステムとしても観念することができる。すなわち、本発明に係るシステムは、所定のタスクを半自律的に実行する複数のロボットシステムと、１又は複数のオペレータ端末と、前記ロボットシステム及び前記オペレータ端末と接続されたサーバと、から成るシステムであって、前記サーバは、前記ロボットシステムからサポート要請を受信する、サポート要請受信部と、複数の前記サポート要請を受信した場合、前記サポート要請の優先順位を決定する、優先順位決定部と、前記オペレータ端末へと、前記サポート要請に関する情報と前記優先順位に関する情報を提供する、情報提供部と、を備えている。 The present invention can also be thought of as a system. That is, the system according to the present invention is a system including a plurality of robot systems that semi-autonomously execute predetermined tasks, one or a plurality of operator terminals, the robot system, and a server connected to the operator terminals. The server has a support request receiving unit that receives a support request from the robot system, and a priority determination unit that determines the priority of the support request when a plurality of the support requests are received. The operator terminal is provided with an information providing unit that provides information on the support request and information on the priority.
　本発明は方法としても観念することができる。すなわち、本発明に係る方法は、所定のタスクを半自律的に実行する複数のロボットシステムと、１又は複数のオペレータ端末と、接続されたサーバの制御方法であって、前記ロボットシステムからサポート要請を受信する、サポート要請受信ステップと、複数の前記サポート要請を受信した場合、前記サポート要請の優先順位を決定する、優先順位決定ステップと、前記オペレータ端末へと、前記サポート要請に関する情報と前記優先順位に関する情報を提供する、情報提供ステップと、
を備えている。 The present invention can also be considered as a method. That is, the method according to the present invention is a control method for a plurality of robot systems that semi-autonomously execute predetermined tasks, one or a plurality of operator terminals, and a connected server, and the robot system requests support. The support request receiving step, the priority determination step for determining the priority of the support request when a plurality of the support requests are received, and the information regarding the support request and the priority to the operator terminal. Information provision steps and information provision steps that provide information about rankings,
It is equipped with.
　本発明はコンピュータプログラムとしても観念することができる。すなわち、本発明に係るプログラムは、所定のタスクを半自律的に実行する複数のロボットシステムと、１又は複数のオペレータ端末と、接続されたサーバの制御プログラムであって、前記ロボットシステムからサポート要請を受信する、サポート要請受信ステップと、複数の前記サポート要請を受信した場合、前記サポート要請の優先順位を決定する、優先順位決定ステップと、前記オペレータ端末へと、前記サポート要請に関する情報と前記優先順位に関する情報を提供する、情報提供ステップと、を備えている。 The present invention can also be thought of as a computer program. That is, the program according to the present invention is a control program of a server connected to a plurality of robot systems that semi-autonomously execute predetermined tasks, one or a plurality of operator terminals, and a support request is made from the robot system. The support request receiving step, the priority determination step for determining the priority of the support request when a plurality of the support requests are received, and the information regarding the support request and the priority to the operator terminal. It has an information provision step, which provides information on the ranking.
　本発明によれば、半自律的にタスクを実行するロボットシステムから複数のサポート要請を受信した場合であっても、オペレータが複数のサポート要請に対して容易に対処することができるシステム等を提供することができる。 According to the present invention, there is provided a system or the like in which an operator can easily deal with a plurality of support requests even when a plurality of support requests are received from a robot system that executes a task semi-autonomously. can do.
図１は、システムの全体構成図である。FIG. 1 is an overall configuration diagram of the system. 図２は、システムの機能ブロック図である。FIG. 2 is a functional block diagram of the system. 図３は、ロボットシステムＩＤとタスクとの対応関係について示すテーブルである。FIG. 3 is a table showing the correspondence between the robot system ID and the task. 図４は、タスクと、タスクを構成する単位動作であるアクション、及び各アクションの所要時間との対応関係について示すテーブルである。FIG. 4 is a table showing the correspondence between the task, the action which is a unit operation constituting the task, and the required time of each action. 図５は、オペレータＩＤと、オペレータに付与されている権限との対応関係を示すテーブルである。FIG. 5 is a table showing the correspondence between the operator ID and the authority given to the operator. 図６は、ロボットシステムの動作について説明するフローチャートである。FIG. 6 is a flowchart illustrating the operation of the robot system. 図７は、管理サーバの情報収集動作に関するフローチャートである。FIG. 7 is a flowchart relating to the information collection operation of the management server. 図８は、オペレータ用システムへと情報提供を行う処理に関するフローチャートである。FIG. 8 is a flowchart relating to the process of providing information to the operator system. 図９は、優先順位決定処理の詳細動作に関するフローチャートである。FIG. 9 is a flowchart relating to the detailed operation of the priority determination process. 図１０は、一連のタスクに係るアローダイアグラムである。FIG. 10 is an arrow diagram relating to a series of tasks. 図１１は、動作状況一覧画面の表示例である。FIG. 11 is a display example of the operation status list screen. 図１２は、サポート処理に関する動作フローチャートである。FIG. 12 is an operation flowchart relating to the support process. 図１３は、サポート用画面の表示例である。FIG. 13 is a display example of the support screen.
　以下、本発明の好適な実施の形態について添付の図を参照しつつ詳細に説明する。 Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached figures.
　（１．第１の実施形態）
  第１の実施形態として、本発明を、一連のタスク、すなわち、物品選り分けの後に箱詰め・テープ閉じ等を行うタスクをそれぞれ実行する複数のロボットシステムを含む、物流倉庫内システムへと適用する例について説明する。 (1. First Embodiment)
As a first embodiment, there is an example of applying the present invention to a system in a distribution warehouse including a plurality of robot systems that execute a series of tasks, that is, tasks such as boxing and tape closing after sorting articles. explain.
　なお、本実施形態において、ロボットシステムは、物流倉庫システムへと適用されるもののそのような適用例に限定されず、他の様々な場面、例えば工場等にも適用可能である。また、本実施形態において、ロボットシステムは、ロボットアームを備えるものとして説明するが、ロボットアームの他、様々なハードウェアを採用し得る。なお、ロボットシステムは固定されている必要はなく、例えば、移動マニピュレータ等を採用することもできる。また、ロボットシステムのタスクとしては上述のものに限定されるものではなく、他の様々なタスクにも適用可能である。 Although the robot system is applied to a distribution warehouse system in the present embodiment, it is not limited to such an application example, and can be applied to various other situations such as factories. Further, in the present embodiment, the robot system will be described as including the robot arm, but various hardware may be adopted in addition to the robot arm. The robot system does not have to be fixed, and for example, a mobile manipulator or the like can be adopted. Further, the task of the robot system is not limited to the above-mentioned one, and can be applied to various other tasks.
　（１．１　システムの構成）
  図１は、本実施形態に係るシステム２００の全体構成図である。同図から明らかな通り、システム２００は、複数のロボットシステム１、管理サーバ３及びオペレータ用システム５を備えており、それらは互いにＬＡＮ（ローカル・エリア・ネットワーク）を介して有線又は無線により接続されている。なお、本実施形態においては、各装置間はＬＡＮにより接続されるものとして説明するが、インターネットを介して接続してもよい。 (1.1 System configuration)
FIG. 1 is an overall configuration diagram of the system 200 according to the present embodiment. As is clear from the figure, the system 200 includes a plurality of robot systems 1, a management server 3, and an operator system 5, which are connected to each other via a LAN (local area network) by wire or wirelessly. ing. In the present embodiment, it is assumed that the devices are connected by LAN, but they may be connected via the Internet.
　各ロボットシステム１は、後述の種々の情報処理を行うロボット側情報処理装置１１と、ロボット側情報処理装置１１と接続されたセンサ１５と、ロボット側情報処理装置１１と接続された多関節ロボットアーム１７とから構成されている。 Each robot system 1 includes a robot-side information processing device 11 that performs various information processing described later, a sensor 15 connected to the robot-side information processing device 11, and an articulated robot arm connected to the robot-side information processing device 11. It is composed of 17.
　なお、本実施形態において、ロボット側情報処理装置１１とセンサ１５とロボットアーム１７とは物理的には離間して配置されつつも一のシステムを構成するものとして説明するものの、その一部又は全部を一体に構成してもよい。 Although the robot-side information processing device 11, the sensor 15, and the robot arm 17 are described as constituting one system while being physically separated from each other in the present embodiment, a part or all of them are described. May be integrally configured.
　ロボット側情報処理装置１１は、ＣＰＵ等から成り種々のプログラムを実行する制御部と、ＲＯＭやＲＡＭ、ハードディスク等から成る記憶部とを備えている。 The robot-side information processing device 11 includes a control unit composed of a CPU and the like for executing various programs, and a storage unit composed of a ROM, RAM, a hard disk, and the like.
　センサ１５は、認識の基礎となる画像情報を取得する環境に固定された３次元カメラであり、例えば、物品選り分けタスクを行う場合には対象となる多数の製品等のワークが入っている箱の中を撮像し、箱詰めタスクを行う場合にはコンベヤ上のワークを撮像する。センサから取得された情報は、ロボット側情報処理装置１１へと送信される。 The sensor 15 is a three-dimensional camera fixed to an environment for acquiring image information that is the basis of recognition. For example, when performing an article selection task, a box containing a large number of target products and other workpieces. The inside is imaged, and the work on the conveyor is imaged when performing the boxing task. The information acquired from the sensor is transmitted to the information processing device 11 on the robot side.
　このようにセンサ１５である３次元カメラが環境中に固定され、後述するように、認識と遠隔操作の両方に共用される構成によれば、同一の視点を利用できるため、構成が簡潔となると共にコスト的に有利となる。なお、センサ１５の取り付け方法はこのように環境中に固定する例に限定されず、ロボットアーム１７等に取り付けられるハンドアイカメラを利用したり、既設の監視カメラを流用すること等してもよい。 In this way, the three-dimensional camera, which is the sensor 15, is fixed in the environment, and as will be described later, according to the configuration shared for both recognition and remote control, the same viewpoint can be used, so that the configuration is simplified. At the same time, it is advantageous in terms of cost. The method of attaching the sensor 15 is not limited to the example of fixing the sensor 15 in the environment in this way, and a hand-eye camera attached to the robot arm 17 or the like may be used, or an existing surveillance camera may be used. ..
　多関節ロボットアーム１７は、先端にグリッパを有し、自在に動作することができる。例えば、物品選り分けタスクを行う場合には、ワークを挟持してベルトコンベア等の上に載置する動作を行う。また、箱詰め・テープ閉じタスクにおいては、ベルトコンベヤ上に載置されたワークの把持と移動、テープ閉じ動作等を行う。 The articulated robot arm 17 has a gripper at the tip and can move freely. For example, when performing an article sorting task, an operation of sandwiching a work and placing it on a belt conveyor or the like is performed. Further, in the boxing / tape closing task, the work placed on the belt conveyor is gripped and moved, and the tape closing operation is performed.
　管理サーバ３は、後述のように、ロボットシステム１に関する情報とオペレータ用システム５に関する情報を適宜に収集する。また、必要に応じて種々の情報をロボットシステム１及びオペレータ用システム５へと提供する。 The management server 3 appropriately collects information on the robot system 1 and information on the operator system 5, as will be described later. Further, various information is provided to the robot system 1 and the operator system 5 as needed.
　オペレータ用システム５は、オペレータ側情報処理装置５１、表示装置５２及びハプティック入力装置５３を備えている。オペレータ側情報処理装置５１は、オペレータ用システム５の全体制御を行う情報処理装置であり、例えば、ＰＣ（パーソナルコンピュータ）である。表示装置５２は、ディスプレイ装置であり、後述するように、サポート要請を発したロボットシステム１の一覧等を表示する。 The operator system 5 includes an operator-side information processing device 51, a display device 52, and a haptic input device 53. The operator-side information processing device 51 is an information processing device that controls the entire operator system 5, and is, for example, a PC (personal computer). The display device 52 is a display device, and displays a list or the like of the robot system 1 that has issued a support request, as will be described later.
　ハプティック入力装置５３は、各関節につき角度センサが設けられており、ロボットアーム１７の手先の６軸姿勢とグリッパの開閉を制御することができる。ハプティック入力装置５３は、後述するように、オペレータにより操作され、サポート要請に応じてロボットアーム１７のタスク実行をサポートする。なお、ハプティック入力装置５３は、ロボットアーム１７に備えられたセンサに基づいて、力フィードバックがなされるものであってもよい。 The haptic input device 53 is provided with an angle sensor for each joint, and can control the 6-axis posture of the hand of the robot arm 17 and the opening / closing of the gripper. As will be described later, the haptic input device 53 is operated by an operator to support task execution of the robot arm 17 in response to a support request. The haptic input device 53 may be one in which force feedback is performed based on the sensor provided in the robot arm 17.
　表示装置や入力装置は、本実施形態の例に限定されず、既知の様々な装置を利用することができる。例えば、表示装置はヘッドマウントディスプレイであってもよいし、入力装置は、スティックレバーを用いた入力装置やキーボード等であってもよい。 The display device and the input device are not limited to the examples of the present embodiment, and various known devices can be used. For example, the display device may be a head-mounted display, and the input device may be an input device using a stick lever, a keyboard, or the like.
　また、本実施形態において、オペレータ側情報処理装置５１と表示装置５２とハプティック入力装置５３とは物理的には離間して配置されつつも一のシステムを構成するものとして説明するものの、その一部又は全部を一体に構成してもよい。 Further, in the present embodiment, the operator-side information processing device 51, the display device 52, and the haptic input device 53 are described as being physically separated from each other to form one system, but a part thereof. Alternatively, the whole may be integrally configured.
　図２は、システム２００の機能ブロック図である。同図から明らかな通り、ロボットシステム１は、それぞれの送受信部を介して管理サーバ３及びオペレータ用システム５と情報の授受を行う。 FIG. 2 is a functional block diagram of the system 200. As is clear from the figure, the robot system 1 exchanges information with the management server 3 and the operator system 5 via the respective transmission / reception units.
　ロボットシステム１は、ロボット側情報処理装置１１と、センサ１５と、ロボットアーム１７とから構成されている。 The robot system 1 includes a robot-side information processing device 11, a sensor 15, and a robot arm 17.
　ロボット側情報処理装置１１は、センサ１５における検出情報を取得するセンサ情報取得部１１１と、センサ情報に基づいて所定の認識処理を行う認識処理部１１２とを備えている。認識処理部１１２による認識結果は、判定部１１４へと提供される。判定部１１４は、認識結果に基づいて後述のサポート要請を行うべきか否かを判定し、動作生成部１１８へと判定結果を提供する。動作生成部１１８と動作指令部１１９は、ロボットアーム１７に対して動作指令を行う。 The robot-side information processing device 11 includes a sensor information acquisition unit 111 that acquires detection information in the sensor 15, and a recognition processing unit 112 that performs predetermined recognition processing based on the sensor information. The recognition result by the recognition processing unit 112 is provided to the determination unit 114. The determination unit 114 determines whether or not to make a support request described later based on the recognition result, and provides the determination result to the motion generation unit 118. The motion generation unit 118 and the motion command unit 119 issue motion commands to the robot arm 17.
　サポート処理実行部１２１は、送受信部１２７を介してサポート情報を受信すると、サポート情報に応じた種々のサポート処理を実行する。例えば、ロボットアーム１７の動作制御等を行う。 When the support process execution unit 121 receives the support information via the transmission / reception unit 127, the support process execution unit 121 executes various support processes according to the support information. For example, the operation of the robot arm 17 is controlled.
　管理サーバ３は、サポート要請検出部３２と情報取得処理部３３とを備えている。サポート要請検出部３２は、ロボットシステム１から送信されたサポート要請を検出する。また、優先順位決定処理部３５は、サポート要請を複数受信した場合には、サポート要請について優先順位を決定する。情報提供処理部３６は、送受信部を介して、サポート要請に関する情報や優先順位に関する情報をオペレータ用システム５へと送信する。 The management server 3 includes a support request detection unit 32 and an information acquisition processing unit 33. The support request detection unit 32 detects the support request transmitted from the robot system 1. Further, when a plurality of support requests are received, the priority determination processing unit 35 determines the priority of the support request. The information provision processing unit 36 transmits information regarding support requests and information regarding priorities to the operator system 5 via the transmission / reception unit.
　情報取得処理部３３は、ロボットシステム１及びオペレータ用システム５のステータス情報等を取得する。取得された情報は、記憶処理部３８により図示しない記憶部へと記憶される。また、リスト制御部３７は、ステータス情報等に基づき、後述の所定のリスト情報を管理する。 The information acquisition processing unit 33 acquires status information and the like of the robot system 1 and the operator system 5. The acquired information is stored in a storage unit (not shown) by the storage processing unit 38. Further, the list control unit 37 manages predetermined list information described later based on the status information and the like.
　次に、予め管理サーバ３に設定情報として記憶されているテーブル情報について説明する。なお、以下では、テーブル情報は予め記憶されているものとして記載しているものの、本発明はそのような構成に限定されず、運用中にオペレータ用システム５等を介して設定されてもよい。 Next, the table information stored in advance as setting information in the management server 3 will be described. In the following, although the table information is described as being stored in advance, the present invention is not limited to such a configuration and may be set via the operator system 5 or the like during operation.
　図３は、管理サーバ３の記憶部へと予め記憶された、ロボットシステムＩＤとタスクとの対応関係について示すテーブルである。同図から明らかな通り、各ロボットシステム１には「ｒｏｂｏｔ０１」、「ｒｏｂｏｔ０２」のように識別情報が付与されており、当該識別情報に対応して実行すべきタスクの識別情報が、例えば「ＴＡＳＫ＿Ａ」（タスクＡ）、「ＴＡＳＫ＿Ｂ」（タスクＢ）のように付与されている。すなわち、このようなテーブルに基づき「ｒｏｂｏｔ０１」が「ＴＡＳＫ＿Ａ」を実行するよう設定されている。 FIG. 3 is a table showing the correspondence between the robot system ID and the task stored in advance in the storage unit of the management server 3. As is clear from the figure, identification information is given to each robot system 1 such as "robot01" and "robot02", and the identification information of the task to be executed corresponding to the identification information is, for example, "TASK_A". "(Task A)," TASK_B "(task B). That is, "robot01" is set to execute "TASK_A" based on such a table.
　図４は、管理サーバ３の記憶部へと予め記憶された、タスクと、タスクを構成する単位動作であるアクション、及び各アクションの所要時間（［ｓ］）との対応関係について示すテーブルである。同図から明らかな通り、「ＴＡＳＫ＿Ａ」は、「ＡＣＴＩＯＮ＿０１」から「ＡＣＴＩＯＮ＿０７」までの７つのアクションで構成されており、各アクションには、同図で示される所要時間が対応付けられている。 FIG. 4 is a table showing the correspondence relationship between the task stored in advance in the storage unit of the management server 3, the action which is a unit operation constituting the task, and the required time ([s]) of each action. .. As is clear from the figure, "TASK_A" is composed of seven actions from "ACTION_01" to "ACTION_07", and each action is associated with the required time shown in the figure.
　図５は、管理サーバ３の記憶部へと予め記憶された、オペレータの識別情報であるオペレータＩＤと、オペレータに付与されている権限との対応関係を示すテーブルである。同図から明らかな通り、オペレータＩＤ「Ｓａｋａｍｏｔｏ」には、「ＴＡＳＫ＿Ａ」と「ＴＡＳＫ＿Ｂ」の２つのアクションに対するサポート権限が付与されている。また、オペレータＩＤ「Ｋａｗａｎｉｓｈｉ」には、「ＴＡＳＫ＿Ｂ」に対するサポート権限のみが付与されている。ただし、「Ｓａｋａｍｏｔｏ」には、「ｒｏｂｏｔ０３」についてはサポート権限が付与されていない。 FIG. 5 is a table showing the correspondence between the operator ID, which is the identification information of the operator, stored in advance in the storage unit of the management server 3 and the authority given to the operator. As is clear from the figure, the operator ID "Sakamoto" is given support authority for the two actions "TASK_A" and "TASK_B". Further, the operator ID "Kawanishi" is given only the support authority for "TASK_B". However, "Sakamoto" is not granted support authority for "robot03".
　なお、本実施形態においてサポート権限は、タスクとロボットＩＤの単位で定義されているものの、他の方法で定義されてもよい。従って、例えば、アクション単位等、他の手段によりサポート権限を定義してもよい。 Although the support authority is defined in units of the task and the robot ID in this embodiment, it may be defined by another method. Therefore, the support authority may be defined by other means such as an action unit.
　また、本実施形態におけるシステム構成は例示であって、ネットワークに接続される各装置の数や構成は自在に変更可能である。 Further, the system configuration in this embodiment is an example, and the number and configuration of each device connected to the network can be freely changed.
　（１．２　システムの動作）
  次に、システム２００の動作について説明する。 (1.2 System operation)
Next, the operation of the system 200 will be described.
　図６は、ロボットシステム１の動作について説明するフローチャートである。概して言えば、ロボットシステム１は、半自律的に目標動作やタスクを実行しつつ、適切な認識ができない場合や目標動作に失敗した場合に、管理サーバ３へとサポート要請を送信する。 FIG. 6 is a flowchart illustrating the operation of the robot system 1. Generally speaking, the robot system 1 executes a target motion or a task semi-autonomously, and sends a support request to the management server 3 when an appropriate recognition cannot be performed or the target motion fails.
　同図から明らかな通り、処理が開始すると、センサ情報取得部１１１は、センサ１５からのセンサ検出情報、本実施形態においては箱の中やコンベヤ上を撮影することにより得られた画像情報を取得する（Ｓ１）。認識処理部１１２は、センサ情報に基づいて、ロボットアーム１７による把持の対象となるワーク等を認識する処理を行う（Ｓ２）。なお、本実施形態においては、画像情報に基づいて認識を行うものとして説明するが、本発明はこのような構成に限定されない。従って、例えば、画像に代えて、又は画像と共に３次元カメラにて取得された３次元情報を利用するような構成としてもよい。 As is clear from the figure, when the process starts, the sensor information acquisition unit 111 acquires the sensor detection information from the sensor 15, and the image information obtained by photographing the inside of the box or the conveyor in the present embodiment. (S1). The recognition processing unit 112 performs a process of recognizing a work or the like to be gripped by the robot arm 17 based on the sensor information (S2). In the present embodiment, the recognition is performed based on the image information, but the present invention is not limited to such a configuration. Therefore, for example, the configuration may be such that the three-dimensional information acquired by the three-dimensional camera is used instead of the image or together with the image.
　この認識処理の後、判定部１１４は、サポートの必要性について判定する処理を行う（Ｓ４）。より詳細には、判定部１１４は、認識部１１２による認識が正常に行われた場合、例えば、十分な認識精度にて把持対象となるワークを認識することができた場合には、サポートが不要と判定し、一方、認識が正常に行われなかった場合にはサポートが必要と判定する。 After this recognition process, the determination unit 114 performs a process of determining the necessity of support (S4). More specifically, the determination unit 114 does not need support when recognition by the recognition unit 112 is normally performed, for example, when the work to be gripped can be recognized with sufficient recognition accuracy. On the other hand, if the recognition is not performed normally, it is determined that support is required.
　なお、本実施形態においては、正常な認識が出来るか否かによりサポートの必要性について判定を行うものとしたが、本発明はこのような構成に限定されず、種々の判定方法が採用可能である。例えば、把持対象物に対する把持解が算出できるか否かといった基準に基づいて判定処理を行ってもよい。 In the present embodiment, the necessity of support is determined based on whether or not normal recognition is possible, but the present invention is not limited to such a configuration, and various determination methods can be adopted. be. For example, the determination process may be performed based on a criterion such as whether or not a gripping solution for the gripping object can be calculated.
　判定処理の結果、認識処理が正常に行われてサポートが不要と判定された場合には（Ｓ４ＮＯ）、動作生成部１１８により目標動作が生成される（Ｓ６）。目標動作とは、ロボットシステム１に対して設定されたタスク動作等のロボットシステム１が実行を期待される動作である。その後、動作指令部１１９は、生成された目標動作を実行するようロボットアーム１７に対して動作指令を行い目標動作を実行する（Ｓ７）。 As a result of the determination process, if the recognition process is performed normally and it is determined that support is unnecessary (S4NO), the motion generation unit 118 generates a target motion (S6). The target motion is an motion expected to be executed by the robot system 1 such as a task motion set for the robot system 1. After that, the motion command unit 119 issues an motion command to the robot arm 17 so as to execute the generated target motion, and executes the target motion (S7).
　この動作の後、センサ１５情報に基づいて、目標動作に成功したか否かを評価する処理が行われる（Ｓ８）。この評価処理の結果、目標動作に成功したと判定された場合（Ｓ８ＹＥＳ）、終了判定を行い（Ｓ９）、所定の終了信号等が検出される場合には（Ｓ９ＹＥＳ）、処理は終了する。一方、終了信号が検出されない場合には、再度センサ情報の取得処理（Ｓ１）から一連の処理が再度繰り返される（Ｓ９ＮＯ）。なお、目標動作に失敗したと判定された場合（Ｓ８ＮＯ）、後述のサポート要請の送信処理が行われる（Ｓ１１）。 After this operation, a process of evaluating whether or not the target operation was successful is performed based on the sensor 15 information (S8). As a result of this evaluation process, if it is determined that the target operation is successful (S8YES), the end determination is performed (S9), and if a predetermined end signal or the like is detected (S9YES), the process ends. On the other hand, when the end signal is not detected, a series of processes from the sensor information acquisition process (S1) are repeated again (S9NO). If it is determined that the target operation has failed (S8NO), the support request transmission process described later is performed (S11).
　なお、サポートとは、半自律ロボットに対して目標動作の実行補助を目的として操作や指示等を行うことを意味する。また、サポート要請とは、そのような補助をオペレータに対して要求する情報を意味する。なお、本実施形態において、ロボットシステム1からのサポート要請をアラートと称することがある。 Note that support means to perform operations and instructions to the semi-autonomous robot for the purpose of assisting the execution of the target movement. Also, the support request means the information requesting the operator for such assistance. In this embodiment, the support request from the robot system 1 may be referred to as an alert.
　サポートが必要と判定された場合（Ｓ４ＹＥＳ）又は目標動作に失敗した場合（Ｓ８ＮＯ）、管理サーバ３へとサポート要請の送信処理が行われる（Ｓ１１）。その後、ロボット側情報処理装置１１は、サポート情報を受信するまで待機状態となる（Ｓ１２ＮＯ）。 When it is determined that support is necessary (S4YES) or when the target operation fails (S8NO), the support request transmission process is performed to the management server 3 (S11). After that, the information processing apparatus 11 on the robot side goes into a standby state until the support information is received (S12NO).
　この状態においてサポート情報を受信すると（Ｓ１２ＹＥＳ）、サポート処理実行部１２１は、サポート情報に応じたサポートの実行処理を行う（Ｓ１４）。その後、所定の終了判定を行い、終了信号が検出されない場合、再び受信待機状態（Ｓ１５ＮＯ）となり一連の処理が繰り返される。一方、終了信号が検出された場合（Ｓ１５ＹＥＳ）、上述の終了判定処理（Ｓ９）が行われる。 When the support information is received in this state (S12YES), the support processing execution unit 121 performs the support execution processing according to the support information (S14). After that, a predetermined end determination is performed, and if the end signal is not detected, the reception standby state (S15NO) is set again and a series of processes are repeated. On the other hand, when the end signal is detected (S15YES), the above-mentioned end determination process (S9) is performed.
　図７は、ロボットシステム１とオペレータ用システム５から様々な情報を取得する管理サーバ３の情報収集動作に関するフローチャートである。同図から明らかな通り、処理が開始すると、情報取得処理部３３は、ロボットシステム１のステータス情報（Ｓ２１）、サポート要請（Ｓ３１）、サポート完了信号（Ｓ４１）及びオペレータ情報（Ｓ５１）の４つの情報を取得する処理を平行して実行し、それぞれの処理は、該当する情報を取得するまでそれぞれ待機状態となる（Ｓ２２ＮＯ、Ｓ３２ＮＯ、Ｓ４２ＮＯ、Ｓ５２ＮＯ）。 FIG. 7 is a flowchart relating to the information collection operation of the management server 3 that acquires various information from the robot system 1 and the operator system 5. As is clear from the figure, when the processing starts, the information acquisition processing unit 33 has four status information (S21), a support request (S31), a support completion signal (S41), and an operator information (S51) of the robot system 1. The processes for acquiring information are executed in parallel, and each process is in a standby state until the corresponding information is acquired (S22NO, S32NO, S42NO, S52NO).
　ロボットシステム１のステータス情報は、ロボットシステム１の現在の動作状態を表す情報であり、例えば、実行中のタスク、アクション、動作開始からの経過時間、ロボットアーム１７の関節角度情報、センサ１５により検出されたセンサ情報等の情報を含む。サポート要請は、上述の処理によりロボットシステム１から送信されたオペレータによるサポートを求める情報であり、例えば、サポート要請を行ったロボットシステム１のＩＤ、サポート要請が生成された時刻等の情報を含む。 The status information of the robot system 1 is information representing the current operating state of the robot system 1, and is detected by, for example, a task being executed, an action, an elapsed time from the start of the operation, joint angle information of the robot arm 17, and a sensor 15. Includes information such as sensor information. The support request is information for requesting support by the operator transmitted from the robot system 1 by the above processing, and includes, for example, information such as the ID of the robot system 1 that made the support request and the time when the support request was generated.
　サポート完了信号は、オペレータによるサポートが完了した時にオペレータ用システム５から送信される情報であり、例えば、サポート完了時刻等を含む。オペレータ情報は、オペレータ側情報処理装置５１へとログインすること等により対応付けられたオペレータに関する情報であり、例えば、オペレータＩＤ等の情報を含むものである。なお、ログイン処理は、管理サーバ３により記憶されたアカウント情報との照合を行うことにより行われる。 The support completion signal is information transmitted from the operator system 5 when the support by the operator is completed, and includes, for example, the support completion time. The operator information is information about an operator associated with the operator by logging in to the information processing apparatus 51 on the operator side, and includes, for example, information such as an operator ID. The login process is performed by collating with the account information stored by the management server 3.
　このように、本実施形態においては、オペレータ用システム５は、ログインによりオペレータと関連付けられるので、同一のオペレータ用システム５を異なるオペレータが共用することができる。 As described above, in the present embodiment, since the operator system 5 is associated with the operator by login, the same operator system 5 can be shared by different operators.
　記憶処理部３８は、ロボットシステム１からステータス情報を受信すると（Ｓ２２ＹＥＳ）、ステータス情報を記憶する処理を行い（Ｓ２３）、再び待機状態となる。 When the storage processing unit 38 receives the status information from the robot system 1 (S22YES), the storage processing unit 38 performs a process of storing the status information (S23), and is in the standby state again.
　リスト制御部３７は、ロボットシステム１からサポート要請を受信すると（Ｓ３２ＹＥＳ）、サポート要請を行ったロボットシステム１のＩＤ情報を所定のサポート要請リストへと追加する処理を行う（Ｓ３３）。また、このとき、記憶処理部３８によりサポート要請を行ったロボットシステム１のＩＤ情報が記憶部へと記憶される。その後、再び待機状態となる。 When the list control unit 37 receives the support request from the robot system 1 (S32YES), the list control unit 37 performs a process of adding the ID information of the robot system 1 that made the support request to the predetermined support request list (S33). At this time, the ID information of the robot system 1 for which the support is requested by the storage processing unit 38 is stored in the storage unit. After that, it goes into the standby state again.
　また、リスト制御部３７は、サポート完了信号を受信すると（Ｓ４２ＹＥＳ）、サポートが完了したロボットシステム１のＩＤ情報を所定のサポート要請リストから削除する処理を行う（Ｓ４３）。また、このとき、記憶処理部３８によりサポート要請を行ったロボットシステム１のＩＤ情報が記憶部から消去される。その後、再び待機状態となる。 Further, when the list control unit 37 receives the support completion signal (S42YES), the list control unit 37 performs a process of deleting the ID information of the robot system 1 for which support has been completed from the predetermined support request list (S43). At this time, the ID information of the robot system 1 for which the support is requested by the storage processing unit 38 is deleted from the storage unit. After that, it goes into the standby state again.
　さらに、記憶処理部３８は、オペレータ用システム５からオペレータ情報を受信すると（Ｓ５２ＹＥＳ）、オペレータ情報を記憶する処理を行い（Ｓ５３）、再び待機状態となる。 Further, when the storage processing unit 38 receives the operator information from the operator system 5 (S52YES), the storage processing unit 38 performs a process of storing the operator information (S53), and is in the standby state again.
　すなわち、管理サーバ３は、ロボットシステム１とオペレータ用システム５から送信される様々な情報を収集・取得して記憶・管理を行っている。 That is, the management server 3 collects, acquires, stores, and manages various information transmitted from the robot system 1 and the operator system 5.
　図８は、オペレータ用システム５へと情報提供を行う処理に関するフローチャートである。同図左側のフローチャートは、オペレータ用システム５の動作を示し、同図右側のフローチャートは、管理サーバ３の動作を示している。 FIG. 8 is a flowchart relating to the process of providing information to the operator system 5. The flowchart on the left side of the figure shows the operation of the operator system 5, and the flowchart on the right side of the figure shows the operation of the management server 3.
　同図から明らかな通り、処理が開始すると、オペレータ側情報処理装置５１の表示処理部５７は、管理サーバ３へと表示リクエストを行う（Ｓ６１）。この表示リクエストの後、オペレータ側情報処理装置５１は所定の情報の待機状態となる（Ｓ６２ＮＯ）。所定の情報を受信した場合には（Ｓ６２ＹＥＳ）、当該情報に応じた表示処理を表示部５２へと行う（Ｓ６４）。その後は、一連の処理は再び繰り返される（Ｓ６１～Ｓ６４）。すなわち、オペレータ側情報処理装置５１は、常に更新表示処理を行っている。 As is clear from the figure, when the processing starts, the display processing unit 57 of the information processing apparatus 51 on the operator side makes a display request to the management server 3 (S61). After this display request, the information processing apparatus 51 on the operator side goes into a standby state for predetermined information (S62NO). When the predetermined information is received (S62YES), the display process corresponding to the information is performed on the display unit 52 (S64). After that, the series of processes is repeated again (S61 to S64). That is, the information processing device 51 on the operator side is constantly performing update display processing.
　表示リクエストを受信するまで待機状態（Ｓ７１ＮＯ）にあった管理サーバ３のサポート要請検出部３２は、オペレータ側情報処理装置５１から表示リクエストを受信すると（Ｓ７１ＹＥＳ）、サポート要請リストから受信時点においてサポート要請が行われているロボットシステム１を検出する処理を行う（Ｓ７２）。 When the support request detection unit 32 of the management server 3 that has been in the standby state (S71NO) until the display request is received receives the display request from the information processing device 51 on the operator side (S71YES), the support request is received from the support request list at the time of reception. Is performed (S72) to detect the robot system 1 in which the above is performed.
　この検出処理の結果、サポート要請が１個の場合（Ｓ７３ＮＯ）、情報提供処理部３６は、オペレータ用システム５に対して、当該サポート要請をオペレータ用システム５へと提示するための情報を提供する処理を行う（Ｓ７５）。なお、この提供される情報には、管理サーバ３により収集されるあらゆる情報が含まれ得る。 As a result of this detection process, when there is only one support request (S73NO), the information providing processing unit 36 provides the operator system 5 with information for presenting the support request to the operator system 5. Processing is performed (S75). The provided information may include any information collected by the management server 3.
　一方、サポート要請が複数存在する場合（Ｓ７３ＹＥＳ）、優先順位決定処理部３５は、サポート要請間の優先順位を決定する処理（Ｓ７６）を行い、この処理の後、優先順位と共に所定の情報提供処理を行う（Ｓ７８）。なお、この提供される情報には、管理サーバ３により収集されるあらゆる情報が含まれ得る。 On the other hand, when there are a plurality of support requests (S73YES), the priority determination processing unit 35 performs a process (S76) for determining the priority among the support requests, and after this process, a predetermined information provision process together with the priority. (S78). The provided information may include any information collected by the management server 3.
　図９は、優先順位決定処理の詳細動作に関するフローチャートである。同図から明らかな通り、処理が開始すると、変数を初期化すると共に特定のロボットシステム１を参照して初期化処理が行われる（Ｓ８１）。 FIG. 9 is a flowchart relating to the detailed operation of the priority determination process. As is clear from the figure, when the process starts, the variable is initialized and the initialization process is performed with reference to the specific robot system 1 (S81).
　その後、参照したロボットシステム１の行うタスクの後続のタスクの合計所要時間を算出する処理が行われる（Ｓ８３）。より詳細には、上述の、ロボットシステムＩＤとタスクとの対応関係について示すテーブルと、タスクとその所要時間（［ｓ］）との対応関係について示すテーブルとに基づいて、所要時間の合計値を計算する。 After that, a process of calculating the total required time of the task following the task performed by the referenced robot system 1 is performed (S83). More specifically, the total required time is calculated based on the above-mentioned table showing the correspondence between the robot system ID and the task and the table showing the correspondence between the task and its required time ([s]). calculate.
　この後続のタスクの所要時間の合計値の計算の後、全てのロボットシステム１に対して同合計値の算出処理が完了したか否か（Ｓ８４）が判定され、未だ全てのロボットシステム１に対して算出処理が完了していない場合には、参照対象となるロボットシステム１を変更して、再び算出処理（Ｓ８３）が行われる。 After calculating the total value of the required time of the subsequent task, it is determined whether or not the calculation process of the total value is completed for all the robot systems 1 (S84), and still for all the robot systems 1. If the calculation process is not completed, the robot system 1 to be referred to is changed, and the calculation process (S83) is performed again.
　一方、すべてのロボットシステム１に対して算出処理が完了した場合には（Ｓ８４ＹＥＳ）、同合計値に基づいてサポート要請を行っているロボットシステム１の対処の優先順位を決定する処理が行われる（Ｓ８６）。本実施形態においては、後続のタスクの所要時間の合計値が大きいほど、対処の優先順位が高いものとして決定される。その後、優先順位決定処理は終了する。 On the other hand, when the calculation process is completed for all the robot systems 1 (S84YES), the process of determining the priority of the response of the robot system 1 requesting support based on the total value is performed (S84YES). S86). In the present embodiment, the larger the total value of the required time of the subsequent tasks, the higher the priority of the coping is determined. After that, the priority determination process ends.
　図１０を参照しつつ、優先順位の決定に関する具体例について説明する。図１０は、本実施形態に係る一連のタスクに係るアローダイアグラムである。同図から明らかな通り、「ｒｏｂｏｔ０１」により物品の選り分けタスク（タスクＡ）が実行されると、選り分けられた対象物は、３つの平行な工程として、異なるコンベヤ上に載置されて搬送される。コンベヤにより搬送された物品は、後続の「ｒｏｂｏｔ０２」、「ｒｏｂｏｔ０３」及び「ｒｏｂｏｔ０４」により、それぞれ箱詰めされ、箱はそれぞれテープ閉じされる。その後、各ロボットシステム１は、テープ閉じした箱をコンベヤへと載置して搬送する（タスクＢ）。 A specific example of determining the priority will be described with reference to FIG. FIG. 10 is an arrow diagram relating to a series of tasks according to the present embodiment. As is clear from the figure, when the article sorting task (task A) is executed by "robot01", the sorted objects are placed and conveyed on different conveyors as three parallel processes. .. The articles conveyed by the conveyor are boxed by the subsequent "robot02", "robot03" and "robot04", respectively, and the boxes are tape-closed. After that, each robot system 1 places the tape-closed box on a conveyor and conveys it (task B).
　このとき、タスクＡの実行には１１．６秒の時間、タスクＢの実行には３０秒の時間がかかるものとし、その間のコンベヤ搬送には１０秒の時間がかかるものとされている。なお、これらの時間は、実際にロボットシステム１を動作させて計測された時間である。このような構成によれば、所要時間を実測値に基づいて正確に見積もることができる。 At this time, it is assumed that the execution of the task A takes 11.6 seconds, the execution of the task B takes 30 seconds, and the conveyor transfer during that time takes 10 seconds. It should be noted that these times are the times measured by actually operating the robot system 1. According to such a configuration, the required time can be accurately estimated based on the actually measured value.
　この状態において、例えば、ロボットシステム１のうちの「ｒｏｂｏｔ０１」について後続のタスクの合計所要時間を算出すると、後続の３つの並行な工程のうち、１つの系列の工程について合計５０秒（＝１０＋３０＋１０）の時間がかかることから、合計所要時間は１５０秒（＝５０×３）として計算される。同様に「ｒｏｂｏｔ０２」、「ｒｏｂｏｔ０３」及び「ｒｏｂｏｔ０４」について後続のタスクの合計所要時間を算出すると、それぞれ１０秒として計算される。 In this state, for example, when the total required time of the subsequent task is calculated for "robot01" in the robot system 1, the total required time for one series of the following three parallel processes is 50 seconds (= 10 + 30 + 10). Therefore, the total required time is calculated as 150 seconds (= 50 × 3). Similarly, when the total required time of the subsequent tasks is calculated for "robot02", "robot03", and "robot04", each is calculated as 10 seconds.
　すなわち、同図の例にあっては、ロボットシステム１のうち、「ｒｏｂｏｔ０２」、「ｒｏｂｏｔ０３」及び「ｒｏｂｏｔ０４」よりも、「ｒｏｂｏｔ０１」の方が停止等による工程全体への影響が大きいロボットシステム１であることが把握される。従って、後続のタスクの合計所要時間のより大きい「ｒｏｂｏｔ０１」からのサポート要請に対して優先的に対処すべきこととなる。 That is, in the example of the figure, among the robot systems 1, "robot01" has a greater influence on the entire process due to stopping or the like than "robot02", "robot03", and "robot04". It is grasped that. Therefore, the support request from "robot01", which has a larger total time required for subsequent tasks, should be dealt with preferentially.
　図１１は、表示処理（Ｓ６４）の結果、オペレータ用システム５の表示部５２へと表示される動作状況一覧画面の表示例である。同図から明らかな通り、同画面中には中央に動作状況の一覧表示部８０が設けられると共に、画面右上には、現在オペレータ用システム５を利用しているオペレータのＩＤ情報「Ｓａｋａｍｏｔｏ」と現在時刻が表示されている。 FIG. 11 is a display example of the operation status list screen displayed on the display unit 52 of the operator system 5 as a result of the display process (S64). As is clear from the figure, the operation status list display unit 80 is provided in the center of the screen, and the ID information "Sakamoto" of the operator currently using the operator system 5 is displayed in the upper right of the screen. The time is displayed.
　また、動作状況一覧画面には、左から、「稼働状況」、「トラブル発生時刻」、「ロボットＩＤ」、「タスク」、「動作」、「あなたの補助権限」、「補助者」、及び「補助状況」と表示された列が表示されている。 In addition, on the operation status list screen, from the left, "operation status", "trouble occurrence time", "robot ID", "task", "operation", "your auxiliary authority", "assistant", and "assistant". The column labeled "Auxiliary status" is displayed.
　「稼働状況」は、ロボットシステム１が正常に動作しているか又はアラート発生状態にあるか否かを示している。アラート発生状態とは、目標動作を実行できなかったか又は実行出来る見込みがないことから、ロボットシステム１がオペレータに対してサポート要請を行っている状態である。 "Operating status" indicates whether the robot system 1 is operating normally or whether an alert is generated. The alert generation state is a state in which the robot system 1 is requesting support from the operator because the target operation cannot be executed or is unlikely to be executed.
　「トラブル発生時刻」は、サポート要請が発生した日時に関する情報である。「ロボットＩＤ」は、サポート要請に対応するロボットシステム１の識別情報であるＩＤ情報である。 "Trouble occurrence time" is information about the date and time when the support request occurred. The "robot ID" is ID information which is identification information of the robot system 1 corresponding to the support request.
　「タスク」は、各ロボットシステム１に対して割り当てられたタスクであり、同図の例にあっては、ロボットに対して「物品選り分け」タスク（タスクＡ）又は「箱詰め・テープ閉じ」タスク（タスクＢ）のいずれかのタスクが割当てられている。 The "task" is a task assigned to each robot system 1, and in the example of the figure, the "article selection" task (task A) or the "boxing / tape closing" task (boxing / tape closing) for the robot (task A). One of the tasks in task B) is assigned.
　「動作」は、タスクを構成するアクションのうち、いずれのアクションを行っているかを示している。同図の例からは、「ｒｏｂｏｔ０１」にあっては「ＡＣＴＩＯＮ＿０２」の「ケース内対象物把持位置認識」動作を実行中にサポート要請を行ったことが把握される。また、「ｒｏｂｏｔ０２」にあっては「ＡＣＴＩＯＮ＿９」の「グリッパ開放・収納」動作を正常に実行中であることが把握される。 "Action" indicates which of the actions that make up the task is being performed. From the example in the figure, it can be understood that in the case of "robot01", the support request was made during the "recognition of the gripping position of the object in the case" operation of "ACTION_02". Further, in the case of "robot02", it is understood that the "gripper opening / storing" operation of "ACTION_9" is being normally executed.
　「あなたの補助権限」は、サポート要請を行っている各ロボットシステム１に対して、ログインを行っているオペレータがサポート権限を有しているか否かを示すものである。同図から、オペレータ「Ｓａｋａｍｏｔｏ」が、「ｒｏｂｏｔ０３」を除き、タスクＡとタスクＢに対するサポート権限を有していることが把握される。 "Your auxiliary authority" indicates whether or not the operator who is logged in has the support authority for each robot system 1 requesting support. From the figure, it can be seen that the operator "Sakamoto" has the support authority for task A and task B except for "robot03".
　「補助者」は、サポートを行っているオペレータのＩＤを表示するものである。例えば、同図の例にあっては、「ｒｏｂｏｔ０７」についてはオペレータである「Ｋａｗａｎｉｓｈｉ」がサポートを行っていることが把握される。 The "assistant" displays the ID of the operator who is providing support. For example, in the example of the figure, it can be seen that the operator "Kawanishi" provides support for "robot07".
　「補助状況」は、サポート要請に応じて、サポートが行われているか否かを示すものである。例えば、同図の例にあっては、「ｒｏｂｏｔ０７」からのサポート要請について、オペレータである「Ｋａｗａｎｉｓｈｉ」が対応中であることが示されている。 "Assistance status" indicates whether or not support is being provided in response to a support request. For example, in the example of the figure, it is shown that the operator "Kawanishi" is responding to the support request from "robot07".
　同図の例にあっては、サポート要請は、算出された優先順位に基づいて、対処の優先順位が高いものから順に上から「ｒｏｂｏｔ０１」、「ｒｏｂｏｔ０４」、「ｒｏｂｏｔ０７」、「ｒｏｂｏｔ０３」が表示されている。すなわち、サポート要請を行っているロボットシステム１がソートされて、サポート要請を行っていないロボットシステム１と区別可能に表示されている。 In the example of the figure, the support request is displayed as "robot01", "robot04", "robot07", and "robot03" from the top in order from the highest priority of the response based on the calculated priority. Has been done. That is, the robot system 1 requesting support is sorted and displayed so as to be distinguishable from the robot system 1 not requesting support.
　なお、サポート要請を行っているロボットシステム１を区別可能に表示する方法はこのような例に限定されず、他の種々の表示方法を採用することができる。例えば、サポート要請を行っているロボットシステム１のみを表示したり、サポート要請を行っているロボットシステム１に関する表示を着色する等して、強調表示してもよい。 The method for displaying the robot system 1 requesting support in a distinctive manner is not limited to such an example, and various other display methods can be adopted. For example, only the robot system 1 requesting support may be displayed, or the display related to the robot system 1 requesting support may be colored to be highlighted.
　次に、オペレータ用システム５を介してオペレータによりなされるサポート処理について説明する。 Next, the support process performed by the operator via the operator system 5 will be described.
　図１２は、オペレータにより行われるサポート処理に関する動作フローチャートである。同図左側のフローチャートは、オペレータ用システム５の動作フローチャートであり、同図右側のフローチャートは、管理サーバ３の動作フローチャートである。 FIG. 12 is an operation flowchart relating to the support process performed by the operator. The flowchart on the left side of the figure is an operation flowchart of the operator system 5, and the flowchart on the right side of the figure is an operation flowchart of the management server 3.
　同図において、処理が開始すると、オペレータ側情報処理装置５１は、動作状況一覧に表示されたサポート要請のいずれかを選択する入力がなされたかを検出する処理を行う（Ｓ１０１）。この検出処理の結果、入力が検出されない場合（Ｓ１０２ＮＯ）、入力検出処理が繰り返される（Ｓ１０１、Ｓ１０２ＮＯ）。 In the figure, when the process starts, the information processing apparatus 51 on the operator side performs a process of detecting whether or not an input for selecting one of the support requests displayed in the operation status list has been made (S101). If no input is detected as a result of this detection process (S102NO), the input detection process is repeated (S101, S102NO).
　検出処理の結果、入力が検出された場合（Ｓ１０２ＹＥＳ）、操作中のオペレータが、選択されたサポート要請に対して対処することを要求するサポートリクエストを送信する処理が行われる（Ｓ１０４）。このサポートリクエストの送信の後、オペレータ側情報処理装置５１は、所定の受信待機状態となる（Ｓ１０５ＮＯ）。 When an input is detected as a result of the detection process (S102YES), a process is performed in which the operating operator sends a support request requesting that the selected support request be dealt with (S104). After the transmission of this support request, the information processing apparatus 51 on the operator side enters a predetermined reception standby state (S105NO).
　一方、サポートリクエストを受信するまで待機状態（Ｓ１１１ＮＯ）にあった管理サーバ３は、サポートリクエストを受信すると（Ｓ１１１ＹＥＳ）、サポートリクエストを送信したオペレータを担当オペレータとしてサポート要請と関連づけて設定する処理を行う（Ｓ１１２）。これにより、例えば、図１１の「補助者」、「補助状況」欄に、対応するオペレータＩＤと「対応中」である旨が表示される。この後、管理サーバ３は、サポートリクエストが設定されたことを示す通知をオペレータ用システム５へと送信する（Ｓ１１４）。 On the other hand, when the management server 3 has been in the standby state (S111NO) until the support request is received (S111YES), the management server 3 performs a process of setting the operator who sent the support request as the operator in charge in association with the support request. (S112). As a result, for example, in the "assistant" and "assistance status" columns of FIG. 11, the corresponding operator ID and "corresponding" are displayed. After that, the management server 3 transmits a notification indicating that the support request has been set to the operator system 5 (S114).
　オペレータ用システム５は、通知を受信すると（Ｓ１０５ＹＥＳ）、サポート用画面を表示する処理を行う。 When the operator system 5 receives the notification (S105YES), the operator system 5 performs a process of displaying the support screen.
　図１３は、オペレータがサポート要請に対して対処することを可能とするサポート用画面の表示例である。同図から明らかな通り、同画面の中央上部には選択した対応中のサポート要請に関するステータス表示部８３が設けられると共に、画面右上には、現在オペレータ用システム５を利用しているオペレータのＩＤ情報「Ｓａｋａｍｏｔｏ」と現在時刻が表示されている。 FIG. 13 is a display example of a support screen that enables the operator to respond to a support request. As is clear from the figure, a status display unit 83 related to the selected support request being handled is provided in the upper center of the screen, and the ID information of the operator currently using the operator system 5 is provided in the upper right of the screen. "Sakamoto" and the current time are displayed.
　同図下部の左側には、ロボットシステム１のカメラ１５により撮像されたカメラビューを表示するカメラビュー表示領域８３が設けられている。同図の例にあっては、カメラビュー表示領域８３には、物品選り分けタスクの対象となる、箱９１内の積み重なったワーク９２の画像が表示されている。ワーク９２は円筒状であり、ロボットアーム１７によりそれぞれ把持されて所定の基準で選り分けられる。 On the left side of the lower part of the figure, a camera view display area 83 for displaying the camera view captured by the camera 15 of the robot system 1 is provided. In the example of the figure, the image of the stacked work 92 in the box 91, which is the target of the article selection task, is displayed in the camera view display area 83. The work 92 has a cylindrical shape, is gripped by the robot arm 17, and is sorted according to a predetermined standard.
　なお、本実施形態においては、カメラビュー表示領域８３には、カメラ画像が表示されるものとして説明するが、本発明はこのような構成に限定されない。従って、例えば、画像に代えて、又は画像と共に３次元カメラにて取得された３次元情報を提示するような構成としてもよい。 Although the camera image is described in the camera view display area 83 in the present embodiment, the present invention is not limited to such a configuration. Therefore, for example, the configuration may be such that the three-dimensional information acquired by the three-dimensional camera is presented in place of the image or together with the image.
　同図下部の中央には、現在姿勢にあるロボットアーム１７に備えられた図示しない関節角度センサから得られた姿勢情報に基づいて生成されたＣＧ（Ｃｏｍｐｕｔｅｒ　Ｇｒａｐｈｉｃｓ）画像８４１を表示するＣＧ画像表示部８４が設けられている。同画像表示部によりオペレータは、ロボットアーム１７が現在どのような姿勢にあるかを瞬時に把握することができる。 In the center of the lower part of the figure, a CG image display unit that displays a CG (Computer Graphics) image 841 generated based on posture information obtained from a joint angle sensor (not shown) provided on the robot arm 17 in the current posture. 84 is provided. The image display unit allows the operator to instantly grasp the current posture of the robot arm 17.
　同図下部の右側には、サポート処理を開始するための作業開始ボタン８５が配置されている。 On the right side of the lower part of the figure, the work start button 85 for starting the support process is arranged.
　図１２に戻り、作業開始ボタン８５が選択されると、オペレータ用システム５と管理サーバ３、及びロボットシステム１が適宜に通信を行うことで、サポート処理が行われる（Ｓ１０８）。 Returning to FIG. 12, when the work start button 85 is selected, the operator system 5, the management server 3, and the robot system 1 appropriately communicate with each other to perform support processing (S108).
　サポート処理の内容は、本実施形態においては、カメラビュー表示領域８３を確認しつつ、ハプティック入力装置５３を用いてロボットアーム１７の遠隔操作を行い、前記目標動作の実行条件を変更する動作である。目標動作の実行条件を変更する動作とは、例えば、シーンや環境条件を変更する動作であり、より具体的には、オペレータによる遠隔操作によりロボットアーム１７を用いて箱を揺らしたり箱の中のワーク９２を把持して移動する等して、ワーク９２の位置・姿勢を変化させる動作である。 In the present embodiment, the content of the support process is an operation of remotely controlling the robot arm 17 using the haptic input device 53 while checking the camera view display area 83 to change the execution condition of the target operation. .. The operation of changing the execution condition of the target operation is, for example, an operation of changing the scene or the environmental condition, and more specifically, the robot arm 17 is used to shake the box or the inside of the box by remote control by the operator. This is an operation of changing the position and posture of the work 92 by gripping and moving the work 92.
　なお、本実施形態においては、サポート処理の内容は、目標動作の実行条件を変更する動作としたが、これに限定されない。従って、例えば、目標動作それ自体を遠隔操作により代替的に実行してもよい。 In the present embodiment, the content of the support process is an operation of changing the execution condition of the target operation, but the content is not limited to this. Therefore, for example, the target operation itself may be performed as an alternative by remote control.
　その後、サポートが一通り完了すると、サポート完了信号が管理サーバ３へと送信され、その後、再び、入力検出処理（Ｓ１０１）から一連の処理が繰り返される。なお、上述の通り、サポート完了信号は、管理サーバ３により取得されて、対応するサポート要請のリストからの消去処理が行われる（Ｓ４１～Ｓ４３）。 After that, when the support is completed, the support completion signal is transmitted to the management server 3, and then the input detection process (S101) repeats a series of processes. As described above, the support completion signal is acquired by the management server 3 and erased from the corresponding list of support requests (S41 to S43).
　このような構成によれば、半自律的にタスクを実行するロボットシステムから複数のサポート要請を受信した場合であっても、サポート要請に対して優先順位を決定することができるので、オペレータが自らどのタスクから優先的に対処すべきかを考える必要がなく、複数のサポート要請に対して容易に対処することができる。 With such a configuration, even when a plurality of support requests are received from a robot system that executes a task semi-autonomously, the priority can be determined for the support requests, so that the operator himself / herself can determine the priority. It is not necessary to consider which task should be prioritized, and it is possible to easily deal with multiple support requests.
　また、このような構成によれば、オペレータは、所要時間に基づいて、工程全体への影響の大きいタスクから対処することができるので、システム全体を効率よく運用することができる。 Further, according to such a configuration, the operator can deal with tasks having a large influence on the entire process based on the required time, so that the entire system can be operated efficiently.
　（２．変形例）
  上述の実施形態においては、後続のタスクの合計所要時間を算出して優先順位を決定する構成としたが、本発明はそのような構成に限定されない。 (2. Modification example)
In the above-described embodiment, the total required time of subsequent tasks is calculated to determine the priority, but the present invention is not limited to such a configuration.
　従って、例えば、当該合計所要時間に対してさらに、参照したロボットシステム１の残りのアクションの所要時間をさらに加算した値に基づいて、優先順位を決定してもよい。このような構成によれば、実行中のタスクの残りのアクションの所要時間も加味するので、実行時間の大きいアクション等が存在する場合でも、より精緻に優先順位を決定することができる。 Therefore, for example, the priority may be determined based on a value obtained by further adding the required time of the remaining actions of the referenced robot system 1 to the total required time. According to such a configuration, the time required for the remaining actions of the task being executed is also taken into consideration, so that even if there is an action having a long execution time, the priority can be determined more precisely.
　また、例えば、当該合計所要時間に対してさらに、ロボットシステム１へのサポートに要すると見込まれる所要時間をさらに加算した値に基づいて、優先順位を決定してもよい。このような構成によれば、サポートに一定の時間を要するような場合であっても、より精緻に優先順位を決定することができる。 Further, for example, the priority may be determined based on a value obtained by further adding the required time expected to be required for the support to the robot system 1 to the total required time. According to such a configuration, even if it takes a certain amount of time for support, the priority can be determined more precisely.
　上述の実施形態においては、オペレータには、サポート要請を行っている全てのロボットシステム１が表示される構成としたが、本発明はそのような構成に限定されない。従って、例えば、サポート要請を行ったロボットシステム１とオペレータ用システム５との組み合わせについて最適化を図ってもよい。 In the above-described embodiment, the operator is configured to display all the robot systems 1 requesting support, but the present invention is not limited to such a configuration. Therefore, for example, the combination of the robot system 1 for which the support is requested and the operator system 5 may be optimized.
　例えば、サポート要請を行ったロボットシステム１とオペレータ用システム５とのすべての組み合わせについて工程全体の所要時間を計算し、その中で最も所要時間が最小となる組み合わせをオペレータへと推薦し又は提供してもよい。このような構成によれば、システム全体において最適化を図ることができ、工程全体のダウンタイムを最小化することができる。 For example, the time required for the entire process is calculated for all combinations of the robot system 1 and the operator system 5 for which support is requested, and the combination with the shortest required time is recommended or provided to the operator. You may. With such a configuration, optimization can be achieved for the entire system, and downtime for the entire process can be minimized.
　また、例えば、サポート要請を行ったロボットシステム１と、当該サポート要請に対してサポート権限のあるオペレータ用システム５とのすべての組み合わせについて工程全体の所要時間を計算し、その中で最も所要時間が最小となる組み合わせをオペレータへと推薦し又は提供してもよい。このような構成によれば、システム全体において最適化を図ることができ、工程全体のダウンタイムを最小化することができる。また、システム２００が大規模化して、ロボットシステム１とオペレータ用システム５の数が増大としても、組み合わせの爆発を防止して演算負荷の適正化と処理の高速化を図ることができる。 Further, for example, the required time of the entire process is calculated for all combinations of the robot system 1 that has made a support request and the operator system 5 that has the support authority for the support request, and the required time is the longest. The minimum combination may be recommended or provided to the operator. With such a configuration, optimization can be achieved for the entire system, and downtime for the entire process can be minimized. Further, even if the system 200 becomes large-scale and the number of the robot system 1 and the operator system 5 increases, it is possible to prevent the combination from exploding, optimize the calculation load, and speed up the processing.
　上述の実施形態においては、後続タスクの所要時間等に基づいて優先順位を決定したが、単にサポート要請が発生した時系列順で表示端末５２へと表示してもよい。このような構成によれば、発生順の処理が望ましいシステムについて最適なサポートを提供することができる。 In the above-described embodiment, the priority is determined based on the required time of the succeeding task and the like, but it may be simply displayed on the display terminal 52 in chronological order in which the support request is generated. With such a configuration, it is possible to provide optimal support for a system in which processing in the order of occurrence is desirable.
　以上、本発明の実施形態について説明したが、上記実施形態は本発明の適用例の一部を示したに過ぎず、本発明の技術的範囲を上記実施形態の具体的構成に限定する趣旨ではない。また、上記の実施形態は、矛盾が生じない範囲で適宜組み合わせ可能である。 Although the embodiments of the present invention have been described above, the above embodiments are only a part of the application examples of the present invention, and the technical scope of the present invention is limited to the specific configuration of the above embodiments. not. Further, the above embodiments can be appropriately combined as long as there is no contradiction.
　本発明は、少なくともロボットシステム等を製造する産業において利用可能である。 The present invention can be used at least in an industry that manufactures robot systems and the like.
　１　ロボットシステム
　１１　ロボット側情報処理装置
　１５　センサ
　１７　ロボットアーム
　３　サーバ
　５　オペレータ用システム
　５１　オペレータ側情報処理装置
　５２　表示装置
　５３　ハプティック入力装置 1 Robot system 11 Robot side information processing device 15 Sensor 17 Robot arm 3 Server 5 Operator system 51 Operator side information processing device 52 Display device 53 Haptic input device

Claims (18)

  1. 　所定のタスクを半自律的に実行する複数のロボットシステムと、１又は複数のオペレータ端末と、接続されたサーバであって、
    　前記ロボットシステムからサポート要請を受信する、サポート要請受信部と、
    　複数の前記サポート要請を受信した場合、前記サポート要請の優先順位を決定する、優先順位決定部と、
    　前記オペレータ端末へと、前記サポート要請に関する情報と前記優先順位に関する情報を提供する、情報提供部と、
    を備える、サーバ。     A server connected to a plurality of robot systems that execute predetermined tasks semi-autonomously, one or more operator terminals, and the like.
    A support request receiving unit that receives a support request from the robot system,
    When a plurality of the support requests are received, a priority determination unit that determines the priority of the support requests and
    An information providing unit that provides information on the support request and information on the priority to the operator terminal.
    A server.
  2. 　前記優先順位決定部は、さらに、
    　前記サポート要請を行った前記ロボットシステム毎に、当該ロボットシステムが実行するタスクの後続のタスクの所要時間の合計値である第１の合計値を算出する、所要時間算出部と、
    　前記第１の合計値が大きい程、前記優先順位が高くなるよう優先順位を決定する、所要時間ベース優先順位決定部と、を備える請求項１に記載のサーバ。     The priority determination unit further
    A required time calculation unit that calculates a first total value, which is a total value of the required times of the tasks succeeding the tasks executed by the robot system, for each of the robot systems for which the support request has been made.
    The server according to claim 1, further comprising a required time-based priority determination unit that determines the priority so that the larger the first total value is, the higher the priority is.
  3. 　前記優先順位決定部は、さらに、
    　前記サポート要請を行った前記ロボットシステム毎に、当該ロボットシステムが実行するタスクの後続のタスクの所要時間の合計値に、実行中のタスクの残りの所要時間を加えた第２の合計値を算出する、第２の所要時間算出部と、
    　前記第２の合計値が大きい程、前記優先順位が高くなるよう優先順位を決定する、第２の所要時間ベース優先順位決定部と、を備える請求項１に記載のサーバ。     The priority determination unit further
    For each robot system that has made the support request, a second total value is calculated by adding the remaining time required for the task being executed to the total time required for the task following the task executed by the robot system. The second required time calculation unit and
    The server according to claim 1, further comprising a second time-based priority determination unit that determines the priority so that the larger the second total value is, the higher the priority is.
  4. 　前記優先順位決定部は、さらに、
    　前記サポート要請を行った前記ロボットシステム毎に、当該ロボットシステムが実行するタスクの後続のタスクの所要時間の合計値に、実行中のタスクの残りの所要時間と、必要なサポート動作の所要時間と、を加えた第３の合計値を算出する、第３の所要時間算出部と、
    　前記第３の合計値が大きい程、前記優先順位が高くなるよう優先順位を決定する、第３の所要時間ベース優先順位決定部と、を備える請求項１に記載のサーバ。     The priority determination unit further
    For each robot system that has made the support request, the total time required for the task following the task executed by the robot system, the remaining time required for the running task, and the required support operation time. The third required time calculation unit, which calculates the third total value by adding, and
    The server according to claim 1, further comprising a third required time-based priority determination unit that determines the priority so that the larger the total value of the third is, the higher the priority is.
  5. 　いずれの前記所要時間も前記ロボットシステムを運用することにより計測された実測値に基づいて決定されるものである、請求項２～４のいずれか１項に記載のサーバ。 The server according to any one of claims 2 to 4, wherein any of the required times is determined based on an actually measured value measured by operating the robot system.
  6. 　前記サーバは、さらに、
    　各前記オペレータ端末に対応するオペレータ毎に、前記ロボットシステム、前記タスク及び／又は前記タスクを構成する一部の動作に対するサポート権限を記憶する、サポート権限記憶部を備え、
    　前記情報提供部は、前記サポート権限に基づいて、前記オペレータ端末へと前記サポート要請に関する情報を提供する、請求項１～５のいずれか１項に記載のサーバ。     The server further
    For each operator corresponding to each operator terminal, a support authority storage unit for storing support authority for the robot system, the task, and / or a part of the operation constituting the task is provided.
    The server according to any one of claims 1 to 5, wherein the information providing unit provides information regarding the support request to the operator terminal based on the support authority.
  7. 　前記サーバは、さらに、
    　前記サポート要請と前記オペレータ端末に対応するオペレータとの各組み合わせについて、前記第１の合計値、前記第２の合計値又は前記第３の合計値の総和を算出する、総和算出部を備え、
    　前記情報提供部は、さらに、前記総和が最小となる組み合わせに基づいて、前記オペレータ端末へと前記サポート要請に関する情報を提供する、請求項２～５のいずれか１項に記載のサーバ。     The server further
    A total calculation unit is provided for calculating the total of the first total value, the second total value, or the third total value for each combination of the support request and the operator corresponding to the operator terminal.
    The server according to any one of claims 2 to 5, wherein the information providing unit further provides information regarding the support request to the operator terminal based on the combination that minimizes the total.
  8. 　前記サーバは、さらに、
    　各前記オペレータ端末に対応するオペレータ毎に、前記ロボットシステム、前記タスク及び／又は前記タスクを構成する一部の動作に対するサポート権限を記憶する、サポート権限記憶部を備え、
    　前記サポート要請と、当該サポート要請に対する前記サポート権限を有する前記オペレータ端末に対応するオペレータとの各組み合わせについて、前記第１の合計値、前記第２の合計値又は前記第３の合計値の総和を算出する、総和算出部を備え、
    　前記情報提供部は、さらに、前記総和が最小となる組み合わせに基づいて、前記オペレータ端末へと前記サポート要請に関する情報を提供する、請求項２～５のいずれか１項に記載のサーバ。     The server further
    For each operator corresponding to each operator terminal, a support authority storage unit for storing support authority for the robot system, the task, and / or a part of the operation constituting the task is provided.
    For each combination of the support request and the operator corresponding to the operator terminal having the support authority for the support request, the sum of the first total value, the second total value, or the third total value. Equipped with a total calculation unit to calculate
    The server according to any one of claims 2 to 5, wherein the information providing unit further provides information regarding the support request to the operator terminal based on the combination that minimizes the total.
  9. 　前記優先順位は、前記ロボットシステムからサポート要請を受信した時刻が古い程高くなるよう決定される、請求項１に記載のサーバ。 The server according to claim 1, wherein the priority is determined so that the older the time when the support request is received from the robot system, the higher the priority.
  10. 　前記サーバは、前記ロボットシステム、前記タスク及び／又は前記タスクの一部を構成する動作について予め定められた優先度情報を記憶した、優先度情報記憶部を備え、
    　前記優先順位は、前記優先度情報に基づいて決定される、請求項１に記載のサーバ。     The server includes a priority information storage unit that stores predetermined priority information about the robot system, the task, and / or an operation constituting a part of the task.
    The server according to claim 1, wherein the priority is determined based on the priority information.
  11. 　前記オペレータ端末は、
    　接続された前記ロボットシステムの一覧を表示する、一覧表示部と、
    　前記サポート要請に関する情報に基づいて、前記サポート要請を行っている前記ロボットシステムを強調表示する、強調表示部と、を備える、請求項１に記載のサーバ。     The operator terminal is
    A list display unit that displays a list of the connected robot systems,
    The server according to claim 1, further comprising a highlighting unit that highlights the robot system making the support request based on the information regarding the support request.
  12. 　前記オペレータ端末は、さらに、
    　前記優先順位に関する情報に基づいて、前記サポート要請を行っている前記ロボットシステムを前記優先順位でソート表示する、ソート表示部を備える、請求項１１に記載のサーバ。     The operator terminal further
    The server according to claim 11, further comprising a sort display unit that sorts and displays the robot system making the support request based on the information regarding the priority.
  13. 　前記サーバは、さらに、
    　前記オペレータ端末のオペレータ毎に設定された前記ロボットシステム、前記タスク及び／又は前記タスクを構成する一部の動作に対するサポート権限を含む、アカウント情報を管理する、アカウント情報管理部を備え、
    　前記オペレータ端末は、さらに、
    　前記オペレータ端末を利用するオペレータに対して前記アカウント情報との照合を提供するログイン処理を提供する、ログイン処理部と、を備える、請求項１に記載のサーバ。     The server further
    It is provided with an account information management unit that manages account information, including support authority for the robot system, the task, and / or a part of operations constituting the task, which are set for each operator of the operator terminal.
    The operator terminal further
    The server according to claim 1, further comprising a login processing unit that provides a login process that provides a collation with the account information to an operator who uses the operator terminal.
  14. 　前記オペレータ端末は、さらに、
    　前記サポート要請を行っている前記ロボットシステムのうち、ログイン処理を行った前記オペレータが前記サポート権限を有する前記ロボットシステムを他のロボットシステムと区別可能に表示する、権限別表示部を備える、請求項１３に記載のサーバ。     The operator terminal further
    The claim is provided with an authority-specific display unit for displaying the robot system for which the operator who has performed the login process has the support authority among the robot systems for which the support is requested so as to be distinguishable from other robot systems. 13. The server according to 13.
  15. 　前記ロボットシステムは、移動マニピュレータを含む、請求項１～１４に記載のサーバ。 The server according to claim 1 to 14, wherein the robot system includes a mobile manipulator.
  16. 　所定のタスクを半自律的に実行する複数のロボットシステムと、１又は複数のオペレータ端末と、前記ロボットシステム及び前記オペレータ端末と接続されたサーバと、から成るシステムであって、
    　前記サーバは、
    　前記ロボットシステムからサポート要請を受信する、サポート要請受信部と、
    　複数の前記サポート要請を受信した場合、前記サポート要請の優先順位を決定する、優先順位決定部と、
    　前記オペレータ端末へと、前記サポート要請に関する情報と前記優先順位に関する情報を提供する、情報提供部と、
    を備える、システム。     A system consisting of a plurality of robot systems that execute predetermined tasks semi-autonomously, one or a plurality of operator terminals, the robot system, and a server connected to the operator terminals.
    The server
    A support request receiving unit that receives a support request from the robot system,
    When a plurality of the support requests are received, a priority determination unit that determines the priority of the support requests and
    An information providing unit that provides information on the support request and information on the priority to the operator terminal.
    The system.
  17. 　所定のタスクを半自律的に実行する複数のロボットシステムと、１又は複数のオペレータ端末と、接続されたサーバの制御方法であって、
    　前記ロボットシステムからサポート要請を受信する、サポート要請受信ステップと、
    　複数の前記サポート要請を受信した場合、前記サポート要請の優先順位を決定する、優先順位決定ステップと、
    　前記オペレータ端末へと、前記サポート要請に関する情報と前記優先順位に関する情報を提供する、情報提供ステップと、
    を備える、制御方法。     It is a control method of a plurality of robot systems that execute a predetermined task semi-autonomously, one or a plurality of operator terminals, and a connected server.
    A support request receiving step for receiving a support request from the robot system,
    When a plurality of the support requests are received, the priority determination step for determining the priority of the support requests and the priority determination step.
    An information providing step that provides the operator terminal with information regarding the support request and information regarding the priority.
    A control method.
  18. 　所定のタスクを半自律的に実行する複数のロボットシステムと、１又は複数のオペレータ端末と、接続されたサーバの制御プログラムであって、
    　前記ロボットシステムからサポート要請を受信する、サポート要請受信ステップと、
    　複数の前記サポート要請を受信した場合、前記サポート要請の優先順位を決定する、優先順位決定ステップと、
    　前記オペレータ端末へと、前記サポート要請に関する情報と前記優先順位に関する情報を提供する、情報提供ステップと、
    を備える、制御プログラム。     A control program for a server connected to a plurality of robot systems that execute predetermined tasks semi-autonomously, one or more operator terminals, and the like.
    A support request receiving step for receiving a support request from the robot system,
    When a plurality of the support requests are received, the priority determination step for determining the priority of the support requests and the priority determination step.
    An information providing step that provides the operator terminal with information regarding the support request and information regarding the priority.
    A control program.
PCT/JP2021/030433 2020-10-12 2021-08-19 Server, control method therefor, program, and system WO2022080007A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020-172092 2020-10-12
JP2020172092A JP2022063710A (en) 2020-10-12 2020-10-12 Server, control method and program for the same, and system

Publications (1)

Publication Number Publication Date
WO2022080007A1 true WO2022080007A1 (en) 2022-04-21

Family

ID=81208289

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/030433 WO2022080007A1 (en) 2020-10-12 2021-08-19 Server, control method therefor, program, and system

Country Status (2)

Country Link
JP (1) JP2022063710A (en)
WO (1) WO2022080007A1 (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003193342A (en) * 2001-12-12 2003-07-09 Mas Fab Rieter Ag Method and alarm system for maintenance and/or removal of trouble
JP2014048685A (en) * 2012-08-29 2014-03-17 Toyota Motor East Japan Inc Automatic production system
JP2019016134A (en) * 2017-07-06 2019-01-31 日本電気株式会社 Production support system and production support method
JP2020095296A (en) * 2018-12-10 2020-06-18 パナソニックＩｐマネジメント株式会社 Production management device and production management method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003193342A (en) * 2001-12-12 2003-07-09 Mas Fab Rieter Ag Method and alarm system for maintenance and/or removal of trouble
JP2014048685A (en) * 2012-08-29 2014-03-17 Toyota Motor East Japan Inc Automatic production system
JP2019016134A (en) * 2017-07-06 2019-01-31 日本電気株式会社 Production support system and production support method
JP2020095296A (en) * 2018-12-10 2020-06-18 パナソニックＩｐマネジメント株式会社 Production management device and production management method

Also Published As

Publication number Publication date
JP2022063710A (en) 2022-04-22

Similar Documents

Publication Publication Date Title
CN111194452B (en) Motion prediction system and motion prediction method
US8965580B2 (en) Training and operating industrial robots
US20190061151A1 (en) Article stacking apparatus and machine learning apparatus
US10166673B2 (en) Portable apparatus for controlling robot and method thereof
US20110288667A1 (en) Industrial robot system
US11826913B2 (en) Control system, robot system and control method
US11052541B1 (en) Autonomous robot telerobotic interface
Bolano et al. Virtual reality for offline programming of robotic applications with online teaching methods
CN112469538B (en) Data generation device and method, data generation program, and remote operation system
US11389954B2 (en) Robot control device
WO2022080007A1 (en) Server, control method therefor, program, and system
JP2020082314A (en) Learning device, robot control method, and robot control system
CN110914021A (en) Operating device with an operating device for carrying out at least one work step, and method and computer program
KR101398215B1 (en) Dual arm robot control apparatus and method with error recovery function
EP3703915B1 (en) Method of performing assembling of an object, and assembly system
Sibona et al. PoinTap system: a human-robot interface to enable remotely controlled tasks
WO2022080006A1 (en) Robot system, control method for same, program, and system
US20230278223A1 (en) Robots, tele-operation systems, computer program products, and methods of operating the same
CN111899629A (en) Flexible robot teaching system and method
WO2023068352A1 (en) Robot data processing server and robot program calculation method
Lehmann et al. Assembly process digitization through self-learning assistance systems in production
US20230069565A1 (en) Systems and Methods for Doubles Detection and Mitigation
WO2022209638A1 (en) Work instruction system
US20240012385A1 (en) Control device
JP2712834B2 (en) Failure support equipment

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21879749

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21879749

Country of ref document: EP

Kind code of ref document: A1