WO2020194392A1 - Computer, method, and program for generating teaching data for autonomous robot - Google Patents
Computer, method, and program for generating teaching data for autonomous robot Download PDFInfo
- Publication number
- WO2020194392A1 WO2020194392A1 PCT/JP2019/012180 JP2019012180W WO2020194392A1 WO 2020194392 A1 WO2020194392 A1 WO 2020194392A1 JP 2019012180 W JP2019012180 W JP 2019012180W WO 2020194392 A1 WO2020194392 A1 WO 2020194392A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- remote
- log
- manual
- teacher data
- remote control
- Prior art date
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J3/00—Manipulators of master-slave type, i.e. both controlling unit and controlled unit perform corresponding spatial movements
Definitions
- the present invention relates to a teacher data generation computer for an autonomous robot, a method and a program, and more specifically, to generate teacher data for making an autonomous robot learn advanced tasks such as those performed by humans.
- the present invention provides a teacher data generation computer for an autonomous robot, a method, and a method capable of generating teacher data for causing an autonomous robot to work even in an advanced work such as that performed by a human.
- the purpose is to provide a program.
- the present invention is a teacher for an autonomous robot that generates teacher data for making an autonomous robot that performs autonomous work learn from a remote control log in which a remote operator in a remote location remotely controls the remote controlled robot.
- a data generation computer the first acquisition means for acquiring a remote operation log by having the remote operator remotely control a remote control robot according to a manual, and the second acquisition for acquiring sensor data for the remote operation.
- the means, the determination means that analyzes the acquired sensor data and determines whether or not the remote control is performed according to the manual, and as a result of the determination, it is determined that the remote control is according to the manual.
- a teacher data generation computer for an autonomous robot provided with a generation means for generating teacher data for training an autonomous robot performing a work.
- the present invention is an autonomous robot that generates teacher data for making an autonomous robot that performs autonomous work learn from a remote control log in which a remote operator in a remote place remotely controls the remote control robot.
- the teacher data generation method a step of acquiring a remote operation log in which the remote operator remotely controls a remote control type robot according to a manual, a step of acquiring sensor data for the remote operation, and the above-mentioned
- a method for generating teacher data for an autonomous robot which comprises a step of generating teacher data for training.
- the present invention generates teacher data for making a remote-controlled robot learn from a remote-controlled log in which a remote-controlled operator at a remote location remotely controls a remote-controlled robot.
- the teacher data for making an autonomous robot that performs autonomous work learn from a remote control log in which a remote operator in a remote place remotely controls a remote control robot. Therefore, even if it is an advanced work that a person does, the teacher data for letting the autonomous robot learn, that is, the teacher data that can teach the autonomous robot that the work can be done according to the manual by doing this movement is provided. Can be generated.
- a person can generate teacher data for learning an autonomous robot that performs autonomous work from a remote control log in which a remote operator in a remote location remotely controls the remote control robot. Even if the altitude is work, it generates teacher data for the autonomous robot to learn, that is, teacher data that can teach the autonomous robot that the work can be done according to the manual when this movement is performed. is there.
- FIG. 1 is a conceptual diagram showing an outline of a teacher data generation system for an autonomous robot according to the present embodiment.
- the server 50 of the teacher data generation system 100 for an autonomous robot can communicate with the remote operator 10, the remote-controlled robot 20, and the sensor 30 via the communication units 16, 24, and 34.
- the remote operator 10 is made to remotely control the remote control robot 20 according to the manual operation (arrow F1), and the server 50 receives the remote control log 26 transmitted from the communication unit 24 of the remote control robot 20. , Acquire the remote control log 26 (arrow F2).
- the remote control referred to here includes, for example, cooking, cleaning, picking up items, washing dishes, and the like, but any other remote control may be used.
- the server 50 acquires the sensor data 36 by receiving the sensor data 36 for the remote control transmitted from the sensor 30 (arrow F3).
- the sensor 30 detects the operation itself of the remote-controlled robot 20 and the result of the operation.
- the operation itself is, for example, a cooking operation, and the result of the operation is the temperature or smell of the cooked food.
- Examples of the sensor 30 include, but are not limited to, an image sensor (camera), a temperature sensor, an odor sensor, and the like.
- the server 50 analyzes the acquired sensor data 36 and determines whether or not the remote control has been performed according to the manual. For example, image analysis is performed to determine whether or not the result is as per the manual, and temperature analysis is performed to determine whether or not the temperature is as per the manual. This determination may be made automatically by comparing the manual with the analysis result, or may be determined by a person. When a person makes a judgment, it can be performed when the manual is used, such as looking at the thermometer, looking at the value of the odor sensor, and looking at the value of the taste sensor.
- the server 50 records the remote operation log corresponding to the remote operation among the acquired remote operation logs 26 as a success log when it is determined as the manual as a result of the determination. Then, based on the recorded success log and sensor data, teacher data for learning by an autonomous robot that autonomously performs the work according to the manual is generated. Specifically, it is possible to teach the autonomous robot that this movement can be performed according to the manual.
- the success log but also the remote operation log that does not follow the manual may be recorded as a failure log, and the teacher data for failure may be generated based on the failure log and the sensor data. Specifically, if this movement does not work according to the manual, the autonomous robot can be informed of the failure. Basically, only success is used as teacher data, so teacher data for failure may be generated as needed.
- the server 50 feeds back the sensor data (image, detected value, etc.) 36 to the terminal of the remote operator 10, and allows the remote operator 10 to visually check whether or not the work can be performed according to the manual. Good (arrow F4).
- the server 50 transmits an instruction to the sensor 30, the sensor 30 that has received the instruction transmits the sensor data 36 to the terminal of the remote operator 10 and feeds back the sensor data 36.
- the remote operator 10 receives the feedback sensor data 36 and the received sensor data 36 is displayed as an image on the headset 14, the remote operator 10 can work according to the manual while visually observing the image. Check if. For example, when the manual says that the onion is cut into 1 mm square, the operation is performed while visually checking the fed-back camera image to see if the onion is actually cut into 1 mm square. This makes it possible to improve the accuracy of remote control.
- the remote operator 10 remotely controls the remote control robot 20 using the operation side equipment (terminal) 12 according to the manual.
- the operation side equipment 12 includes an operation unit 14 and a communication unit 16.
- the operation unit 14 includes, but is not limited to, the headset 14A and the like described above.
- the communication unit 16 transmits operation data to the remote-controlled robot 20 via a network, and receives sensor data 36 from the sensor 30.
- the remote-controlled robot 20 is a remote controlled robot 20 performed by the remote-controlled robot 10 using the operating-side equipment 12 at a place away from the remote-controlled robot 10.
- the remote control type robot 20 includes a drive unit 22, a communication unit 24, and a remote control log 26.
- the drive unit 22 is a portion that is actually driven based on the remote control of the remote operator 10.
- the communication unit 24 receives a remote control instruction of the remote operator 10 via a network, and transmits the remote control log 26 to the server 50.
- the remote control log 26 is an operation log of the remote control robot 20, and is stored in a storage or the like (not shown).
- the sensor 30 includes a detection unit 32, a communication unit 34, and sensor data 36.
- the detection unit 32 detects the operation of the remote-controlled robot 20 and the result of the operation. For example, a camera acquires an image, and a temperature sensor detects a temperature.
- the communication unit 34 transmits the sensor data 36 to the server 50 via the network, and transmits the sensor data 36 to the terminal of the remote operator 10 based on the instruction from the server 50.
- the sensor data 36 is a detection result by the detection unit 32 and is stored in a storage or the like (not shown).
- the server 50 includes a processor 52, a memory 54, a storage 56, and a communication unit 58, which are connected by a bus (not shown).
- the processor 52 is configured by, for example, a CPU (Central Processing Unit), and performs various processes by reading and executing various programs stored in the memory 54.
- the memory 54 stores a program to be executed by the CPU 52, and is composed of, for example, a ROM (Read Only Memory) or a RAM (Random Access Memory). For example, various means shown in FIG. 4 are stored.
- the storage 56 stores remote control log 58, sensor data 60, success log 62, failure log 64, teacher data 66, a control program (not shown), and the like.
- the communication unit 70 receives the remote operation log 26 from the remote operation type robot 20 and the sensor data 36 from the sensor 30 via the network, thereby acquiring the remote operation log 26 and the sensor data 36, and the sensor 30. Instruct data is transmitted to the remote operator 10 to provide the sensor data 36. Of course, other data may be acquired or provided as needed.
- the server 50 includes a first acquisition means 80, a second acquisition means 82, a determination means 84, a recording means 86, a generation means 88, and a feedback means 90.
- the first acquisition means 80 acquires the remote control log 26 in which the remote operator 10 at a remote location remotely controls the remote control robot 20 according to the manual. Specifically, the remote control log 26 transmitted from the communication unit 24 of the remote control robot 20 is received. The first acquisition means 80 receives the remote operation log and stores it in the storage 56 of the server 50 as the remote operation log 58.
- the remote control includes, but is not limited to, operations such as cooking, cleaning, taking out goods, and washing dishes.
- the second acquisition means 82 acquires the sensor data 36 for remote control.
- the sensor data 36 transmitted from the communication unit 34 of the sensor 30 is acquired by the second acquisition means 82, and the acquired sensor data is stored in the storage 56 of the server 50 as the sensor data 60.
- the sensor data is an image or a detected value detected by an image sensor (camera), a temperature sensor, an odor sensor, or the like.
- the determination means 84 analyzes the sensor data 60 acquired by the second acquisition means 82, and determines whether or not the remote control has been performed according to the manual. For example, it is determined whether the result is as per the manual by performing image analysis, or whether the temperature is as per the manual by performing temperature analysis. In addition, this determination may be made by a person as needed. When looking at a thermometer, looking at the value of an odor sensor, looking at the value of a taste sensor, etc. are manualized, human judgment can be made.
- the recording means 86 When the remote operation is determined to be as per the manual as a result of the determination by the determination means 84, the recording means 86 records the remote operation log corresponding to the remote operation among the acquired remote operation logs 58. It is recorded in the storage 56 of the server 50 as the success log 62. Further, when it is determined that the remote operation is not according to the manual as a result of the determination by the determination means 84, the recording means 86 is the remote corresponding to the remote operation among the acquired remote operation logs 58. The operation log is recorded in the storage 56 of the server 50 as the failure log 64. In the teacher data generation system 100 for an autonomous robot of the present embodiment, since only success is taught to the autonomous robot, the failure log 64 may be recorded as needed.
- the generation means 88 is based on the success log 62 recorded by the recording means 86, and the teacher data for learning from the success log 62 and the sensor data 60 to an autonomous robot that autonomously performs the work according to the manual.
- the generated teacher data 66 is stored in the storage 56 of the server 50. This teacher data is teacher data for success, and it is possible to teach the autonomous robot that the work can be performed according to the manual by making this movement.
- the generation means 88 generates failure teacher data for making the autonomous robot learn the failure from the failure log 64 and the sensor data 60 based on the failure log 64 recorded by the recording means 86. To do.
- the generated teacher data 66 is stored in the storage 56 of the server 50.
- the teacher data for this failure can tell the autonomous robot the failure if the work does not follow the manual when this movement is performed.
- teacher data for failure may be generated as necessary.
- the feedback means 90 feeds back the sensor data 36 to the operating side equipment 12 of the remote operator 10 so that the remote operator 10 can visually check whether or not the work is performed according to the manual.
- the feedback means 90 transmits a feedback instruction to the sensor 30, and the sensor 30 that receives the instruction transmits the sensor data 36 to the communication unit 16 of the operation side equipment 12 of the remote operator 10. Feedback is given. For example, if the manual cuts the onion into 1 mm squares, the accuracy of the remote control is improved by feeding back the camera image to the remote operator 10 to see if the onions are actually cut into 1 mm squares. be able to.
- the server 50 instructs the sensor 30 to feed back the sensor data 36 directly from the sensor 30 to the remote operator 10, but the server 50 sends the sensor data 60 to the remote operator 10. It may be configured to give feedback.
- FIG. 4 is a flowchart showing an example of the teacher data generation process for the autonomous robot of the present embodiment.
- the first acquisition means 80 of the server 50 receives the remote control log 26 transmitted from the communication unit 24 of the remote control robot 20 and acquires the remote control log 26 (step S10).
- the remote control log is a log in which a remote operator 10 at a remote location remotely controls the remote control robot 20 according to a manual.
- the acquired remote control log is stored in the storage 56 of the server 50 as the remote control log 58.
- the remote control referred to here includes, for example, cooking, cleaning, picking up items, washing dishes, and the like, but any other remote control may be used.
- the second acquisition means 82 of the server 50 acquires the sensor data 36 by receiving the sensor data 36 for the remote control transmitted from the sensor 30 (step S12).
- the acquired sensor data 36 is stored as sensor data 60 in the storage 56 of the server 50.
- the sensor 30 detects the operation itself of the remote-controlled robot 20 and the result of the operation.
- the operation itself is, for example, a cooking operation, and the result of the operation is the temperature or smell of the cooked food. is there.
- Examples of the sensor 30 include, but are not limited to, an image sensor (camera), a temperature sensor, an odor sensor, and the like.
- the second acquisition means 80 acquires the sensor data 36 detected by such a sensor 30.
- the determination means 84 of the server 50 analyzes the acquired sensor data 60 and determines whether or not the remote control has been performed according to the manual (step S14). For example, image analysis is performed to determine whether or not the result is as per the manual, and temperature analysis is performed to determine whether or not the temperature is as per the manual. This determination may be made automatically by comparing the manual with the analysis result, or may be determined by a person. When a person makes a judgment, it can be performed when the manual is used, such as looking at the thermometer, looking at the value of the odor sensor, and looking at the value of the taste sensor.
- the recording means 86 of the server 50 succeeds in the remote operation log corresponding to the remote operation among the acquired remote operation logs 58. It is recorded in the storage 56 as the log 62 (step S16).
- the generation means 88 of the server 50 generates teacher data for learning by the autonomous robot that autonomously performs the work according to the manual based on the recorded success log 62 and the sensor data (step S18). ..
- the generated teacher data 66 is stored in the storage 56. By using such success teacher data, it is possible to teach the autonomous robot that the work can be done according to the manual by making this movement.
- the recording means 86 of the server 50 has the remote operation log corresponding to the remote operation among the acquired remote operation logs 58. Is stored in the storage as a failure log 64 (step S20).
- the generation means 88 of the server 50 generates teacher data for failure for making the autonomous robot learn the failure based on the recorded failure log 64 and the sensor data (step S22).
- the generated teacher data 66 for failure is stored in the storage 56. By using such teacher data for failure, the failure can be taught to the autonomous robot if the work does not follow the manual if this movement is performed. Since the purpose of this embodiment is to teach the autonomous robot only success, the failure log 64 may be recorded and the teacher data for failure may be generated as needed.
- the feedback means 90 of the server 50 feeds back the sensor data (image, detected value, etc.) 36 to the remote operator 10, and the work is performed according to the manual. You may visually check whether it is made.
- the server 50 transmits a feedback instruction to the sensor 30, the sensor 30 that has received the instruction feeds back the sensor data 36 by transmitting the sensor data 36 to the operating side equipment 12 of the remote operator 10.
- the remote operator 10 visually confirms that the work is performed according to the manual while visually observing the image that is fed back and displayed on the headset 14A.
- the accuracy of remote control can be improved by visually checking the feedback camera image to see if the onions are actually cut into 1 mm squares. Can be raised.
- the sensor data 60 may be fed back from the server 50 to the remote operator 10.
- a remote control log 26 in which a remote operator 10 in a remote location remotely controls a remote control robot 20 according to a manual is acquired, and sensor data is obtained from the sensor 30. 36 is acquired, the acquired sensor data is analyzed, and when the remote control is performed according to the manual, among the acquired remote control logs 58, the remote control log corresponding to the remote control is a success log. It was decided to record as 62, and to generate teacher data 66 for making the autonomous robot learn based on the recorded success log 62 and sensor data. Therefore, if this movement is successful, the autonomous robot can be taught, and even if it is an advanced task such as a human being, it is possible to generate teacher data for the autonomous robot to learn.
- the acquired sensor data is analyzed, and if the remote control is not performed according to the manual, the remote control log corresponding to the remote control is displayed among the acquired remote control logs 58. It may be recorded as a failure log 64, and a failure teacher data 66 for making the autonomous robot learn the failure may be generated based on the recorded failure log 64 and the sensor data. According to the teacher data 66 for failure, if this movement fails, the autonomous robot can be taught.
- the present invention may be provided as a program executed on the server 50, or may be provided as a program executed on the edge side.
- the program may be provided as recorded on a computer-readable recording medium or may be downloaded over a network.
- the present invention may also be provided as an invention of the method.
- the present invention it is decided to generate teacher data for making an autonomous robot that performs autonomous work learn from a remote control log in which a remote operator in a remote place remotely controls a remote control robot. Therefore, it is suitable as a system for generating teacher data for making an autonomous robot learn advanced work such as that performed by a human.
Abstract
[Problem] To generate teaching data for causing an autonomous robot to operate even with high-level operations such as are performed by a person. [Solution] A remotely manipulated robot 20 is remotely manipulated, as planned, by a remote manipulator 10 present in a remote location (arrow F1), a remote manipulation log 26 is acquired from the remotely manipulated robot 20 (arrow F2), and sensor data 36 is acquired from a sensor 30 (arrow F3). A server 50: analyzes the acquired sensor data; and, when the remote manipulation is performed as planned, records a remote manipulation log that corresponds to the remote manipulation from among acquired remote manipulations as a success log, and generates teaching data for causing the autonomous robot to learn on the basis of the recorded success log and the sensor data. The server 50 also feeds back the sensor data 36 to the remote manipulator 10 (arrow F4), enabling visual inspection of the fed-back sensor data 36 and improving the precision of remote manipulation.
Description
本発明は、自律型ロボット用教師データ生成コンピュータ、方法及びプログラムに関し、更に具体的には、人が行うような高度な作業を自律型ロボットに学習させるための教師データの生成に関するものである。
The present invention relates to a teacher data generation computer for an autonomous robot, a method and a program, and more specifically, to generate teacher data for making an autonomous robot learn advanced tasks such as those performed by humans.
近年、自律的に作業を行う自律型ロボットが注目されている。例えば、人とロボットが協働して作業を行うロボットに学習させるために、人とロボットが協働して作業を行う期間中に、ロボットの状態を示す状態変数を観測して、人の負担度及び作業効率のうち少なくとも一方に関する判定データを取得して、ロボットの行動を設定するための訓練データセットを学習する技術が提供されている（下記特許文献１）。
In recent years, autonomous robots that work autonomously have been attracting attention. For example, in order to train a robot that works in collaboration with a human and a robot, a state variable indicating the state of the robot is observed during a period in which the human and the robot work in collaboration, and the burden on the human is increased. A technique for learning a training data set for setting a robot's behavior by acquiring judgment data regarding at least one of degree and work efficiency is provided (Patent Document 1 below).
自律型ロボットは、教師データを用いて学習することで、自律的に作業を行うことができるようになる。しかしながら、人が行うような高度な作業の場合、その高度な作業の教師データを生成することが難しい。上述した特許文献１に記載の技術では、人が行うような高度な作業である場合、自律型ロボットに学習させるための教師データを生成することができない。
Autonomous robots will be able to work autonomously by learning using teacher data. However, in the case of advanced work such as that performed by humans, it is difficult to generate teacher data for the advanced work. With the technique described in Patent Document 1 described above, it is not possible to generate teacher data for making an autonomous robot learn when it is an advanced work such as performed by a human.
本発明は、以上の課題に鑑み、人が行うような高度な作業であっても、自律型ロボットに作業させるための教師データを生成することができる自律型ロボット用教師データ生成コンピュータ、方法及びプログラムを提供することを目的とする。
In view of the above problems, the present invention provides a teacher data generation computer for an autonomous robot, a method, and a method capable of generating teacher data for causing an autonomous robot to work even in an advanced work such as that performed by a human. The purpose is to provide a program.
本発明は、遠隔地にいる遠隔操作者に遠隔操作型ロボットを遠隔操作させた遠隔操作ログから、自律的に作業を行う自律型ロボットに学習させるための教師データを生成する自律型ロボット用教師データ生成コンピュータであって、前記遠隔操作者に、マニュアル通りに遠隔操作型ロボットを遠隔操作させた、遠隔操作ログを取得する第１取得手段と、前記遠隔操作に対するセンサデータを取得する第２取得手段と、前記取得されたセンサデータを解析して、前記遠隔操作が、前記マニュアル通りに行われたかどうかを判定する判定手段と、前記判定された結果、前記遠隔操作がマニュアル通りだと判定された場合に、前記取得された遠隔操作ログのうち、当該遠隔操作に対応する遠隔操作ログを成功ログとして記録する記録手段と、前記記録された成功ログを基に、自律的に前記マニュアル通りの作業を行う自律型ロボットに学習させるための教師データを生成する生成手段と、を備える自律型ロボット用教師データ生成コンピュータを提供する。
The present invention is a teacher for an autonomous robot that generates teacher data for making an autonomous robot that performs autonomous work learn from a remote control log in which a remote operator in a remote location remotely controls the remote controlled robot. A data generation computer, the first acquisition means for acquiring a remote operation log by having the remote operator remotely control a remote control robot according to a manual, and the second acquisition for acquiring sensor data for the remote operation. The means, the determination means that analyzes the acquired sensor data and determines whether or not the remote control is performed according to the manual, and as a result of the determination, it is determined that the remote control is according to the manual. In that case, among the acquired remote control logs, the recording means for recording the remote control log corresponding to the remote control as a success log, and based on the recorded success log, autonomously follow the manual. Provided is a teacher data generation computer for an autonomous robot provided with a generation means for generating teacher data for training an autonomous robot performing a work.
また、本発明は、遠隔地にいる遠隔操作者に遠隔操作型ロボットを遠隔操作させた遠隔操作ログから、自律的に作業を行う自律型ロボットに学習させるための教師データを生成する自律型ロボット用教師データ生成方法であって、前記遠隔操作者に、マニュアル通りに遠隔操作型ロボットを遠隔操作させた、遠隔操作ログを取得するステップと、前記遠隔操作に対するセンサデータを取得するステップと、前記取得されたセンサデータを解析して、前記遠隔操作が、前記マニュアル通りに行われたどうかを判定するステップと、前記判定された結果、前記遠隔操作がマニュアル通りだと判定された場合に、前記取得された遠隔操作ログのうち、当該遠隔操作に対応する遠隔操作ログを成功ログとして記録するステップと、前記記録された成功ログを基に、自律的に前記マニュアル通りの作業を行う自律型ロボットに学習させるための教師データを生成するステップと、を備える自律型ロボット用教師データ生成方法を提供する。
Further, the present invention is an autonomous robot that generates teacher data for making an autonomous robot that performs autonomous work learn from a remote control log in which a remote operator in a remote place remotely controls the remote control robot. In the teacher data generation method, a step of acquiring a remote operation log in which the remote operator remotely controls a remote control type robot according to a manual, a step of acquiring sensor data for the remote operation, and the above-mentioned The step of analyzing the acquired sensor data to determine whether or not the remote control was performed according to the manual, and when the determination results in determining that the remote control is according to the manual, the above Of the acquired remote control logs, an autonomous robot that autonomously performs the work according to the manual based on the step of recording the remote control log corresponding to the remote control as a success log and the recorded success log. Provided is a method for generating teacher data for an autonomous robot, which comprises a step of generating teacher data for training.
更に、本発明は、コンピュータに、遠隔地にいる遠隔操作者に遠隔操作型ロボットを遠隔操作させた遠隔操作ログから、自律的に作業を行う自律型ロボットに学習させるための教師データを生成する自律型ロボット用教師データ生成処理を実行させるプログラムであって、前記遠隔操作者に、マニュアル通りに遠隔操作型ロボットを遠隔操作させた、遠隔操作ログを取得するステップと、前記遠隔操作に対するセンサデータを取得するステップと、前記取得されたセンサデータを解析して、前記遠隔操作が、前記マニュアル通りに行われたどうかを判定するステップと、前記判定された結果、前記遠隔操作がマニュアル通りだと判定された場合に、前記取得された遠隔操作ログのうち、当該遠隔操作に対応する遠隔操作ログを成功ログとして記録するステップと、前記記録された成功ログを基に、自律的に前記マニュアル通りの作業を行う自律型ロボットに学習させるための教師データを生成するステップと、を実行させるためのプログラムを提供する。
Further, the present invention generates teacher data for making a remote-controlled robot learn from a remote-controlled log in which a remote-controlled operator at a remote location remotely controls a remote-controlled robot. A program that executes a teacher data generation process for an autonomous robot, in which the remote operator remotely controls the remote-controlled robot according to the manual, a step of acquiring a remote control log, and sensor data for the remote control. The step of acquiring the above, the step of analyzing the acquired sensor data to determine whether or not the remote control was performed according to the manual, and the result of the determination that the remote control is according to the manual. When it is determined, among the acquired remote operation logs, the step of recording the remote operation log corresponding to the remote operation as a success log and the step of recording the recorded success log autonomously according to the manual. It provides a step of generating teacher data for training an autonomous robot that performs the work of, and a program for executing.
本発明によれば、遠隔地にいる遠隔操作者に遠隔操作型ロボットを遠隔操作させた遠隔操作ログから、自律的に作業を行う自律型ロボットに学習させるための教師データを生成することとしたので、人が行うような高度な作業であっても、自律型ロボットに学習させるための教師データ、すなわち、この動きをするとマニュアル通りの作業ができると自律型ロボットに教えることができる教師データを生成することができる。
According to the present invention, it is decided to generate teacher data for making an autonomous robot that performs autonomous work learn from a remote control log in which a remote operator in a remote place remotely controls a remote control robot. Therefore, even if it is an advanced work that a person does, the teacher data for letting the autonomous robot learn, that is, the teacher data that can teach the autonomous robot that the work can be done according to the manual by doing this movement is provided. Can be generated.
本発明は、遠隔地にいる遠隔操作者に遠隔操作型ロボットを遠隔操作させた遠隔操作ログから、自律的に作業を行う自律型ロボットに学習させるための教師データを生成することで、人が行うような高度が作業であっても、自律型ロボットに学習させるための教師データ、すなわち、この動きをするとマニュアル通りの作業ができると自律型ロボットに教えることができる教師データを生成するものである。
According to the present invention, a person can generate teacher data for learning an autonomous robot that performs autonomous work from a remote control log in which a remote operator in a remote location remotely controls the remote control robot. Even if the altitude is work, it generates teacher data for the autonomous robot to learn, that is, teacher data that can teach the autonomous robot that the work can be done according to the manual when this movement is performed. is there.
＜全体構成＞・・・図１は、本実施形態に係る自律型ロボット用教師データ生成システムの概要を示す概念図である。自律型ロボット用教師データ生成システム１００のサーバ５０は、遠隔操作者１０、遠隔操作型ロボット２０及びセンサ３０と、通信部１６、２４、３４を介して通信可能となっている。まず、遠隔操作者１０に、マニュアル操作通りに遠隔操作型ロボット２０を遠隔操作させ（矢印Ｆ１）、サーバ５０は、遠隔操作型ロボット２０の通信部２４から送信された遠隔操作ログ２６を受信し、遠隔操作ログ２６を取得する（矢印Ｆ２）。ここでいう遠隔操作は、例えば、調理、清掃、品出し、皿洗いなどが挙げられるが、その他のどのような遠隔操作であってもよい。
<Overall configuration> ... FIG. 1 is a conceptual diagram showing an outline of a teacher data generation system for an autonomous robot according to the present embodiment. The server 50 of the teacher data generation system 100 for an autonomous robot can communicate with the remote operator 10, the remote-controlled robot 20, and the sensor 30 via the communication units 16, 24, and 34. First, the remote operator 10 is made to remotely control the remote control robot 20 according to the manual operation (arrow F1), and the server 50 receives the remote control log 26 transmitted from the communication unit 24 of the remote control robot 20. , Acquire the remote control log 26 (arrow F2). The remote control referred to here includes, for example, cooking, cleaning, picking up items, washing dishes, and the like, but any other remote control may be used.
また、サーバ５０は、センサ３０から送信された遠隔操作に対するセンサデータ３６を受信することで、センサデータ３６を取得する（矢印Ｆ３）。センサ３０は、遠隔操作型ロボット２０の動作そのものや、動作の結果を検出するもので、動作そのものとしては、例えば、調理する動作、動作の結果としては、調理物の温度やにおいなどである。センサ３０としては、例えば、画像センサ（カメラ）、温度センサ、匂いセンサなどが挙げられるが、これに限定されるものではない。
Further, the server 50 acquires the sensor data 36 by receiving the sensor data 36 for the remote control transmitted from the sensor 30 (arrow F3). The sensor 30 detects the operation itself of the remote-controlled robot 20 and the result of the operation. The operation itself is, for example, a cooking operation, and the result of the operation is the temperature or smell of the cooked food. Examples of the sensor 30 include, but are not limited to, an image sensor (camera), a temperature sensor, an odor sensor, and the like.
サーバ５０は、取得されたセンサデータ３６を解析して、遠隔操作がマニュアル通りに行われたかどうかを判定する。例えば、画像解析を行ってマニュアル通りの結果になっているかどうかや、温度解析してマニュアル通りの温度になっているかどうかなどを判定する。この判定は、マニュアルと解析結果を比較して自動的に行うようにしてもよいし、人が判断してもよい。人が判断する場合、温度計を見る、匂いセンサの値を見る、味覚センサの値を見る、などがマニュアル化されている場合に行うことができる。
The server 50 analyzes the acquired sensor data 36 and determines whether or not the remote control has been performed according to the manual. For example, image analysis is performed to determine whether or not the result is as per the manual, and temperature analysis is performed to determine whether or not the temperature is as per the manual. This determination may be made automatically by comparing the manual with the analysis result, or may be determined by a person. When a person makes a judgment, it can be performed when the manual is used, such as looking at the thermometer, looking at the value of the odor sensor, and looking at the value of the taste sensor.
次に、サーバ５０は、判定された結果、マニュアル通りだと判定された場合に、取得された遠隔操作ログ２６のうち、当該遠隔操作に対応する遠隔操作ログを成功ログとして記録する。そして、記録された成功ログとセンサデータをもとに、自律的にマニュアル通りの作業を行う自律型ロボットに学習させるための教師データを生成する。具体的には、この動きをするとマニュアル通りの作業ができると、自律型ロボットに教えることができる。
Next, the server 50 records the remote operation log corresponding to the remote operation among the acquired remote operation logs 26 as a success log when it is determined as the manual as a result of the determination. Then, based on the recorded success log and sensor data, teacher data for learning by an autonomous robot that autonomously performs the work according to the manual is generated. Specifically, it is possible to teach the autonomous robot that this movement can be performed according to the manual.
また、成功ログのみならず、マニュアル通りでない遠隔操作ログを失敗ログとして記録し、当該失敗ログとセンサデータをもとに、失敗用の教師データを生成してもよい。具体的には、この動きをするとマニュアル通りの作業にならないと、失敗を自律型ロボットに教えることができる。なお、基本的には、成功だけを教師データをするため、失敗用の教師データの生成は、必要に応じて行えばよい。
Further, not only the success log but also the remote operation log that does not follow the manual may be recorded as a failure log, and the teacher data for failure may be generated based on the failure log and the sensor data. Specifically, if this movement does not work according to the manual, the autonomous robot can be informed of the failure. Basically, only success is used as teacher data, so teacher data for failure may be generated as needed.
また、サーバ５０は、遠隔操作者１０の端末に、センサデータ（画像や検出値など）３６を、フィードバックして、マニュアル通りに作業ができているかどうかを、遠隔操作者１０に目視させてもよい（矢印Ｆ４）。サーバ５０がセンサ３０に指示を送信することで、指示を受信したセンサ３０は、センサデータ３６を遠隔操作者１０の端末へ送信し、センサデータ３６をフィードバックする。遠隔操作者１０は、例えば、フィードバックされたセンサデータ３６を受信し、受信したセンサデータ３６が、ヘッドセット１４に画像として表示されると、当該画像を目視しながら、マニュアル通りに作業ができているか確認する。例えば、マニュアルが、玉ねぎを１ｍｍ角に切る、という場合、実際に玉ねぎを１ｍｍ角に切れているかどうかを、フィードバックされたカメラ画像を目視で確認しながら操作を行う。これによって、遠隔操作の精度を上げることができる。
Further, the server 50 feeds back the sensor data (image, detected value, etc.) 36 to the terminal of the remote operator 10, and allows the remote operator 10 to visually check whether or not the work can be performed according to the manual. Good (arrow F4). When the server 50 transmits an instruction to the sensor 30, the sensor 30 that has received the instruction transmits the sensor data 36 to the terminal of the remote operator 10 and feeds back the sensor data 36. For example, when the remote operator 10 receives the feedback sensor data 36 and the received sensor data 36 is displayed as an image on the headset 14, the remote operator 10 can work according to the manual while visually observing the image. Check if. For example, when the manual says that the onion is cut into 1 mm square, the operation is performed while visually checking the fed-back camera image to see if the onion is actually cut into 1 mm square. This makes it possible to improve the accuracy of remote control.
＜遠隔操作者側の構成＞・・・図１に示すように、遠隔操作者１０は、操作側機材（端末）１２を用いて遠隔操作型ロボット２０を、マニュアル通りに遠隔操作する。操作側機材１２には、操作部１４と通信部１６が含まれる。操作部１４としては、上述したヘッドセット１４Ａなどが含まれるが、これに限定されるものではない。通信部１６は、ネットワークを介して遠隔操作型ロボット２０に操作用データを送信したり、センサ３０からセンサデータ３６を受信したりするものである。
<Structure on the remote operator side> ... As shown in FIG. 1, the remote operator 10 remotely controls the remote control robot 20 using the operation side equipment (terminal) 12 according to the manual. The operation side equipment 12 includes an operation unit 14 and a communication unit 16. The operation unit 14 includes, but is not limited to, the headset 14A and the like described above. The communication unit 16 transmits operation data to the remote-controlled robot 20 via a network, and receives sensor data 36 from the sensor 30.
＜遠隔操作型ロボットの構成＞・・・図１に示すように、遠隔操作型ロボット２０は、遠隔操作者１０とは離れた場所において、遠隔操作者１０が操作側機材１２を用いて行う遠隔操作に従って各種の動作を行うものである。遠隔操作型ロボット２０には、駆動部２２、通信部２４、遠隔操作ログ２６が含まれる。駆動部２２は、遠隔操作者１０の遠隔操作に基づいて実際に駆動する部分である。また、通信部２４は、ネットワークを介して遠隔操作者１０の遠隔操作の指示を受信したり、サーバ５０に遠隔操作ログ２６を送信するものである。遠隔操作ログ２６は、遠隔操作型ロボット２０の操作ログであって、図示しないストレージ等に記憶される。
<Structure of remote-controlled robot> ... As shown in FIG. 1, the remote-controlled robot 20 is a remote controlled robot 20 performed by the remote-controlled robot 10 using the operating-side equipment 12 at a place away from the remote-controlled robot 10. Various operations are performed according to the operation. The remote control type robot 20 includes a drive unit 22, a communication unit 24, and a remote control log 26. The drive unit 22 is a portion that is actually driven based on the remote control of the remote operator 10. Further, the communication unit 24 receives a remote control instruction of the remote operator 10 via a network, and transmits the remote control log 26 to the server 50. The remote control log 26 is an operation log of the remote control robot 20, and is stored in a storage or the like (not shown).
＜センサの構成＞・・・センサ３０は、画像センサ（カメラ）、温度センサ、匂いセンサなど各種のものが用いられ、遠隔操作型ロボット２０の動作や、動作の結果について検出を行う。センサ３０には、検知部３２、通信部３４、センサデータ３６が含まれる。検知部３２は、遠隔操作型ロボット２０の動作や、動作の結果について検出を行うものである。例えば、カメラであれば画像を取得し、温度センサであれば温度を検出する。通信部３４は、ネットワークを介してサーバ５０にセンサデータ３６を送信したり、サーバ５０からの指示に基づいて、センサデータ３６を、遠隔操作者１０の端末へ送信したりする。センサデータ３６は、検知部３２による検知結果であって、図示しないストレージ等に記憶される。
<Sensor configuration> ... Various sensors such as an image sensor (camera), a temperature sensor, and an odor sensor are used, and the operation of the remote-controlled robot 20 and the result of the operation are detected. The sensor 30 includes a detection unit 32, a communication unit 34, and sensor data 36. The detection unit 32 detects the operation of the remote-controlled robot 20 and the result of the operation. For example, a camera acquires an image, and a temperature sensor detects a temperature. The communication unit 34 transmits the sensor data 36 to the server 50 via the network, and transmits the sensor data 36 to the terminal of the remote operator 10 based on the instruction from the server 50. The sensor data 36 is a detection result by the detection unit 32 and is stored in a storage or the like (not shown).
＜サーバのハードウェア構成＞・・・次に、図２を参照して、サーバ５０のハードウェア構成を説明する。サーバ５０は、プロセッサ５２、メモリ５４、ストレージ５６、通信部５８を備え、これらは図示しないバスにより接続されている。プロセッサ５２は、例えば、ＣＰＵ（Central Processing Unit）により構成され、メモリ５４に記憶された各種プログラムを読み出して実行することで、各種処理を行う。前記メモリ５４は、ＣＰＵ５２により実行させるプログラムを記憶するものであり、例えば、ＲＯＭ（Read Only Memory）やＲＡＭ（Random Access Memory）により構成される。例えば、図４に示す各種手段が記憶されている。ストレージ５６は、遠隔操作ログ５８、センサデータ６０、成功ログ６２、失敗ログ６４、教師データ６６や、図示しない制御プログラムなどを記憶するものである。
<Server hardware configuration> ... Next, the hardware configuration of the server 50 will be described with reference to FIG. The server 50 includes a processor 52, a memory 54, a storage 56, and a communication unit 58, which are connected by a bus (not shown). The processor 52 is configured by, for example, a CPU (Central Processing Unit), and performs various processes by reading and executing various programs stored in the memory 54. The memory 54 stores a program to be executed by the CPU 52, and is composed of, for example, a ROM (Read Only Memory) or a RAM (Random Access Memory). For example, various means shown in FIG. 4 are stored. The storage 56 stores remote control log 58, sensor data 60, success log 62, failure log 64, teacher data 66, a control program (not shown), and the like.
通信部７０は、ネットワークを介して、遠隔操作型ロボット２０から遠隔操作ログ２６を受信し、センサ３０からセンサデータ３６を受信することで、遠隔操作ログ２６やセンサデータ３６を取得し、センサ３０に遠隔操作者１０へセンサデータ３６を提供するよう指示データを送信したりするものである。むろん、必要に応じて、他のデータの取得や提供を行うようにしてもよい。
The communication unit 70 receives the remote operation log 26 from the remote operation type robot 20 and the sensor data 36 from the sensor 30 via the network, thereby acquiring the remote operation log 26 and the sensor data 36, and the sensor 30. Instruct data is transmitted to the remote operator 10 to provide the sensor data 36. Of course, other data may be acquired or provided as needed.
＜サーバの機能構成＞・・・次に、図４を参照して、サーバ５０の機能構成を説明する。サーバ５０は、第１取得手段８０と、第２取得手段８２と、判定手段８４と、記録手段８６と、生成手段８８と、フィードバック手段９０を備えている。
<Functional configuration of server> ... Next, the functional configuration of the server 50 will be described with reference to FIG. The server 50 includes a first acquisition means 80, a second acquisition means 82, a determination means 84, a recording means 86, a generation means 88, and a feedback means 90.
第１取得手段８０は、遠隔地にいる遠隔操作者１０に、マニュアル通りに遠隔操作型ロボット２０を遠隔操作させた遠隔操作ログ２６を取得するものである。具体的には、遠隔操作型ロボット２０の通信部２４から送信された遠隔操作ログ２６を受信する。第１取得手段８０は、遠隔操作ログを受信し、サーバ５０のストレージ５６に遠隔操作ログ５８として記憶する。なお、遠隔操作は、例えば、調理、清掃、品出し、皿洗いなどの操作が挙げられるが、これらに限定されるものではない。
The first acquisition means 80 acquires the remote control log 26 in which the remote operator 10 at a remote location remotely controls the remote control robot 20 according to the manual. Specifically, the remote control log 26 transmitted from the communication unit 24 of the remote control robot 20 is received. The first acquisition means 80 receives the remote operation log and stores it in the storage 56 of the server 50 as the remote operation log 58. The remote control includes, but is not limited to, operations such as cooking, cleaning, taking out goods, and washing dishes.
第２取得手段８２は、遠隔操作に対するセンサデータ３６を取得するものである。センサ３０の通信部３４から送信されたセンサデータ３６を、第２取得手段８２が受信することで取得し、取得したセンサデータを、サーバ５０のストレージ５６にセンサデータ６０として記憶する。なお、センサデータは、画像センサ（カメラ）、温度センサ、匂いセンサが検出した画像ないし検出値などである。
The second acquisition means 82 acquires the sensor data 36 for remote control. The sensor data 36 transmitted from the communication unit 34 of the sensor 30 is acquired by the second acquisition means 82, and the acquired sensor data is stored in the storage 56 of the server 50 as the sensor data 60. The sensor data is an image or a detected value detected by an image sensor (camera), a temperature sensor, an odor sensor, or the like.
判定手段８４は、前記第２取得手段８２により取得されたセンサデータ６０を解析して、前記遠隔操作が、マニュアル通りに行われたかどうかを判定する。例えば、画像解析をしてマニュアル通りの結果になっているか、温度解析をしてマニュアル通りの温度になっているか、などを判定する。なお、この判定は、必要に応じて人が行ってもよい。温度計を見る、匂いセンサの値を見る、味覚センサの値を見る、などがマニュアル化されているなどの場合に人による判定を行うことができる。
The determination means 84 analyzes the sensor data 60 acquired by the second acquisition means 82, and determines whether or not the remote control has been performed according to the manual. For example, it is determined whether the result is as per the manual by performing image analysis, or whether the temperature is as per the manual by performing temperature analysis. In addition, this determination may be made by a person as needed. When looking at a thermometer, looking at the value of an odor sensor, looking at the value of a taste sensor, etc. are manualized, human judgment can be made.
記録手段８６は、前記判定手段８４により判定された結果、前記遠隔操作がマニュアル通りだと判定された場合に、前記取得された遠隔操作ログ５８のうち、当該遠隔操作に対応する遠隔操作ログを成功ログ６２としてサーバ５０のストレージ５６に記録する。また、記録手段８６は、前記判定手段８４により判定された結果、前記遠隔操作がマニュアル通りではないと判定された場合に、前記取得された遠隔操作ログ５８のうち、当該遠隔操作に対応する遠隔操作ログを失敗ログ６４としてサーバ５０のストレージ５６に記録する。なお、本実施形態の自律型ロボット用教師データ生成システム１００では、成功のみを自律型ロボットに教えるため、失敗ログ６４の記録は必要に応じて行えばよい。
When the remote operation is determined to be as per the manual as a result of the determination by the determination means 84, the recording means 86 records the remote operation log corresponding to the remote operation among the acquired remote operation logs 58. It is recorded in the storage 56 of the server 50 as the success log 62. Further, when it is determined that the remote operation is not according to the manual as a result of the determination by the determination means 84, the recording means 86 is the remote corresponding to the remote operation among the acquired remote operation logs 58. The operation log is recorded in the storage 56 of the server 50 as the failure log 64. In the teacher data generation system 100 for an autonomous robot of the present embodiment, since only success is taught to the autonomous robot, the failure log 64 may be recorded as needed.
生成手段８８は、前記記録手段８６によって記録された成功ログ６２をもとに、前記成功ログ６２とセンサデータ６０から、自律的にマニュアル通りの作業を行う自律型ロボットに学習させるための教師データを生成する。生成された教師データ６６は、サーバ５０のストレージ５６に記憶される。この教師データは、成功用の教師データであって、この動きをするとマニュアル通りの作業ができると、自律型ロボットに教えることができる。
The generation means 88 is based on the success log 62 recorded by the recording means 86, and the teacher data for learning from the success log 62 and the sensor data 60 to an autonomous robot that autonomously performs the work according to the manual. To generate. The generated teacher data 66 is stored in the storage 56 of the server 50. This teacher data is teacher data for success, and it is possible to teach the autonomous robot that the work can be performed according to the manual by making this movement.
また、生成手段８８は、前記記録手段８６によって記録された失敗ログ６４をもとに、前記失敗ログ６４とセンサデータ６０から、自律型ロボットに失敗を学習させるための失敗用の教師データを生成する。生成された教師データ６６は、サーバ５０のストレージ５６に記憶される。この失敗用の教師データは、この動きをするとマニュアル通りの作業にならないと、失敗を自律型ロボットに教えることができる。なお、本実施形態の自律型ロボット用教師データ生成システム１００では、成功のみを自律型ロボットに教えるため、失敗用の教師データの生成は必要に応じて行えばよい。
Further, the generation means 88 generates failure teacher data for making the autonomous robot learn the failure from the failure log 64 and the sensor data 60 based on the failure log 64 recorded by the recording means 86. To do. The generated teacher data 66 is stored in the storage 56 of the server 50. The teacher data for this failure can tell the autonomous robot the failure if the work does not follow the manual when this movement is performed. In the teacher data generation system 100 for an autonomous robot of the present embodiment, since only success is taught to the autonomous robot, teacher data for failure may be generated as necessary.
フィードバック手段９０は、遠隔操作者１０の操作側機材１２に、センサデータ３６をフィードバックして、遠隔操作者１０にマニュアル通りに作業ができているかどうかを目視させるものである。具体的には、フィードバック手段９０が、センサ３０にフィードバックの指示を送信し、指示を受信したセンサ３０が、遠隔操作者１０の操作側機材１２の通信部１６にセンサデータ３６を送信することでフィードバックが行われる。例えば、マニュアルが、玉ねぎを１ｍｍ角に切る、の場合、実際に玉ねぎを１ｍｍ角に切れているのかどうか、カメラ画像をフィードバックして遠隔操作者１０に目視させることで、遠隔操作の精度を上げることができる。なお、図１の例では、センサ３０から直接センサデータ３６を遠隔操作者１０にフィードバックするようにサーバ５０がセンサ３０に指示することとしているが、サーバ５０からセンサデータ６０を遠隔操作者１０へフィードバックする構成としてもよい。
The feedback means 90 feeds back the sensor data 36 to the operating side equipment 12 of the remote operator 10 so that the remote operator 10 can visually check whether or not the work is performed according to the manual. Specifically, the feedback means 90 transmits a feedback instruction to the sensor 30, and the sensor 30 that receives the instruction transmits the sensor data 36 to the communication unit 16 of the operation side equipment 12 of the remote operator 10. Feedback is given. For example, if the manual cuts the onion into 1 mm squares, the accuracy of the remote control is improved by feeding back the camera image to the remote operator 10 to see if the onions are actually cut into 1 mm squares. be able to. In the example of FIG. 1, the server 50 instructs the sensor 30 to feed back the sensor data 36 directly from the sensor 30 to the remote operator 10, but the server 50 sends the sensor data 60 to the remote operator 10. It may be configured to give feedback.
＜自律型ロボット用教師データ生成処理＞・・・次に、自律型ロボット用教師データ生成システム１００による自律型ロボット用教師データ生成処理の一例について、図４も参照して説明する。図４は、本実施形態の自律型ロボット用教師データ生成処理の一例を示すフローチャートである。まず、サーバ５０の第１取得手段８０は、遠隔操作型ロボット２０の通信部２４から送信された遠隔操作ログ２６を受信し、遠隔操作ログ２６を取得する（ステップＳ１０）。遠隔操作ログとは、遠隔地にいる遠隔操作者１０に、マニュアル通りに遠隔操作型ロボット２０を遠隔操作させたログである。取得した遠隔操作ログは、サーバ５０のストレージ５６に、遠隔操作ログ５８として記憶される。なお、ここでいう遠隔操作は、例えば、調理、清掃、品出し、皿洗いなどが挙げられるが、その他のどのような遠隔操作であってもよい。
<Teacher data generation process for an autonomous robot> ... Next, an example of a teacher data generation process for an autonomous robot by the teacher data generation system 100 for an autonomous robot will be described with reference to FIG. FIG. 4 is a flowchart showing an example of the teacher data generation process for the autonomous robot of the present embodiment. First, the first acquisition means 80 of the server 50 receives the remote control log 26 transmitted from the communication unit 24 of the remote control robot 20 and acquires the remote control log 26 (step S10). The remote control log is a log in which a remote operator 10 at a remote location remotely controls the remote control robot 20 according to a manual. The acquired remote control log is stored in the storage 56 of the server 50 as the remote control log 58. The remote control referred to here includes, for example, cooking, cleaning, picking up items, washing dishes, and the like, but any other remote control may be used.
次に、サーバ５０の第２取得手段８２は、センサ３０から送信された遠隔操作に対するセンサデータ３６を受信することで、センサデータ３６を取得する（ステップＳ１２）。取得したセンサデータ３６は、サーバ５０のストレージ５６にセンサデータ６０として記憶される。なお、センサ３０は、遠隔操作型ロボット２０の動作そのものや、動作の結果を検出するもので、動作そのものとしては、例えば、調理する動作、動作の結果としては、調理物の温度やにおいなどである。センサ３０としては、例えば、画像センサ（カメラ）、温度センサ、匂いセンサなどが挙げられるが、これに限定されるものではない。第２取得手段８０は、このようなセンサ３０により検出されたセンサデータ３６を取得する。
Next, the second acquisition means 82 of the server 50 acquires the sensor data 36 by receiving the sensor data 36 for the remote control transmitted from the sensor 30 (step S12). The acquired sensor data 36 is stored as sensor data 60 in the storage 56 of the server 50. The sensor 30 detects the operation itself of the remote-controlled robot 20 and the result of the operation. The operation itself is, for example, a cooking operation, and the result of the operation is the temperature or smell of the cooked food. is there. Examples of the sensor 30 include, but are not limited to, an image sensor (camera), a temperature sensor, an odor sensor, and the like. The second acquisition means 80 acquires the sensor data 36 detected by such a sensor 30.
次に、サーバ５０の判定手段８４は、取得されたセンサデータ６０を解析して、遠隔操作がマニュアル通りに行われたかどうかを判定する（ステップＳ１４）。例えば、画像解析を行ってマニュアル通りの結果になっているかどうかや、温度解析してマニュアル通りの温度になっているかどうかなどを判定する。この判定は、マニュアルと解析結果を比較して自動的に行うようにしてもよいし、人が判断してもよい。人が判断する場合、温度計を見る、匂いセンサの値を見る、味覚センサの値を見る、などがマニュアル化されている場合に行うことができる。
Next, the determination means 84 of the server 50 analyzes the acquired sensor data 60 and determines whether or not the remote control has been performed according to the manual (step S14). For example, image analysis is performed to determine whether or not the result is as per the manual, and temperature analysis is performed to determine whether or not the temperature is as per the manual. This determination may be made automatically by comparing the manual with the analysis result, or may be determined by a person. When a person makes a judgment, it can be performed when the manual is used, such as looking at the thermometer, looking at the value of the odor sensor, and looking at the value of the taste sensor.
前記判定手段８４により、マニュアル通りだと判定された場合（ステップＳ１４でＹｅｓ）、サーバ５０の記録手段８６は、取得された遠隔操作ログ５８のうち、当該遠隔操作に対応する遠隔操作ログを成功ログ６２としてストレージ５６に記録する（ステップＳ１６）。
When it is determined by the determination means 84 that the manual is correct (Yes in step S14), the recording means 86 of the server 50 succeeds in the remote operation log corresponding to the remote operation among the acquired remote operation logs 58. It is recorded in the storage 56 as the log 62 (step S16).
そして、サーバ５０の生成手段８８は、記録された成功ログ６２とセンサデータをもとに、自律的にマニュアル通りの作業を行う自律型ロボットに学習させるための教師データを生成する（ステップＳ１８）。生成された教師データ６６は、ストレージ５６に記憶される。なお、このような成功用の教師データを用いることで、この動きをするとマニュアル通りの作業ができると、自律型ロボットに教えることができる。
Then, the generation means 88 of the server 50 generates teacher data for learning by the autonomous robot that autonomously performs the work according to the manual based on the recorded success log 62 and the sensor data (step S18). .. The generated teacher data 66 is stored in the storage 56. By using such success teacher data, it is possible to teach the autonomous robot that the work can be done according to the manual by making this movement.
一方、前記判定手段８４により、マニュアル通りでないと判定された場合（ステップＳ１４でＮｏ）、サーバ５０の記録手段８６は、取得された遠隔操作ログ５８のうち、当該遠隔操作に対応する遠隔操作ログを失敗ログ６４としてストレージに記憶する（ステップＳ２０）。
On the other hand, when it is determined by the determination means 84 that the procedure is not as per the manual (No in step S14), the recording means 86 of the server 50 has the remote operation log corresponding to the remote operation among the acquired remote operation logs 58. Is stored in the storage as a failure log 64 (step S20).
そして、サーバ５０の生成手段８８は、記録された失敗ログ６４とセンサデータをもとに、自律型ロボットに失敗を学習させるための失敗用の教師データを生成する（ステップＳ２２）。生成された失敗用の教師データ６６は、ストレージ５６に記憶される。なお、このような失敗用の教師データを用いることで、この動きをするとマニュアル通りの作業にならないと、失敗を自律型ロボットに教えることができる。なお、本実施形態では、自律型ロボットに成功のみを教えること目的とするため、失敗ログ６４の記録や、失敗用の教師データの生成は、必要に応じて行えばよい。
Then, the generation means 88 of the server 50 generates teacher data for failure for making the autonomous robot learn the failure based on the recorded failure log 64 and the sensor data (step S22). The generated teacher data 66 for failure is stored in the storage 56. By using such teacher data for failure, the failure can be taught to the autonomous robot if the work does not follow the manual if this movement is performed. Since the purpose of this embodiment is to teach the autonomous robot only success, the failure log 64 may be recorded and the teacher data for failure may be generated as needed.
以上のような自律型ロボット用教師データ生成処理の間、サーバ５０のフィードバック手段９０は、遠隔操作者１０に、センサデータ（画像や検出値など）３６を、フィードバックして、マニュアル通りに作業ができているかどうかを目視させてもよい。サーバ５０がセンサ３０にフィードバックの指示を送信すると、指示を受信したセンサ３０は、遠隔操作者１０の操作側機材１２にセンサデータ３６を送信することでセンサデータ３６のフィードバックを行う。遠隔操作者１０は、例えば、フィードバックされてヘッドセット１４Ａに表示された画像を目視しながら、マニュアル通りに作業ができているか確認する。例えば、マニュアルが、玉ねぎを１ｍｍ角に切る、という場合、実際に玉ねぎを１ｍｍ角に切れているかどうかを、フィードバックされたカメラ画像を目視で確認しながら操作を行うことで、遠隔操作の精度を上げることができる。なお、サーバ５０から遠隔操作者１０にセンサデータ６０をフィードバックする構成としてもよい。
During the teacher data generation process for the autonomous robot as described above, the feedback means 90 of the server 50 feeds back the sensor data (image, detected value, etc.) 36 to the remote operator 10, and the work is performed according to the manual. You may visually check whether it is made. When the server 50 transmits a feedback instruction to the sensor 30, the sensor 30 that has received the instruction feeds back the sensor data 36 by transmitting the sensor data 36 to the operating side equipment 12 of the remote operator 10. For example, the remote operator 10 visually confirms that the work is performed according to the manual while visually observing the image that is fed back and displayed on the headset 14A. For example, if the manual says that the onion is cut into 1 mm squares, the accuracy of remote control can be improved by visually checking the feedback camera image to see if the onions are actually cut into 1 mm squares. Can be raised. The sensor data 60 may be fed back from the server 50 to the remote operator 10.
＜効果＞・・・以上説明した実施形態によれば、遠隔地にいる遠隔操作者１０に遠隔操作型ロボット２０をマニュアル通りに遠隔操作させた遠隔操作ログ２６を取得し、センサ３０からセンサデータ３６を取得し、取得したセンサデータを解析して、遠隔操作がマニュアル通りに行われている場合には、取得された遠隔操作ログ５８のうち、当該遠隔操作に対応する遠隔操作ログを成功ログ６２として記録し、記録された成功ログ６２とセンサデータをもとに、自律型ロボットに学習させるための教師データ６６を生成することとした。このため、この動きをすると成功すると自律型ロボットに教えることができ、人が行うような高度な作業であっても、自律型ロボットに学習させるための教師データを生成できる。
<Effect> ... According to the embodiment described above, a remote control log 26 in which a remote operator 10 in a remote location remotely controls a remote control robot 20 according to a manual is acquired, and sensor data is obtained from the sensor 30. 36 is acquired, the acquired sensor data is analyzed, and when the remote control is performed according to the manual, among the acquired remote control logs 58, the remote control log corresponding to the remote control is a success log. It was decided to record as 62, and to generate teacher data 66 for making the autonomous robot learn based on the recorded success log 62 and sensor data. Therefore, if this movement is successful, the autonomous robot can be taught, and even if it is an advanced task such as a human being, it is possible to generate teacher data for the autonomous robot to learn.
また、必要に応じて、取得したセンサデータを解析して、遠隔操作がマニュアル通りに行われていない場合には、取得された遠隔操作ログ５８のうち、当該遠隔操作に対応する遠隔操作ログを失敗ログ６４として記録し、記録された失敗ログ６４とセンサデータをもとに、自律型ロボットに失敗を学習させるための失敗用の教師データ６６を生成してもよい。失敗用の教師データ６６により、この動きをすると失敗すると、自律型ロボットに教えることができる。
In addition, if necessary, the acquired sensor data is analyzed, and if the remote control is not performed according to the manual, the remote control log corresponding to the remote control is displayed among the acquired remote control logs 58. It may be recorded as a failure log 64, and a failure teacher data 66 for making the autonomous robot learn the failure may be generated based on the recorded failure log 64 and the sensor data. According to the teacher data 66 for failure, if this movement fails, the autonomous robot can be taught.
また、遠隔操作者１０に、センサデータ３６をフィードバックして、マニュアル通りに作業ができているかを目視させることで、遠隔操作の精度を向上させることが可能である。
Further, it is possible to improve the accuracy of remote control by feeding back the sensor data 36 to the remote operator 10 and visually confirming whether the work is performed according to the manual.
なお、上述した実施形態は一例であり、同様の効果を奏する範囲内で適宜変更が可能である。また、本発明は、サーバ５０で実行されるプログラムとして提供されていもよいし、エッジ側で実行されるプログラムとして提供されてもよい。このプログラムは、コンピュータが読取可能な記録媒体に記録された状態で提供されていてもよいし、ネットワークを介してダウンロードしてもよい。また、本発明は、方法の発明として提供されてもよい。
Note that the above-described embodiment is an example, and can be appropriately changed within a range in which the same effect is obtained. Further, the present invention may be provided as a program executed on the server 50, or may be provided as a program executed on the edge side. The program may be provided as recorded on a computer-readable recording medium or may be downloaded over a network. The present invention may also be provided as an invention of the method.
本発明によれば、遠隔地にいる遠隔操作者に遠隔操作型ロボットを遠隔操作させた遠隔操作ログから、自律的に作業を行う自律型ロボットに学習させるための教師データを生成することとしたので、人が行うような高度な作業を、自律型ロボットに学習させるための教師データを生成するためのシステムとして好適である。
According to the present invention, it is decided to generate teacher data for making an autonomous robot that performs autonomous work learn from a remote control log in which a remote operator in a remote place remotely controls a remote control robot. Therefore, it is suitable as a system for generating teacher data for making an autonomous robot learn advanced work such as that performed by a human.
１０：遠隔操作者
１２：操作側機材
１４：操作部
１４Ａ：ヘッドセット
１６：通信部
２０：遠隔操作型ロボット
２２：駆動部
２４：通信部
２６：遠隔操作ログ
３０：センサ
３２：検知部
３４：通信部
３６：センサデータ
５０：サーバ
５２：プロセッサ
５４：メモリ
５６：ストレージ
５８：遠隔操作ログ
６０：センサデータ
６２：成功ログ
６４：失敗ログ
６６：教師データ
７０：通信部
８０：第１取得手段
８２：第２取得手段
８４：判定手段
８６：記録手段
８８：生成手段
９０：フィードバック手段
１００：自律型ロボット用教師データ生成システム
10: Remote operator 12: Operation side equipment 14: Operation unit 14A: Headset 16: Communication unit 20: Remote operation type robot 22: Drive unit 24: Communication unit 26: Remote operation log 30: Sensor 32: Detection unit 34: Communication unit 36: Sensor data 50: Server 52: Processor 54: Memory 56: Storage 58: Remote operation log 60: Sensor data 62: Success log 64: Failure log 66: Teacher data 70: Communication unit 80: First acquisition means 82 : Second acquisition means 84: Judgment means 86: Recording means 88: Generation means 90: Feedback means 100: Teacher data generation system for autonomous robots
１２：操作側機材
１４：操作部
１４Ａ：ヘッドセット
１６：通信部
２０：遠隔操作型ロボット
２２：駆動部
２４：通信部
２６：遠隔操作ログ
３０：センサ
３２：検知部
３４：通信部
３６：センサデータ
５０：サーバ
５２：プロセッサ
５４：メモリ
５６：ストレージ
５８：遠隔操作ログ
６０：センサデータ
６２：成功ログ
６４：失敗ログ
６６：教師データ
７０：通信部
８０：第１取得手段
８２：第２取得手段
８４：判定手段
８６：記録手段
８８：生成手段
９０：フィードバック手段
１００：自律型ロボット用教師データ生成システム
10: Remote operator 12: Operation side equipment 14: Operation unit 14A: Headset 16: Communication unit 20: Remote operation type robot 22: Drive unit 24: Communication unit 26: Remote operation log 30: Sensor 32: Detection unit 34: Communication unit 36: Sensor data 50: Server 52: Processor 54: Memory 56: Storage 58: Remote operation log 60: Sensor data 62: Success log 64: Failure log 66: Teacher data 70: Communication unit 80: First acquisition means 82 : Second acquisition means 84: Judgment means 86: Recording means 88: Generation means 90: Feedback means 100: Teacher data generation system for autonomous robots
Claims (5)
- 遠隔地にいる遠隔操作者に遠隔操作型ロボットを遠隔操作させた遠隔操作ログから、自律的に作業を行う自律型ロボットに学習させるための教師データを生成する自律型ロボット用教師データ生成コンピュータであって、
前記遠隔操作者に、マニュアル通りに遠隔操作型ロボットを遠隔操作させた、遠隔操作ログを取得する第１取得手段と、
前記遠隔操作に対するセンサデータを取得する第２取得手段と、
前記取得されたセンサデータを解析して、前記遠隔操作が、前記マニュアル通りに行われたかどうかを判定する判定手段と、
前記判定された結果、前記遠隔操作がマニュアル通りだと判定された場合に、前記取得された遠隔操作ログのうち、当該遠隔操作に対応する遠隔操作ログを成功ログとして記録する記録手段と、
前記記録された成功ログを基に、自律的に前記マニュアル通りの作業を行う自律型ロボットに学習させるための教師データを生成する生成手段と、
を備える自律型ロボット用教師データ生成コンピュータ。 A teacher data generation computer for autonomous robots that generates teacher data for learning autonomous robots that work autonomously from remote control logs that allow remote operators in remote locations to remotely control remote-controlled robots. There,
The first acquisition means for acquiring the remote operation log by having the remote operator remotely control the remote control type robot according to the manual,
A second acquisition means for acquiring sensor data for the remote control,
A determination means for analyzing the acquired sensor data and determining whether or not the remote control has been performed according to the manual.
As a result of the determination, when it is determined that the remote operation is as per the manual, the recording means for recording the remote operation log corresponding to the remote operation as a success log among the acquired remote operation logs.
Based on the recorded success log, a generation means for generating teacher data for making an autonomous robot that autonomously performs the work according to the manual learn.
A teacher data generation computer for autonomous robots. - 前記記録手段は、
前記判定手段による判定の結果、前記遠隔操作がマニュアル通りではないと判定された場合に、前記取得された遠隔操作ログのうち、当該遠隔操作に対応する遠隔操作ログを失敗ログとして記録し、
前記生成手段は、
前記記録された失敗ログを基に、自律的に前記マニュアル通りの作業を行う自律型ロボットに、失敗を学習させるための教師データを生成する請求項１記載の自律型ロボット用教師データ生成コンピュータ。 The recording means
As a result of the determination by the determination means, when it is determined that the remote operation is not according to the manual, among the acquired remote operation logs, the remote operation log corresponding to the remote operation is recorded as a failure log.
The generation means
The teacher data generation computer for an autonomous robot according to claim 1, wherein the autonomous robot that autonomously performs the work according to the manual is generated teacher data for learning the failure based on the recorded failure log. - 前記遠隔操作者の端末に、前記センサデータをフィードバックして、前記マニュアル通りに作業ができているかどうかを目視させるフィードバック手段、
を備える請求項１又は２記載の自律型ロボット用教師データ生成コンピュータ。 A feedback means that feeds back the sensor data to the terminal of the remote operator and visually confirms whether or not the work is performed according to the manual.
The teacher data generation computer for an autonomous robot according to claim 1 or 2. - 遠隔地にいる遠隔操作者に遠隔操作型ロボットを遠隔操作させた遠隔操作ログから、自律的に作業を行う自律型ロボットに学習させるための教師データを生成する自律型ロボット用教師データ生成方法であって、
前記遠隔操作者に、マニュアル通りに遠隔操作型ロボットを遠隔操作させた、遠隔操作ログを取得するステップと、
前記遠隔操作に対するセンサデータを取得するステップと、
前記取得されたセンサデータを解析して、前記遠隔操作が、前記マニュアル通りに行われたどうかを判定するステップと、
前記判定された結果、前記遠隔操作がマニュアル通りだと判定された場合に、前記取得された遠隔操作ログのうち、当該遠隔操作に対応する遠隔操作ログを成功ログとして記録するステップと、
前記記録された成功ログを基に、自律的に前記マニュアル通りの作業を行う自律型ロボットに学習させるための教師データを生成するステップと、
を備える自律型ロボット用教師データ生成方法。 A teacher data generation method for autonomous robots that generates teacher data for learning by an autonomous robot that works autonomously from a remote control log that allows a remote operator in a remote location to remotely control a remote-controlled robot. There,
The step of acquiring the remote control log by having the remote operator remotely control the remote control robot according to the manual, and
The step of acquiring sensor data for the remote control and
A step of analyzing the acquired sensor data and determining whether or not the remote control was performed according to the manual.
As a result of the determination, when it is determined that the remote operation is as per the manual, a step of recording the remote operation log corresponding to the remote operation as a success log among the acquired remote operation logs.
Based on the recorded success log, a step of generating teacher data for learning by an autonomous robot that autonomously performs the work according to the manual, and
A teacher data generation method for autonomous robots. - コンピュータに、遠隔地にいる遠隔操作者に遠隔操作型ロボットを遠隔操作させた遠隔操作ログから、自律的に作業を行う自律型ロボットに学習させるための教師データを生成する自律型ロボット用教師データ生成処理を実行させるプログラムであって、
前記遠隔操作者に、マニュアル通りに遠隔操作型ロボットを遠隔操作させた、遠隔操作ログを取得するステップと、
前記遠隔操作に対するセンサデータを取得するステップと、
前記取得されたセンサデータを解析して、前記遠隔操作が、前記マニュアル通りに行われたどうかを判定するステップと、
前記判定された結果、前記遠隔操作がマニュアル通りだと判定された場合に、前記取得された遠隔操作ログのうち、当該遠隔操作に対応する遠隔操作ログを成功ログとして記録するステップと、
前記記録された成功ログを基に、自律的に前記マニュアル通りの作業を行う自律型ロボットに学習させるための教師データを生成するステップと、
を実行させるためのプログラム。
Teacher data for autonomous robots that generates teacher data for making autonomous robots that work autonomously learn from remote control logs that allow a computer to remotely control a remote operator at a remote location. A program that executes the generation process
The step of acquiring the remote control log by having the remote operator remotely control the remote control robot according to the manual, and
The step of acquiring sensor data for the remote control and
A step of analyzing the acquired sensor data and determining whether or not the remote control was performed according to the manual.
As a result of the determination, when it is determined that the remote operation is as per the manual, a step of recording the remote operation log corresponding to the remote operation as a success log among the acquired remote operation logs.
Based on the recorded success log, a step of generating teacher data for learning by an autonomous robot that autonomously performs the work according to the manual, and
A program to execute.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/JP2019/012180 WO2020194392A1 (en) | 2019-03-22 | 2019-03-22 | Computer, method, and program for generating teaching data for autonomous robot |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/JP2019/012180 WO2020194392A1 (en) | 2019-03-22 | 2019-03-22 | Computer, method, and program for generating teaching data for autonomous robot |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2020194392A1 true WO2020194392A1 (en) | 2020-10-01 |
Family
ID=72610390
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/JP2019/012180 WO2020194392A1 (en) | 2019-03-22 | 2019-03-22 | Computer, method, and program for generating teaching data for autonomous robot |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2020194392A1 (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016132398A1 (en) * | 2015-02-19 | 2016-08-25 | 日揮株式会社 | Cell culture processing equipment |
| WO2016189924A1 (en) * | 2015-05-28 | 2016-12-01 | 株式会社日立製作所 | Robot operation device and program |
| JP2017064910A (en) * | 2015-07-31 | 2017-04-06 | ファナック株式会社 | Machine learning device for learning taking-out operation of workpiece, robot system, and machine learning method |
| JP2018161692A (en) * | 2017-03-24 | 2018-10-18 | キヤノン株式会社 | Information processing system, information processing method and program |
-
2019
- 2019-03-22 WO PCT/JP2019/012180 patent/WO2020194392A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016132398A1 (en) * | 2015-02-19 | 2016-08-25 | 日揮株式会社 | Cell culture processing equipment |
| WO2016189924A1 (en) * | 2015-05-28 | 2016-12-01 | 株式会社日立製作所 | Robot operation device and program |
| JP2017064910A (en) * | 2015-07-31 | 2017-04-06 | ファナック株式会社 | Machine learning device for learning taking-out operation of workpiece, robot system, and machine learning method |
| JP2018161692A (en) * | 2017-03-24 | 2018-10-18 | キヤノン株式会社 | Information processing system, information processing method and program |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11701772B2 (en) | Operation prediction system and operation prediction method | |
| US9138893B2 (en) | Robot teaching system, robot teaching assistant device, robot teaching method, and computer-readable recording medium | |
| US9946342B2 (en) | Programmable display device, portable terminal device, data processing method, and program | |
| US11654556B2 (en) | Determination apparatus for determining an operation of an industrial robot | |
| JP2021067981A (en) | Work analysis device and work analysis method | |
| JP2019206041A5 (en) | Information processing equipment, robot control equipment, information processing methods and programs | |
| US20210031358A1 (en) | Behavior estimation apparatus, behavior estimation method, and behavior estimation program | |
| WO2020039616A1 (en) | Autonomous learning robot device and operation generation method for autonomous learning robot device | |
| WO2013027313A1 (en) | Robotic arm control device and control method, robot, robotic arm control program, and integrated electronic circuit | |
| CN111506235A (en) | Control parameter adjusting device | |
| JP2021000672A (en) | Adjustment support device | |
| WO2018113263A1 (en) | Method, system and apparatus for controlling robot, and robot | |
| US20150356780A1 (en) | Method for providing real time guidance to a user and a system thereof | |
| WO2020194392A1 (en) | Computer, method, and program for generating teaching data for autonomous robot | |
| CN111065979B (en) | Plant monitoring device and distributed control system | |
| Kaiser et al. | Robot skill acquisition via human demonstration | |
| Osa et al. | Autonomous penetration detection for bone cutting tool using demonstration-based learning | |
| JP2020113262A (en) | Learned-model generation device, robot control device, and program | |
| JP2009070181A (en) | Operation program creation support system | |
| Zimmerman et al. | Smart Manufacturing and The Promotion of Artificially-Intelligent Human-Robot Collaborations in Small-and Medium-sized Enterprises | |
| EP3811167B1 (en) | Workbench system | |
| Antonello et al. | Force controller tuning for a master-slave system with proximity based haptic feedback | |
| US20190377626A1 (en) | Method for analyzing a cause of at least one deviation | |
| EP3989023A1 (en) | Workbench system | |
| EP3584655A1 (en) | Workbench system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19920829 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 19920829 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: JP |