WO2020032211A1

WO2020032211A1 - Data generating device, data generating method, data generating program, and remote operation system

Info

Publication number: WO2020032211A1
Application number: PCT/JP2019/031495
Authority: WO
Inventors: 康彦橋本; 掃部　雅幸; 繁次田中; 佳彦丸山
Original assignee: 川崎重工業株式会社
Priority date: 2018-08-10
Filing date: 2019-08-08
Publication date: 2020-02-13
Also published as: CN112469538A; KR102518766B1; TW202014278A; KR20210041048A; CN112469538B

Abstract

This data generating device is a data generating device for generating at least some data used to generate an image to be displayed on a display unit, wherein: a working space model which models an actual working space is displayed on the display unit as a moving image; the working space model includes a robot model which models an actual robot, and a peripheral object model which models prescribed peripheral objects at the periphery of the actual robot; the robot model is created in such a way as to act in accordance with operations performed by an operator with respect to an operating device; and the data generating device is provided with a state information acquiring unit which acquires state information indicating the state of the peripheral objects, and a predicting unit which predicts the state of the peripheral objects a prescribed length of time after the current time, on the basis of the state information, and generates the predicted results as peripheral object model data to be used to generate the peripheral object model to be displayed on the display unit.

Description

Data generation device, data generation method, data generation program, and remote operation system

The present invention relates to a data generation device, a data generation method, a data generation program, and a remote operation system in a system in which an operator remotely controls a remote robot through a network.

Conventionally, there is known a system in which an operator remotely controls a robot at a remote place through a network. In this type of system, there is a possibility that a communication delay may occur between the robot and the operating device operated by the operator. For example, Patent Literature 1 discloses a remote operation system that can grasp a communication delay between a slave device and a master device.

遠隔 The remote control system disclosed in Patent Document 1 includes a master device and a slave device. The slave device includes a slave robot that operates according to operation information corresponding to an operator's operation sent from the master device. The slave robot has an imaging device that images its own work environment, and the slave device sequentially shoots the work environment and transmits the photographed real image to the master device. The master device includes a simulator that performs an operation simulation based on operation information sent to the slave robot, and combines a real image sent from the slave device with a simulation image obtained by the operation simulation to display a display image. Generate. The ratio of the synthesis of the real image and the simulation image in the display image is changed according to the communication delay between the master device and the slave device. Specifically, when the communication delay is large, the ratio of the simulation image is increased to generate the display image. The actual image is an image that reflects the background of the work environment as well as the status of the slave robot, whereas the simulation image is an image that displays only the status of the slave robot. For this reason, it is possible for the operator to easily grasp the degree of the communication delay from the display image, for example, whether the background of the work environment appears dark.

JP 2015-47666 A

However, in the system of Patent Document 1, it is necessary for the operator to determine the degree of the communication delay from the combination ratio of the real image and the simulation image, and when it is determined that the communication delay is large, the communication delay is given to the master device. It is necessary to take appropriate measures such as reducing the amount of operation. For this reason, in the above-described system, the operator needs to make a situation determination due to the communication delay in addition to the operation of the robot, and the operator may not be able to concentrate on the operation of the robot.

Accordingly, the present invention provides a data generation device, a data generation method, a data generation program, and a remote operation system that can suppress the influence of a communication delay between an operation terminal operated by an operator and a robot on the operation of the operator. The purpose is to provide.

In order to solve the above problems, a data generation device according to one embodiment of the present invention includes an operation terminal having an operation device that receives an operation of an operator and a display device that can be visually recognized by the operator; A remote operation system including a real robot installed and connected via a network capable of data communication with the operation terminal, wherein at least a part of data used for generation of an image displayed on the display is generated. A working space model obtained by modeling the real working space as a moving image (time-varying image) on the display, and the working space model displays the real robot. A robot model, and a peripheral object model that models an actual peripheral object around the real robot, wherein the robot model is The data generation device is created to operate in response to the operation of the operator, the data generating device acquires a state information acquisition unit that acquires state information indicating the state of the real surroundings, and a predetermined time from the current time based on the state information A prediction unit that predicts a state of the real peripheral object after a minute, and generates a predicted result as peripheral object model data used for creating the peripheral object model displayed on the display device. .

If there is a communication delay between the operating terminal and the real robot, the robot model operates at the same time due to the time lag between when the operator operates the operating device and when the real robot performs the operation corresponding to the operation. And the operation of the real robot may be shifted. According to the configuration, the prediction unit predicts the state of the real peripheral object a predetermined time later than the current time, and the prediction result is used as peripheral object model data used for creating a peripheral object model displayed on the display. Generated. Therefore, the same time lag between the state of the peripheral object model and the state of the real peripheral object as that between the robot model and the real robot can be generated. Thereby, the time axis of the robot model and the time axis of the peripheral object model can be made to coincide with each other, so that the influence of the communication delay on the operation of the operator can be suppressed.

The real peripheral object may include at least one of a work to be worked by the real robot, a transfer device for transferring the work, and a moving device for moving the real robot.

The state information may include imaging information generated by an imaging device installed in the work space imaging the real surroundings.

状態 The state information may include setting information set in a peripheral device as the real peripheral object.

The data generation device described above, the deviation of the state between the state of the work space model displayed on the display at a predetermined time and the state of the real work space after the predetermined time by the predetermined time A deviation detecting unit for detecting the degree may be further provided. For example, when the displacement detected by the displacement detection unit is larger than a predetermined value, it is possible to take measures such as stopping the operation of the actual robot or correcting the model displayed on the display.

The data generation device further includes a model correction unit that corrects the work space model so that the deviation is eliminated when the deviation detected by the deviation detecting unit exceeds a preset range. Is also good.

The operation terminal may be a game device including a controller as the operation device.

The operation terminal may be at least one of a personal information terminal (PDA (Personal Data Assistant)), a smartphone, a personal computer, a tablet, and a remote controller dedicated to a robot.

In addition, a data generation method according to one embodiment of the present invention includes an operation device having an operation device that accepts an operation of an operator and a display device that can be visually recognized by the operator; and an operation terminal installed in a work space; In a remote operation system including a real robot connected via a communicable network, a data generating method for generating at least a part of data used for generating an image displayed on the display. A work space model that models the real work space is displayed as a moving image on the display, and the work space model is located around the real robot and a robot model that models the real robot. A peripheral object model obtained by modeling a peripheral object, wherein the robot model is created so as to operate in accordance with the operation of the operator on the operation device. Wherein the data generation method includes: a state information obtaining step of obtaining state information indicating a state of a peripheral object around the real robot; and predicting a state of the peripheral object after the present time based on the state information. A prediction step of generating a predicted result as peripheral object model data used for creating the peripheral object model displayed on the display.

In addition, a data generation program according to one embodiment of the present invention includes an operation device having an operation device that receives an operation of an operator and a display device that can be visually recognized by the operator; And a real robot connected via a communicable network, wherein at least a part of data used for generating an image displayed on the display is generated by being executed by a computer. A data generation program, wherein the display device displays, as a moving image, a work space model that models the real work space, wherein the work space model includes a robot model that models the real robot, A peripheral object model that models a peripheral object around the real robot, wherein the robot model is The data generation program is generated by an operation of the operator, and the data generation program obtains state information indicating a state of a peripheral object around the real robot, and based on the state information, A prediction step of predicting a state of the peripheral object and generating a result of the prediction as peripheral object model data used for creating the peripheral object model displayed on the display device.

The data generation program is stored in a storage device. The storage device is a readable / writable or readable device built in or external to the computer, and may be, for example, a hard disk, a flash memory, an optical disk, or the like. The program stored in the storage device may be executed on a computer to which the storage device is directly connected, or may be downloaded and executed on a computer connected to the storage device via a network (for example, the Internet). You may.

Further, the remote operation system according to one aspect of the present invention is an operation terminal having an operation device that receives an operation of an operator and a display device that can be visually recognized by the operator, and is installed in an actual work space, and the operation terminal A real robot connected via a network capable of data communication, wherein the display device displays a working space model obtained by modeling the real working space as a moving image, The space model includes a robot model that models the real robot and a peripheral object model that models a real peripheral object around the real robot, wherein the robot model is operated by the operator with respect to the operating device. The remote operation system is configured to operate in accordance with the state information acquisition unit that acquires state information indicating the state of the real surroundings; And predicting the state of the real peripheral object a predetermined time later than the current time, and generating the predicted result as peripheral object model data used for creating the peripheral object model displayed on the display. And a data generation device including a prediction unit that performs the calculation.

According to the present invention, it is possible to suppress the influence of the communication delay between the operation terminal operated by the operator and the robot on the operation of the operator.

It is a block diagram showing the whole remote control system composition concerning one embodiment of the present invention. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a game device, a relay device, and an intermediary device in FIG. 1. FIG. 2 is a diagram schematically illustrating an example of the robot system in FIG. 1. FIG. 2 is a block diagram illustrating a functional configuration of a control unit of the game device in FIG. 1. FIG. 2 is a block diagram illustrating a functional configuration of a control unit of the relay device of FIG. 1. FIG. 4 is a diagram showing a processing flow before starting work of a robot in each of the game device and the relay device. It is a figure showing the processing flow after the start of work of the robot in each of the game device and the relay device. It is a figure for explaining an example of processing performed as time passes in each of a game device and a relay device.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

(Overview of the system)
First, an outline of a remote operation system 1 according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing the overall configuration of the remote operation system 1. The remote operation system 1 of the present embodiment includes a well-known various game device 2 (operation terminal) and a remote work site (work space; hereinafter, also referred to as an “actual work space”) different from the place where the operator is located. Is connected via the communication network 4 to the robot 51 installed in the robot. Then, the operator remotely controls the robot 51 using the game device 2 to cause the robot 51 to perform a predetermined operation.

The remote control system 1 includes a plurality of game apparatuses 2, one intermediary apparatus 6, and a plurality of robot systems 5, which can communicate with each other via a communication network 4 such as the Internet. The game device 2 is, for example, a stationary game device placed at the home of the operator or a portable game device carried by the operator.

The robot system 5 includes a robot 51 to be remotely controlled by an operator, one or more peripheral devices 52 installed around the robot 51, and a relay device 53. The robot 51, the peripheral device 52, and the relay device 53 are all installed at a remote work site remote from a place where an operator operating the game apparatus 2 is located. At the work site, one or more robot systems 5 exist.

Note that some or all of the plurality of robot systems 5 included in the remote control system 1 may be installed at the same work site or at different work sites. Also, a plurality of robot systems 5 installed at the same work site may have peripheral devices 52 shared by each other. Note that the plurality of robot systems 5 may include a plurality of peripheral devices 52 of the same type or different types. However, in FIG. 1, for simplification of the drawing, a block indicating the peripheral devices 52 is one for each robot system 5. Only one is shown.

The relay device 53 is communicably connected to each of the robot 51 and the peripheral device 52 of the same robot system 5 including the relay device 53. The relay device 53 sends information transmitted from the game device 2 or the mediation device 6 to the robot 51 or the peripheral device 52 via the communication network 4 or transmits information of the robot 51 or the peripheral device 52 to the game device 2 or the mediation device 6. Or send it to

The mediation device 6 assigns one robot system 5 to one operator (one game device 2). More specifically, the operator accesses the intermediary device 6 from the game device 2 and registers the user in advance, and the user ID is given to the operator by the user registration. When the operator inputs his / her user ID in the game device 2 and sends an operation request to the mediation device 6, the mediation device 6 that has received the operation request associates the game device 2 with one of the robot systems 5, The game device 2 is connected to the relay device 53 of the robot system 5 associated with the game device 2 via the communication network 4.

For example, upon receiving an operation request from the game device 2, the intermediary device 6 sends to the game device 2 work list information indicating the content of the work or the like. The operation request includes desired condition information input by the operator. The desired condition information includes the type of the robot, the work content of the robot, the target work, the work amount, and part or all of the work time. Upon receiving the operation request, the intermediary device 6 filters, based on the desired condition information included in the operation request, those that meet the conditions desired by the operator, and sends the filtered work list to the game device 2. When the operator designates one desired item from the work list displayed on the display device 25 of the game device 2, designation information corresponding to the designation of the operator is transmitted from the game device 2 to the intermediary device 6. . The mediation device 6 connects the game device 2 to the relay device 53 of the robot system 5 corresponding to the designated information.

As described above, in the remote control system 1, the operator can remotely control the robots 51 at various work sites physically far apart using the game apparatus 2. For example, the remote control system 1 allows the operator to operate the robot 51 at the work site on the back side of the earth while staying at home. Between the game device 2 and the robot system 5 (more specifically, the robot 51) connected to each other, a communication environment of each of the game device 2 and the robot system 5 and a physical distance between the game device 2 and the robot system 5 are set. For example, a communication delay due to the above may occur. In the present embodiment, as will be described later, the remote operation system 1 that suppresses the influence of a communication delay occurring between the game device 2 and the robot 51 connected to each other on the operation of the operator is realized.

(Hardware configuration)
FIG. 2 illustrates an example of a hardware configuration of the game device 2, the relay device 53, and the mediation device 6. In FIG. 2, only one game device 2 of the plurality of game devices 2 is shown, and only one robot system 5 of the plurality of robot systems 5 is shown. The game device 2 includes a game device body 2a, a display device 25 (display), a speaker 27, and a controller 28 (operator) connected thereto.

As shown in FIG. 2, the game apparatus main body 2a includes a control unit 21, a communication unit 22, and a storage unit 23 such as a hard disk or a memory card on a bus 20. The control unit 21 generates operation information to be sent to the robot 51 via the communication network 4 based on the operation of the controller 28. The robot 51 operates based on this operation information. Further, the control unit 21 generates an image displayed on the display device 25 based on an operation of the controller 28. The control unit 21 includes a CPU 210, a ROM (flash memory) 211, a RAM 212, an image processor 213, an audio processor 214, and an operation unit 215.

The CPU 210 controls the operation of each unit of the game device 2. The ROM 211 stores a basic program of the game apparatus 2 and the like. The storage unit 23 stores a remote operation program for operating the robot 51 by remote operation, a game program for executing various games, and the like. In the RAM 212, a work area used when the CPU 210 executes the game program is set. In the present embodiment, the storage of the remote control program in the storage unit 23 is essential, but the storage of the game program is not essential. In the following description, the ROM 211, the RAM 212, the storage unit 23, and the like of the control unit 21 in which various programs and data are stored are collectively referred to as a storage device of the game device 2.

The image processor 213 includes a GPU (Graphics Processing Unit) capable of generating a game screen. A video RAM (VRAM) 24 is connected to the image processor 213. A display device 25 is connected to the VRAM 24.

The audio processor 214 includes a DSP (Digital Signal Processor) that generates game audio. The sound processor 214 transmits the generated game sound to the amplifier 26 including the D / A converter. The amplifier 26 amplifies this audio signal and transmits it to the speaker 27.

The controller 28 is connected to the operation unit 215 by wireless or wired communication. The controller 28 includes a cross button, a push switch, a joystick, a mouse, a keyboard, a touch panel, and the like. Further, the operation unit 215 detects an operation signal from the user via the controller 28, and transmits the operation signal to the CPU 210.

The communication unit 22 is a communication device that communicates with the mediation device 6 and the relay device 53 via the communication network 4.

The mediation device 6 includes a control unit 61, a communication unit 62, and a storage unit 63. The control unit 61 is configured by, for example, an arithmetic unit having a processor and a memory. Specifically, the arithmetic unit is configured by, for example, a microcontroller, an MPU, an FPGA (Field Programmable Gate Array), a PLC (Programmable Logic Controller), a computer, a personal computer, and the like. The control unit 61 may be configured by a single arithmetic unit that performs centralized control, or may be configured by a plurality of arithmetic units that perform distributed control. The communication unit 62 is a communication device that communicates with the game device 2 and the relay device 53 via the communication network 4. The storage unit 63 is a readable / writable or readable storage device, and is, for example, a hard disk, a flash memory, an optical disk, or the like. The control unit 61 controls the operation of each unit of the mediation device 6. Various programs and data for controlling the operation of the mediation device 6, such as a program for associating the game device 2 with the robot system 5, are stored in the memory and the storage unit 63 of the control unit 61.

The relay device 53 includes a control unit 55, a communication unit 56, and a storage unit 57. The control unit 55 is configured by, for example, an arithmetic unit having a processor and a memory. Specifically, the arithmetic unit is configured by, for example, a microcontroller, an MPU, an FPGA (Field Programmable Gate Array), a PLC (Programmable Logic Controller), a computer, a personal computer, and the like. The control unit 55 may be configured by a single arithmetic unit that performs centralized control, or may be configured by a plurality of arithmetic units that perform distributed control. The communication unit 56 is a communication device that communicates with the game device 2, the mediation device 6, the robot 51, and the peripheral device 52 via the communication network 4. The storage unit 57 is a readable / writable or readable storage device, and is, for example, a hard disk, a flash memory, an optical disk, or the like. The control unit 55 controls the operation of the relay device 53. Various programs and data for controlling the operation of the relay device 53 are stored in the memory and the storage unit 57 of the control unit 55.

FIG. 3 schematically shows an example of the robot system 5. In the robot system 5, the robot 51 performs an operation of picking the work W conveyed by the conveyor 52a. In the robot system 5 shown in FIG. 3, the robot 51 is an industrial robot. The robot 51 includes a robot main body 51a to be remotely operated by an operator, and a robot controller 51b for controlling the operation of the robot main body 51a. The robot main body 51a shown in FIG. 3 is a vertical articulated robot arm having a tool attached to the distal end. In this example, a workpiece W is attached to the distal end of the vertical articulated robot arm as a tool. A gripping hand capable of gripping is mounted. The robot controller 51b includes a processor. The processor performs decoding, arithmetic processing, and the like of stored programs and various signals input from the outside, and controls the operation of the robot body 51a and outputs signals from various output ports. Take control. Further, the robot system 5 of FIG. 3 includes, as the peripheral devices 52, a conveyor 52a for transporting a work to be worked, one or more (two in this example) imaging devices 52b for shooting the work situation of the robot 51, and a work. A sensor 52c for detecting the position of W is provided.

The configuration of the robot system 5 shown in FIG. 3 is an example, and the types of the robot 51 and the peripheral devices 52 are configured according to the work of the robot 51. For example, the operation of the robot 51 may be, for example, a painting operation, a lunch box arrangement operation, a welding operation, or the like, in addition to the picking operation. The robot 51 need not be a vertical articulated robot, but may be an industrial robot such as a horizontal articulated robot, a parallel link robot, a polar coordinate robot, a cylindrical coordinate robot, or a rectangular coordinate robot. Further, the transfer device as the peripheral device 52 for transferring the work W to be worked may be a transfer device other than the conveyor. The peripheral device 52 may include a moving device that moves the robot body 51a. The one or more sensors as the peripheral device 52 may be, for example, a sensor that detects the position or posture of the robot 51 instead of or in addition to the sensor that detects the position of the work W. The one or more sensors as the peripheral device 52 include a sensor that detects the position, orientation, or orientation of the inspection target. Further, the robot system 5 may include a plurality of imaging devices 52b as the peripheral devices 52. As shown in FIG. 3, the imaging device 52b may be attached to the robot main body 51a, or may be provided at a fixed position in the work space.

(Functional configuration)
FIG. 4 is a block diagram showing a functional configuration of the control unit 21 of the game device 2. The control unit 21 of the game device 2 includes, as functional components, a communication control unit 31, an operation-side time management unit 32, a state information acquisition unit 33, a simulation unit (prediction unit) 34, an image display unit 35, a communication delay measurement unit. 36, a deviation detecting unit 37, and a model correcting unit 38. These functional units are functionally realized in cooperation with a predetermined program stored in the storage device of the game device 2. The predetermined program stored in the storage device of the game device 2 includes the “data generation program” of the present invention.

The communication control unit 31 controls the communication unit 22 to send the above-described operation request and designation information to the mediation device 6, and receive list information from the mediation device 6. In addition, the communication control unit 31 controls the communication unit 22 to receive information from the mediation device 6 for the mediation device 6 to make a communication connection with the robot system 5 associated with the game device 2. Further, the communication control unit 31 controls the communication unit 22 to send operation information generated by the operator operating the controller 28 to the corresponding relay device 53 of the robot system 5.

The operation-side time management unit 32 manages the time on the game device 2 side so that the time from when the operator operates the controller 28 to when the robot 51 operates based on the operation is kept constant. I do.

The state information acquisition unit 33 and the simulation unit 34 generate at least a part of data used for generating an image displayed on the display device 25. Specifically, the display device 25 displays, as a moving image, a work space model obtained by modeling a work space (hereinafter, “actual work space”) in which a robot (hereinafter, “actual robot”) 51 actually exists. You. The work space model includes a robot model and a peripheral object model arranged in a virtual work space. The robot model is a model of the real robot 51. The peripheral object model is a model of a predetermined peripheral object around the real robot 51 (hereinafter, “real peripheral object”). The actual peripheral object includes a peripheral device 52 and a work W located around the robot 51, and the peripheral object model includes a corresponding peripheral device model and a work model. For generating an image displayed on the display device 25, robot model data and peripheral object model data are used.

The robot model data includes static information of the real robot 51. The static information of the robot model includes, for example, structural information indicating the structure of the real robot 51 (the number of joints and the link length of the robot main body 51a, the structure of the tool, and the like) and information indicating the position and / or posture before starting the work. (For example, rotation angle information of a servo motor included in the robot body 51a). In addition, the robot model data includes dynamic information of the robot model, that is, operation information (command) of the operator on the controller 28. This operation information is used for operating the robot model, and is also transmitted from the game device 2 to the real robot 51 via the communication network 4 and used for the operation of the real robot 51. That is, when the operator operates the controller 28 while looking at the display screen of the display device 25 to operate the robot model on the display screen, the real robot 51 at the work site also operates. However, as described later, the real robot 51 operates with a certain time delay from the operation of the robot model.

(4) The peripheral object model data includes static information of the actual peripheral object. The static information of the real peripheral object includes structural information of the peripheral device 52, information indicating the position and / or posture of the peripheral device 52 before the start of work, shape data and structure information of the work W to be worked by the real robot 51. , Information indicating the position and / or posture of the work W before the start of the work. Further, the peripheral object model data includes information that predicts the position and orientation of the actual peripheral object a predetermined time ahead. This prediction is performed by the state information acquisition unit 33 and the simulation unit 34. In the present embodiment, the game device 2 including the state information acquisition unit 33 and the simulation unit 34 corresponds to the “data generation device” of the present invention.

Specifically, the state information obtaining unit 33 obtains state information indicating the state of a peripheral device 52 around the real robot 51 or the state of a real peripheral object such as a work W. Then, the simulating unit 34 simulates a change in the position or orientation of the real peripheral object over time based on the state information. For example, the state information acquisition unit 33 outputs, at a certain point, information indicating a transfer speed set in a transfer device (in this example, a conveyor 51a) as the peripheral device 52 and information indicating a position of the work W transferred to the transfer device. Is obtained as the state information, the simulation unit 34 can easily calculate the position and orientation of the work W a predetermined time ahead from the transport speed information and the work position information. As described above, the simulation unit 34 predicts the state of the real peripheral object a predetermined time later than the current time by the simulation. Then, the simulation unit 34 generates the predicted result as peripheral object model data used for creating a peripheral object model.

The image display unit 35 displays the work space model created based on the robot model data and the peripheral object model data on the display device 25. For example, the image display unit 35 arranges a virtual camera in a virtual work space in which a robot model and a peripheral object model created based on the robot model data and the peripheral object model data are arranged. The image captured by the virtual camera is displayed on the display device 25. The position, orientation, zoom, and the like of the virtual camera may be determined in advance, or may be changeable according to, for example, an operation performed on the controller 28 by an operator. The position and orientation of the virtual camera in the virtual work space may correspond to the position and orientation of the imaging device 52b in the real work space, respectively.

The details of the communication delay measuring unit 36, the shift detecting unit 37, and the model correcting unit 38 will be described later.

FIG. 5 is a block diagram showing a functional configuration of the control unit 55 of the relay device 53. The control unit 55 of the relay device 53 has a communication control unit 71 and a robot-side time management unit 72 as a functional configuration. These functional units are functionally realized in cooperation with a predetermined program stored in the control unit 55 and / or the storage unit 57 of the relay device 53.

The communication control unit 71 controls the communication unit 56 to receive, from the game device 2, operation information generated by the operator operating the controller 28.

The robot-side time management unit 72 manages the time on the robot system 5 side so that the time from when the operator operates the controller 28 to when the robot 51 operates based on the operation is kept constant. I do.

(Processing flow)
Next, execution processing in each of the game device 2 and the relay device 53 will be described with reference to FIGS.

FIG. 6 is a diagram showing a processing flow before starting work of the robot 51 in each of the game apparatus 2 and the relay apparatus 53. Before starting the work, the robot model, the peripheral device model, and the work model in the work space model are created so as to be in the same state as the real robot 51, the peripheral device 52, and the work W, respectively.

Specifically, when the game apparatus 2 and the relay apparatus 53 are connected by the intermediary apparatus 6, first, the communication control unit 71 of the relay apparatus 53 transmits information for creating a work space model before the start of work to the game apparatus. 2 (step S201). The information for creating the work space model before the start of the work includes state information indicating the state of the real surroundings around the real robot 51.

In the present embodiment, the state information includes imaging information generated by imaging the peripheral device 52 and the work W by the imaging device 52b installed in the actual work space. Further, the state information includes detection information of a sensor as the peripheral device 52. The detection information of the sensor includes, for example, information indicating whether the work W is at a predetermined position in the work space, information indicating the position or posture of the work W, and the like. The state information includes setting information set in the peripheral device 52. For example, when the robot system 5 includes a transfer device as the peripheral device 52, the setting information may include a transfer speed and a transfer interval set for the transfer device. The transport interval may be the distance between the workpieces W to be transported, or from the time when a certain workpiece W is transported to a predetermined position in front of the robot 51, the next workpiece W is transported to a predetermined location. May be a time interval until the time.

{Circle around (2)} In step S201, the status information sent from the relay device 53 to the game device 2 includes information indicating the status of the real robot 51. The information indicating the state of the real robot 51 may include, for example, posture information and position information of the real robot 51 stored in the robot controller 51b. The information indicating the state of the real robot 51 may include imaging information obtained by the imaging device 52b, which is the peripheral device 52, and detection information of the sensor 52c.

On the game device 2 side, the status information acquisition unit 33 acquires the status information received from the relay device 53 (Step S101). Then, the simulation unit 34 creates a work space model having the same state as the actual work space before the start of the work, based on the state information acquired by the state information acquisition unit 33 (step S102). The state information includes, for example, shape data and structure information of the work W, the position and orientation of the work W, position information and structure information of the peripheral device 52, setting information, and the like.

Specifically, the simulating unit 34 determines the position information (position coordinates in the coordinate system set for the actual work space) and the posture information (the rotation angle of the servo motor of the robot body 51a) before the start of the work. Information, etc.), a robot model is created such that the state (position, posture, etc.) of the robot model is the same as that of the real robot 51. Further, the simulating unit 34 creates a peripheral object model based on the state information of the actual peripheral object before the start of the work so that the peripheral object model is in the same state as the actual peripheral object.

The image display unit 35 generates an image of the work space model created in step S102 and displays the image on the display device 25 (step S103). Thus, preparations for starting work are completed.

FIG. 7 is a processing flow after the operation of the robot 51 in each of the game device 2 and the relay device 53 is started. FIG. 8 is a diagram for explaining an example of processing executed with time in each of game device 2 and relay device 53.

After step S103, the control unit 21 of the game device 2 determines whether or not an operation for instructing to start a work has been performed (step S105). If there is no work start instruction (Step S105: No), the state is a standby state until there is a work start instruction. When the operator performs an operation for instructing the controller 28 to start work (step S105: Yes), the communication control unit 31 transmits a work start instruction to the relay device 53 (step S106). Further, the operation-side time management unit 32 stores the transmission time of the work start instruction in the storage device (the RAM 212 or the storage unit 23 or the like) of the game device 2 as the operation-side reference time t1 (step S107, see also FIG. 8). .

On the robot system 5 side, when the communication control unit 71 of the relay device 53 receives the work start instruction (step S202), the communication control unit 71 sets the work time after waiting for a predetermined time Δt from the reception time of the work start instruction. As a start time, a work start instruction is sent to the robot 51 and the peripheral device 52 (step S203). Further, the robot-side time management unit 72 stores a time later than the reception time of the work start instruction by a predetermined time Δt as the robot-side reference time t2 (step S204, see also FIG. 8).

Thus, as described below, in the work space model displayed on the display device 25, the work of the robot model is started from the operation-side reference time t1, whereas, in the actual work space, the real robot is started from the robot-side reference time t2. Work is started. In other words, in the actual work space, work progresses with a delay (t2−t1) of the reference time from the work space model. For example, the work start instruction sent from the game device 2 to the robot system 5 may include a command to shift the peripheral device 52 from the stopped state to the operating state. For example, the work model of the work space model may start to be transported by the transfer device model from the operation-side reference time t1, and the real work W of the actual work space may be started to be transferred by the transfer device from the robot-side reference time t2.

In the game device 2, after the reference time t1 is set in step S107, the robot 51 can be remotely operated. That is, when an operation for operating the robot 51 is performed on the controller 28 (step S108: Yes), operation information is generated. The simulation unit 34 simulates the operation of the robot model based on the operation information (Step S109). Further, the communication control unit 31 transmits to the relay device 53, together with the operation information, the elapsed time T from the reference time t1 indicating the time elapsed until the operation corresponding to the operation information is performed (step S110).

On the robot system 5 side, when the communication control unit 71 of the relay device 53 receives the operation information and the time information from the game device 2 as a set, after the elapsed time T from the reference time t2, the communication control unit 71 performs the operation based on the received operation information. An operation command is sent to the robot 51 to execute (step S205). In step S110, time information indicating the time at which the operation corresponding to the operation information was performed, such as time information indicating the time at which the operation was performed, may be transmitted instead of the elapsed time T.

(4) Regardless of whether it is determined in step S108 that an operation has been performed, the simulating unit 34 simulates the state of an actual peripheral object (step S111). Specifically, the simulation unit 34 predicts the state of the real peripheral object a predetermined time later than the current time based on the state information, and uses the predicted result as peripheral object model data used for creating a peripheral object model. Generate. Note that the state information need not be the information acquired by the state information acquisition unit 33 in step S101 before the start of the work, and may be information acquired after the start of the work. That is, the latest state information may be sequentially transmitted from the robot system 5 to the game apparatus 2, and the state information acquisition unit 33 may sequentially acquire the latest state information. The simulating unit 34 may predict the state of a real surrounding object based on the latest state information.

The image display unit 35 displays an image indicating the work space model on the display device 25 based on the data generated based on the simulation results in steps S109 and S111 (step S112). The image display unit 35 displays the work space model of the situation ahead of the actual work space situation by the difference (t2-t1) between the robot-side reference time t2 and the operation-side reference time t1. In other words, as shown in FIG. 8, with respect to the work space model displayed on the display device 25, the actual work space is only the difference (t2-t1) between the robot-side reference time t2 and the operation-side reference time t1. Work progresses late.

Steps S108 to S112 are repeated until an operation for terminating the operation is performed on the controller 28 or until a certain operation is completed (Step S113: No).

In the above-described step S203, the robot-side time management unit 72 does not set the reception time of the work start instruction to the robot-side reference time t2, but sets the time after waiting for a predetermined time Δt from the reception time of the work start instruction to the robot-side reference time. It is set as t2. Thus, by providing a constant interval between the operation-side reference time t1 and the robot-side reference time t2, the fluctuation of the communication delay between the game device 2 and the robot system 5 (see Δd1 and Δd2 in FIG. 8) is reduced. Absorbing.

The waiting time Δt may be set based on an actual communication delay between the game apparatus 2 and the robot system 5 (specifically, the relay apparatus 53). The communication delay measuring unit 36 measures a communication delay between the game device 2 and the robot system 5 (more specifically, the relay device 53) associated therewith. The measurement of the communication delay is performed by a known method. The robot-side time management unit 72 determines the length of the waiting time Δt, in other words, the robot-side reference time t2 and the operation-side reference time according to the degree of change in the communication delay measured by the communication delay measurement unit 36 before step S105. A difference (t2−t1) from t1 may be set.

In addition, in this embodiment, when steps S108 to S112 are repeated, the work space model is periodically corrected. The deviation detecting unit 37 detects the degree of deviation between the state of the work space model displayed on the display device 25 at a predetermined time and the state of the actual work space after a predetermined time from the predetermined time. . More specifically, the shift detecting unit 37 determines the state of the work space model at a predetermined time and the actual time after the difference (t2-t1) between the robot-side reference time t2 and the operation-side reference time t1 from the predetermined time. The degree of deviation from the state of the work space is detected.

For example, the shift detecting unit 37 may calculate the state of the work space model at the time ta predicted by the simulation unit 34 based on the state information indicating the state of the actual work space at a certain time ta and the state information acquired before the time ta. May be compared with each other to detect the degree of deviation. The state information indicating the state of the actual work space compared by the shift detection unit 37 may include the state information used by the simulation unit 34 to predict the state of the actual surroundings. The displacement detection unit 37 receives, for example, imaging information of the actual working space at the time ta from the robot system 5 and determines the position and orientation of the real peripheral object (the peripheral device 52 or the work W) from the imaging information by image recognition. Is also good. The shift detection unit 37 determines the position and orientation of the real peripheral object determined from the imaging information at time ta and the work space at time (that is, time (ta− (t2−t1))) corresponding to the real work space at time ta. Compare the position and orientation of the peripheral object model in the model. In other words, the shift detecting unit 37 determines the state of the actual peripheral object (the position and orientation of the peripheral device 52 and the work W, etc.) determined from the imaging information at the time ta and the peripheral object just before the time ta by the time (t2-t1). Compare with model state.

Then, when the displacement detected by the displacement detecting unit 37 exceeds a preset range, the model correcting unit 38 corrects the work space model so that the displacement is eliminated. For example, the model correction unit 38 may adjust the state information used by the simulation unit 34 by the deviation detected by the deviation detection unit 37. Alternatively, the model correction unit 38 may adjust the peripheral object model data generated by the simulation unit 34 by the deviation detected by the deviation detection unit 37. When the simulation unit 34 simulates the state of the real peripheral object after the model correction unit 38 corrects the work space model, the simulation unit 34 simulates the state of the real peripheral object in consideration of the correction of the work space model. To

As described above, according to the game device 2 and the remote operation system 1 as the data generation device of the present embodiment, the state of the real surroundings is predicted by the simulation unit 34 a predetermined time after the current time. The prediction result is generated as peripheral object model data used for creating a peripheral object model displayed on the display device 25. Therefore, the same time lag between the state of the peripheral object model and the state of the real peripheral object as that between the robot model and the real robot can be generated. Thereby, the time axis of the robot model and the time axis of the peripheral object model can be made to coincide with each other, so that the influence of the communication delay on the operation of the operator can be suppressed.

Further, in the present embodiment, the shift detecting unit 37 detects a shift between the state of the work space model displayed on the display device 25 at a predetermined time and the state of the actual work space after a predetermined time from the predetermined time. Is detected, and the model correction unit 38 corrects the work space model such that the deviation is eliminated when the deviation detected by the deviation detecting unit 37 exceeds a preset range. Therefore, it is possible to suppress a deviation between the state of the work space model displayed on the display device 25 and the state of the actual work space.

(Other embodiments)
The present invention is not limited to the above embodiments, and various modifications can be made without departing from the spirit of the present invention.

For example, in the above embodiment, the game device 2 is exemplified as the operation terminal, but the operation terminal in the present invention may not be the game device 2. The operation terminal only needs to have an operation device that receives an operation of the operator and a display device that can be visually recognized by the operator. For example, the game device 2 may be any of a personal information terminal (PDA (Personal Data Assistant)), a smartphone, a personal computer, a tablet, and a remote controller dedicated to a robot, in addition to various known game devices.

For example, the control unit of the mediation device 6 or another server device may function as a state information acquisition unit and a prediction unit by executing a predetermined program. That is, the “data generation device” of the present invention may not be an operation terminal operated by the operator, and may be a device that communicates with the operation terminal. For example, the data generation device of the present invention may be the robot controller 51b, the relay device 53, the mediation device 6, or a server device different from the mediation device 6. The data generation device of the present invention may not include a part or all of the communication delay measurement unit 36, the shift detection unit 37, and the model correction unit 38.

In addition, the “data generation program” of the present invention includes, instead of or in addition to the storage device of the game device 2 as the operation terminal, the robot controller 51b, the relay device 53, the mediation device 6, and a server separate from the mediation device 6. The information may be stored in a storage device provided in at least one of the devices. In addition, the “data generation program” of the present invention is executed by a computer incorporated in each of at least one of the robot controller 51b, the relay device 53, the mediation device 6, and a server device different from the mediation device 6, and What is necessary is just what makes a computer function as a state information acquisition unit and a prediction unit. The processing flows of the game device 2 and the relay device 53 described with reference to FIGS. 6 and 7 do not limit the present invention.

The remote control system 1 may include only one game device 2. Further, the remote control system 1 may include only one robot system 5. Further, the remote control system 1 does not need to include the mediation device 6. Further, the relay device 53 and the robot controller 51b in the robot system 5 may be integrally configured. That is, the robot system 5 may include one control device having both functions of the relay device 53 and the robot controller 51b.

The real robot in the present invention may not be an industrial robot, but may be any robot that operates according to the operation of the operator at the operation terminal. For example, the real robot of the present invention may be a service robot that provides services such as care, medical care, transportation, cleaning, and cooking. Further, the real robot of the present invention may be a humanoid. The real robot operates based on operation information generated by the operation of the operating device by the operator, but the real robot may operate based on not only the operation information but also a preset task program.

In the above embodiment, the data generation program predicts the state of the real peripheral object after the current time based on the state information obtaining step of obtaining state information indicating the state of the real peripheral object and the state information. And a predicting step of generating the result obtained as the peripheral object model data used for creating the peripheral object model displayed on the display device, by the controller 21 of the game apparatus 2 as an operation terminal. However, the data generation program of the present invention may cause another computer to execute the above-described state information acquisition step and prediction step. For example, the data generation program according to the present invention includes a controller provided in at least one of the robot controller 51b, the relay device 53, the mediation device 6, and a server device different from the mediation device 6 for performing the state information acquisition step and the prediction step described above. (A computer). The data generation program may be distributed and stored in a plurality of storage devices. For example, a part of the data generation program may be stored in the storage device of the operation terminal, and the other part may be stored in another storage device (for example, the relay device 53). Further, the data generation program according to the present invention may be configured such that the state of the work space model displayed on the display at a predetermined time and the state of the real work space after the predetermined time by the predetermined time are provided. May be executed by a computer. Further, the data generation program according to the present invention further includes a model correction step of correcting the work space model such that the shift is eliminated when the shift detected in the shift detection step exceeds a preset range. The program may be executed by a computer.

1: Remote operation system 2: Game device (operation terminal)
4: Communication network 5: Robot system 6: Mediation device 25: Display device (display device)
28: Controller (operator)
33: state information acquisition unit 34: simulation unit (prediction unit)
35: Image display unit 36: Communication delay measurement unit 37: Displacement detection unit 38: Model correction unit 51: Robot 51a: Robot body 51b: Robot controller 52: Peripheral device 52a: Conveyor 52b: Imaging device 52c: Sensor 53: Relay device 55: Control unit

Claims

An operation device having an operation device that receives an operation of the operator and a display device that can be visually recognized by the operator, and an actual robot installed in an actual work space and connected to the operation terminal via a network capable of data communication. A data generation device for generating at least a part of data used for generating an image displayed on the display,
On the display, a work space model that models the actual work space is displayed as a moving image,
The work space model includes a robot model that models the real robot and a peripheral object model that models a real peripheral object around the real robot,
The robot model is created to operate in response to an operation of the operator on the operation device,
The data generation device,
A state information acquisition unit that acquires state information indicating the state of the real surroundings,
Based on the state information, predicts the state of the real peripheral object a predetermined time later than the current time, and uses the predicted result as a peripheral object model used to create the peripheral object model displayed on the display. A data generation device comprising: a prediction unit configured to generate data.
2. The data generation device according to claim 1, wherein the real peripheral object includes at least one of a work to be worked by the real robot, a transfer device that transfers the work, and a moving device that moves the real robot.
3. The data generation device according to claim 1, wherein the state information includes imaging information generated by an imaging device installed in the work space imaging the real surroundings. 4.
The data generation device according to any one of claims 1 to 3, wherein the status information includes setting information set in a peripheral device as the real peripheral.
A deviation detecting unit that detects a degree of deviation between a state of the work space model displayed on the display at a predetermined time and a state of the actual work space after the predetermined time by the predetermined time The data generation device according to any one of claims 1 to 4, further comprising:
The data according to claim 5, further comprising: a model correction unit that corrects the work space model so that the deviation is eliminated when the deviation detected by the deviation detection unit exceeds a preset range. Generator.
The data generation device according to any one of claims 1 to 6, wherein the operation terminal is a game device including a controller as the operation device.
The device according to any one of claims 1 to 7, wherein the operation terminal is at least one of a personal information terminal (PDA (Personal Data Assistant)), a smartphone, a personal computer, a tablet, and a remote controller dedicated to a robot. A data generation device as described.
An operation device having an operation device that accepts an operation of the operator and a display device that can be visually recognized by the operator, and a real robot installed in a work space and connected to the operation terminal via a network capable of data communication, A data generation method for generating at least a part of data used for generation of an image displayed on the display device,
On the display, a work space model that models the real work space is displayed as a moving image,
The work space model includes a robot model that models the real robot and a peripheral object model that models a real peripheral object around the real robot,
The robot model is created to operate in response to an operation of the operator on the operation device,
The data generation method includes:
State information acquisition step of acquiring state information indicating the state of the real surroundings,
Based on the state information, predict the state of the real peripheral object after the current time, and generate the predicted result as peripheral object model data used for creating the peripheral object model displayed on the display. A data generation method, comprising: a prediction step.
An operation device having an operation device that accepts an operation of the operator and a display device that can be visually recognized by the operator, and a real robot installed in a work space and connected to the operation terminal via a network capable of data communication, In a remote operation system comprising: a data generation program that, when executed by a computer, generates at least a part of data used for generating an image displayed on the display,
On the display, a work space model that models the actual work space is displayed as a moving image,
The work space model includes a robot model that models the real robot and a peripheral object model that models a real peripheral object around the real robot,
The robot model is generated by an operation of the operator on the operation device,
The data generation program includes:
State information acquisition step of acquiring state information indicating the state of the real surroundings,
Based on the state information, predict the state of the real peripheral object after the current time, and generate the predicted result as peripheral object model data used for creating the peripheral object model displayed on the display. A data generation program for causing the computer to execute a prediction step.
An operation device having an operation device that receives an operation of the operator and a display device that can be visually recognized by the operator, and an actual robot installed in an actual work space and connected to the operation terminal via a network capable of data communication. A remote control system comprising:
On the display, a work space model that models the real work space is displayed as a moving image,
The work space model includes a robot model that models the real robot and a peripheral object model that models a real peripheral object around the real robot,
The robot model is created to operate in response to an operation of the operator on the operation device,
The remote operation system, a state information acquisition unit that acquires state information indicating the state of the real surroundings, and, based on the state information, predicts the state of the real surroundings a predetermined time later than the current time, A prediction unit configured to generate a prediction result as peripheral object model data used for creating the peripheral object model displayed on the display unit.