KR20210041048A - Data generation device, data generation method, data generation program and remote operation system - Google Patents

Abstract

The data generating device is a data generating device that generates at least a portion of data used to generate an image displayed on a display, wherein a working space model modeling an actual working space is displayed as a moving picture, and the working space model, A robot model modeling an actual robot and a surrounding object model modeling an actual surrounding object around an actual robot. The robot model is created to operate according to the operator's manipulation of the manipulator, and the data generating device, A state information acquisition unit that acquires state information indicating the state of the surrounding objects, and predicts the state of the surrounding objects by a predetermined time after the current point, based on the state information, and displays the predicted result as a surrounding object model displayed on the display. It includes a prediction unit that generates data from the surrounding object model used for creation.

Description

Data generation device, data generation method, data generation program and remote operation system

The present invention relates to a data generating apparatus, a data generating method, a data generating program, and a remote operation system in a system in which an operator remotely manipulates a remote robot through a network.

BACKGROUND ART Conventionally, a system in which an operator remotely operates a robot at a remote location through a network is known. In such a system, there is a concern that a communication delay may occur between the robot and the operating device operated by the operator. For example, Patent Document 1 discloses a remote operation system capable of grasping a communication delay between a slave device and a master device.

The remote operation system disclosed in Patent Document 1 includes a master device and a slave device. The slave device includes a slave robot that operates according to operation information corresponding to an operator's operation sent from the master device. The slave robot is provided with an imaging device that captures its own work environment, and the slave device sequentially captures the work environment and transmits the captured actual image to the master device. The master device includes a simulator that performs motion simulation based on operation information sent to the slave robot, and generates a display image by synthesizing an actual image sent from the slave device with a simulation image obtained through motion simulation. The ratio of the synthesis of the actual image and the simulated image in the display image varies depending on the communication delay between the master device and the slave device. Specifically, when the communication delay is large, a display image is generated by increasing the ratio of the simulated image. The actual image is an image that is projected not only to the state of the slave robot, but also to the background of the work environment, while the simulated image is an image that displays only the state of the slave robot. Accordingly, it is possible for the operator to easily grasp how much communication delay has occurred from whether the background of the work environment is darkly lit in the display image, for example.

Japanese Unexamined Patent Application Publication No. 2015-47666

However, in the system of Patent Document 1, the operator needs to determine the degree of communication delay from the combination ratio of the actual image and the simulated image, and when it is determined that the communication delay is large, the amount of operation given to the master device is reduced. You need to respond appropriately. Therefore, in the above-described system, the operator needs to perform a situational determination according to communication delay in addition to the operation of the robot, and there is a fear that the operator cannot concentrate on the operation of the robot.

Accordingly, it is an object of the present invention to provide a data generating device, a data generating method, a data generating program, and a remote operation system capable of suppressing the influence of a communication delay between an operator operated terminal and a robot on the operator's operation. It is done.

In order to solve the above problems, a data generating apparatus according to an aspect of the present invention includes an operation terminal having a manipulator that accepts an operator's operation and an indicator that the operator can visually check, and is installed in an actual work space, In a remote control system including an actual robot connected through a network capable of data communication with the operation terminal, a data generating device for generating at least part of data used for generating an image displayed on the display, wherein the display comprises: A work space model modeled on an actual work space is displayed as a time-varying image, and the work space model is a robot model modeled on the real robot and a real surrounding object in the vicinity of the real robot. A peripheral object model is included, wherein the robot model is created to operate according to an operation of the operator on the manipulator, and the data generating device includes a state information acquisition unit for acquiring state information indicating a state of the actual surrounding object; , Based on the state information, predicts the actual state of the surrounding object after a predetermined time from the current point, while generating the predicted result as surrounding object model data used to create the surrounding object model displayed on the indicator. It is equipped with a predicting unit.

When there is a communication delay between the operation terminal and the actual robot, the operation of the robot model and the actual robot at the same time is determined by the timing between the operator manipulating the manipulator and the actual robot performing the corresponding operation. Mismatches may occur between motions. According to the above configuration, by the prediction unit, the state of the actual surrounding object after a predetermined time from the present point is predicted, and the prediction result is generated as surrounding object model data used to create the surrounding object model displayed on the display. Accordingly, even between the state of the surrounding object model and the actual state of the surrounding object, it is possible to generate the same time difference as between the robot model and the actual robot. Accordingly, since the time axis of the robot model and the surrounding object model can be matched, the influence of the communication delay on the operator's operation can be suppressed.

The actual surroundings may include at least one of a work target of the actual robot, a transport device for transporting the work, and a moving device for moving the actual robot.

The state information may include imaging information generated by an imaging device installed in the working space to capture the actual surrounding object.

The state information may include setting information set in the peripheral device as the actual surrounding object.

The data generating device further includes a deviation detection unit configured to detect a degree of a deviation between the situation of the working space model displayed on the display at a predetermined time point and the condition of the actual work space after the predetermined time period from the predetermined time point. You may have it. For example, when the shift detected by the shift detection unit is greater than a predetermined value, countermeasures such as stopping the work of the actual robot or correcting the model displayed on the display become possible.

The data generating device may further include a model correction unit for correcting the working space model so that the shift is eliminated when the shift detected by the shift detection unit exceeds a preset range.

The operation terminal may be a game device including a controller as the operation device.

The operation terminal may be at least one of a personal information terminal (PDA (Personal Data Assistant)), a smart phone, a personal computer, a tablet, and a robot-only remote control unit.

In addition, a data generation method according to an aspect of the present invention includes an operation terminal having a manipulator that accepts an operator's operation and an indicator that the operator can visually check, and is installed in an actual work space, and data communication with the operation terminal A data generation method for generating at least some data used for generating an image displayed on the display in a remote control system including an actual robot connected through a possible network, wherein the display includes modeling the actual work space. A work space model is displayed as a video, and the work space model includes a robot model modeling the real robot, and a surrounding object model modeling the real surroundings around the real robot, and the robot model, It is created to operate according to the operation of the operator on the manipulator, and the data generation method includes a state information acquisition step of acquiring state information indicating a state of the actual surrounding object, and a predetermined value from the current point based on the state information. And a prediction step of predicting the state of the actual surrounding object after a period of time, and generating a predicted result as surrounding object model data used to create the surrounding object model displayed on the display.

In addition, a data generation program according to an aspect of the present invention includes an operation terminal having an operator that accepts an operator's operation and an indicator that the operator can visually check, and is installed in an actual work space, and data communication with the operation terminal A data generation program that generates at least part of data used for generation of an image displayed on the display by executing it on a computer in a remote control system including an actual robot connected via a network, wherein the display includes: A work space model modeling the work space is displayed as a video, and the work space model includes a robot model modeling the real robot and a surrounding object model modeling the real surroundings around the real robot, The robot model is created to operate according to an operation of the operator on the manipulator, and the data generation program includes a state information acquisition step of acquiring state information indicating a state of the actual surrounding object, and based on the state information , A prediction step of predicting the state of the actual surrounding object after a predetermined time from the current point, and generating the predicted result as peripheral object model data used to create the peripheral object model displayed on the display, the computer. Run with

Here, the data generation program is stored in a storage device. The memory device is a readable and writable device built-in or external to a computer, and for example, a hard disk, a flash memory, an optical disk, or the like may be used. The program stored in the storage device may be executed on a computer to which the storage device is directly connected, or may be downloaded and executed on a computer connected to the storage device through a network (for example, the Internet).

In addition, a remote control system according to an aspect of the present invention includes an operation terminal having a manipulator that accepts an operator's operation and an indicator that the operator can visually check, and is installed in an actual work space, and data communication with the operation terminal A remote control system having a real robot connected through a possible network, wherein a working space model modeled on the real work space is displayed as a moving picture on the display, and the work space model is a robot model modeled on the real robot. And, a peripheral object model modeled of an actual peripheral object in the vicinity of the real robot, the robot model is created to operate according to the operator's manipulation of the manipulator, and the remote control system A state information acquisition unit that acquires state information indicating a state of a surrounding object; and, based on the state information, predicts the state of the actual surrounding object by a predetermined time after the present point, and displays the predicted result on the display. And a data generation device including a prediction unit that generates the surrounding object model data used to create the surrounding object model.

According to the present invention, the influence of the communication delay between the robot and the operation terminal operated by the operator on the operation of the operator can be suppressed.

1 is a block diagram showing the overall configuration of a remote control system according to an embodiment of the present invention.
2 is a block diagram showing an example of a hardware configuration of the game device, the relay device, and the relay device of FIG. 1.
3 is a diagram schematically showing an example of the robot system of FIG. 1.
Fig. 4 is a block diagram showing a functional configuration of a control unit of the game device of Fig. 1.
Fig. 5 is a block diagram showing a functional configuration of a control unit of the relay device of Fig. 1.
[Fig. 6] Fig. 6 is a diagram showing a processing flow of a game device and a relay device before starting work of the robot.
Fig. 7 is a diagram showing a processing flow after starting a robot operation in each of the game device and the relay device.
Fig. 8 is a diagram for explaining an example of processing executed over time in each of the game device and the relay device.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(System overview)

First, an overview of the remote control system 1 according to the present embodiment will be described with reference to FIG. 1. 1 is a block diagram showing the overall configuration of a remote operation system 1. The remote operation system 1 of the present embodiment is a known various game device 2 (operation terminal), and a work site (work space, hereinafter also referred to as "actual work space") in a remote place different from the place where the operator is located. The robot 51 installed in is connected through the communication network 4. Then, the robot 51 is remotely operated by the operator using the game device 2, and the robot 51 performs a predetermined operation.

The remote control system 1 includes a plurality of game devices 2, one intermediary device 6, and a plurality of robot systems 5, and these include, for example, a communication network 4 such as the Internet. It is possible to communicate with each other through. The game device 2 is, for example, a stationary game device placed in the operator's home or a portable game device carried by the operator.

The robot system 5 includes a robot 51 that is a target to be remotely operated by an operator, one or more peripheral devices 52 installed around the robot 51, and a relay device 53. The robot 51, the peripheral device 52, and the relay device 53 are all installed at a work site in a remote place away from a place where an operator who operates the game device 2 is located. At the work site, there are one or more robot systems 5.

Here, some or all of the plurality of robot systems 5 included in the remote operation system 1 may be installed at the same site or may be installed at different work sites. Further, a plurality of robot systems 5 installed in the same work site may be provided with peripheral devices 52 shared with each other. Here, the plurality of robot systems 5 may include a plurality of peripheral devices 52 of the same type or different from each other, but in FIG. 1, for simplicity of illustration, a block representing the peripheral devices is referred to as one robot system. For (5), only one is shown.

The relay device 53 is communicatively connected to each of the robot 51 and the peripheral device 52 of the same robot system 5 including the relay device 53. The relay device 53 sends information sent from the game device 2 or the mediation device 6 to the robot 51 or peripheral device 52 via the communication network 4, or the robot 51 or Information on the peripheral device 52 is sent to the game device 2 or the intermediary device 6.

The intermediary device 6 allocates one robot system 5 to one operator (one game device 2). In more detail, the operator accesses the intermediary device 6 from the game device 2 to perform user registration in advance, and the operator is given a user ID through user registration. The operator inputs his/her user ID from the game device 2 and sends an operation request to the intermediary device 6, and the intermediary device 6 receiving the operation request passes the game device 2 to a robot system. At the same time as corresponding to (5), the game device 2 and the relay device 53 of the robot system 5 associated with each other are communicated with each other via the communication network 4.

For example, when the intermediary device 6 receives an operation request from the game device 2, it sends the work list information indicating the work content and the like to the game device 2. The operation request includes desired condition information input by the operator. The desired condition information includes the type of the robot, the work content of the robot, the target work, the amount of work, and some or all of the work time. When receiving an operation request, the intermediary device 6 filters what meets the condition desired by the operator based on the desired condition information included in the operation request, and sends the filtered work list to the game device 2. If the operator designates one of the job lists displayed on the display device 25 of the game device 2, the game device 2 sends the designated information corresponding to the operator's designation to the intermediary device 6 from the game device 2 Is sent. The intermediate device 6 connects the game device 2 to the relay device 53 of the robot system 5 corresponding to the designated information.

In this way, in the remote operation system 1, the operator can remotely operate the robot 51 at various work sites that are physically distant by using the game device 2. For example, by means of the remote control system 1, it is possible for the operator to operate the robot 51 at the work site on the other side of the earth while staying at home. Between the game device 2 and the robot system 5 (more specifically, the robot 51) connected to each other, each communication environment of the game device 2 and the robot system 5 or the game device 2 and the robot Communication delays may occur due to physical distances between the systems 5 and the like. According to this embodiment, as described later, a remote control system 1 is realized that suppresses the influence of the communication delay occurring between the connected game device 2 and the robot 51 on the operator's operation. I'm doing it.

(Hardware configuration)

FIG. 2 shows an example of the hardware configuration of the game device 2, the relay device 53, and the mediation device 6. As shown in FIG. Here, in FIG. 2, only one of the plurality of game devices 2 is shown, and only one of the plurality of robot systems 5 is shown. The game device 2 includes a game device main body 2a, a display device 25 (display device), a speaker 27, and a controller 28 (operator) connected thereto.

As shown in Fig. 2, the game device main body 2a includes a control unit 21, a communication unit 22, and a storage unit 23 such as a hard disk or a memory card on a bus 20. The control unit 21 generates operation information sent to the robot 51 via the communication network 4 based on the operation of the controller 28. Here, the robot 51 operates based on such operation information. Further, the control unit 21 generates an image to be displayed on the display device 25 based on the operation of the controller 28. Such a control unit 21 includes a CPU 210, a ROM (flash memory) 211, a RAM 212, an image processor 213, an audio processor 214, and an operation unit 215.

The CPU 210 controls the operation of each unit of the game device 2. In the ROM 211, a basic program of the game device 2 and the like are stored. In the storage unit 23, a remote control program for operating the robot 51 by remote operation, a game program for executing various games, and the like are stored. In the RAM 212, a work area used when the CPU 210 executes a game program is set. Here, in this embodiment, storage of the remote operation program in the storage unit 23 is essential, but the storage of the game program is not essential. Here, in the following description, the ROM 211, RAM 212, and storage unit 23 of the control unit 21 in which various programs and data are stored are collectively referred to as a storage device of the game device 2. do.

The image processor 213 includes a Graphics Processing Unit (GPU) capable of generating a game screen. A video RAM (VRAM) 24 is connected to the image processor 213. A display device 25 is connected to this VRAM 24.

The voice processor 214 includes a digital signal processor (DSP) that generates game sound. The voice processor 214 transmits the generated game voice to the amplifier 26 including a D/A converter. The amplifier 26 amplifies such an audio signal and transmits it to the speaker 27.

The controller 28 is connected to the operation unit 215 by wireless or wired communication. The controller 28 includes a cross button, a push switch, a joystick, a mouse, a keyboard, and a touch panel. Further, the operation unit 215 detects an operation signal through the controller 28 by the user, and transmits the operation signal to the CPU 210.

The communication unit 22 is a communication device that communicates with the intermediary device 6 and the relay device 53 via the communication network 4.

The intermediary device 6 includes a control unit 61, a communication unit 62, and a storage unit 63. The control unit 61 is comprised of, for example, a processor and an arithmetic unit including a memory. Specifically, such a calculator is constituted by, for example, a microcontroller, an MPU, a field programmable gate array (FPGA), a programmable logic controller (PLC), a computer, a personal computer, and the like. The control unit 61 may be constituted by a single operator that performs centralized control, or may be constituted by a plurality of operators that perform distributed control. The communication unit 62 is a communication device that communicates with the game device 2 and the relay device 53 via the communication network 4. The storage unit 63 is a readable storage device capable of reading and writing, and is, for example, a hard disk, a flash memory, an optical disk, or the like. The control unit 61 controls the operation of each unit of the intermediary device 6. Various programs and data for controlling the operation of the intermediary device 6, such as a program for linking the game device 2 to the robot system 5, are stored in the memory or the storage unit 63 of the control unit 61.

The relay device 53 includes a control unit 55, a communication unit 56, and a storage unit 57. The control unit 55 is comprised of, for example, a processor and an arithmetic unit including a memory. Specifically, such a calculator is constituted by, for example, a microcontroller, an MPU, a field programmable gate array (FPGA), a programmable logic controller (PLC), a computer, a personal computer, and the like. The control unit 55 may be constituted by a single operator that performs centralized control, or may be constituted by a plurality of operator units that perform distributed control. The communication unit 56 is a communication device that communicates with the game device 2, the intermediary device 6, the robot 51, and the peripheral device 52 via the communication network 4. The storage unit 57 is a readable storage device capable of reading and writing, and is, for example, a hard disk, a flash memory, an optical disk, or the like. The control unit 55 controls the operation of the relay device 53. Various programs and data for controlling the operation of the relay device 53 are stored in the memory or the storage unit 57 of the control unit 55.

3 schematically shows an example of the robot system 5. In such a robot system 5, the robot 51 performs a work of picking the work W conveyed by the conveyor 52a. In the robot system 5 shown in Fig. 3, the robot 51 is an industrial robot. The robot 51 includes a robot main body 51a that is a target to be remotely operated by an operator, and a robot controller 51b that controls the operation of the robot main body 51a. The robot main body 51a shown in FIG. 3 is a vertical articulated robot arm with a tool mounted at its distal end. In this example, as a tool, a workpiece W is provided at the distal end of the vertical articulated robot arm. It is equipped with a gripping hand that can be gripped. The robot controller 51b is equipped with a processor, and executes the decoding operation processing of stored programs or various signals input from the outside by the processor to control the operation of the robot main body 51a, output signals from various output ports, etc. In charge of In addition, the robot system 5 of FIG. 3 is a peripheral device 52, which captures one or more (two in this example) imaging of the work situation of the conveyor 52a and the robot 51 that conveys the work target. A device 52b, and a sensor 52c for detecting the position of the work W are provided.

Here, the configuration of the robot system 5 shown in FIG. 3 is an example, and the types of the robot 51 and the peripheral device 52 are configured according to the work contents of the robot 51. For example, the work of the robot 51 may be, for example, a painting work, a lunch-packing work, a welding work, in addition to the picking work. In addition, the robot 51 may not be a vertical articulated robot, for example, an industrial robot such as a horizontal articulated robot, a parallel link type robot, a polar coordinate robot, a cylindrical coordinate robot, or a rectangular coordinate robot. It may be good. Further, the conveying device as the peripheral device 52 that conveys the work W as a work object may be a conveying device other than a conveyor. The peripheral device 52 may include a moving device that moves the robot main body 51a. The at least one sensor as the peripheral device 52 may be a sensor that detects the position or posture of the robot 51 in place of or in addition to a sensor that detects the position of the work W. The at least one sensor as the peripheral device 52 includes a sensor that detects the position, direction, or direction of the object to be inspected. Further, the robot system 5 may be provided with a plurality of imaging devices 52b as the peripheral device 52. As shown in FIG. 3, the imaging device 52b may be attached to the robot main body 51a, and may be installed in a fixed position in a work space.

(Functional composition)

4 is a block diagram showing the functional configuration of the control unit 21 of the game device 2. The control unit 21 of the game device 2, as a functional configuration, includes a communication control unit 31, an operation-side time management unit 32, a state information acquisition unit 33, a simulation unit (prediction unit) 34, an image display unit. (35) A communication delay measurement unit 36, a deviation detection unit 37, and a model correction unit 38 are provided. Such a functional unit is functionally realized by cooperation with a predetermined program stored in the storage device of the game device 2. Here, the "data generation program" of the present invention is included in the predetermined program stored in the storage device of the game device 2.

The communication control unit 31 controls the communication unit 22 to send the above-described operation request and designation information to the intermediary device 6 or to receive list information from the intermediary device 6. Further, the communication control unit 31 controls the communication unit 22 to receive information from the intermediary device 6 for communication connection with the robot system 5 that the intermediary device 6 has associated with the game device 2. do. Further, the communication control unit 31 controls the communication unit 22 to send operation information generated by the operator operating the controller 28 to the relay device 53 of the corresponding robot system 5.

When the operator manipulates the controller 28, the operation-side time management unit 32 maintains a constant time until the robot 51 operates based on the operation. Manage.

The state information acquisition unit 33 and the simulation unit 34 generate at least some data used for generation of an image displayed on the display device 25. Specifically, on the display device 25, a work space model modeled of a work space in which the robot (hereinafter, "real robot") 51 actually exists (hereinafter, "actual work space") is displayed as a moving image. The work space model includes a robot model and a surrounding object model arranged in a virtual work space. The robot model is a model of an actual robot 51. The peripheral object model is a model of a predetermined peripheral object (hereinafter, "real peripheral object") in the periphery of the actual robot 51. The actual peripheral object includes the peripheral device 52 and the work W positioned around the robot 51, and the peripheral object model includes a peripheral device model and a work model corresponding thereto. In the generation of an image displayed on the display device 25, robot model data or peripheral object model data is used.

The robot model data contains static information of the actual robot 51. The static information of the robot model includes, for example, structural information indicating the structure of the actual robot 51 (the number of joints and link lengths of the robot body 51a, the structure of the tool, etc.), and the position and/or posture before starting work. Information (for example, rotation angle information of the servo motor included in the robot body 51a) and the like are included. Further, the robot model data includes dynamic information of the robot model, that is, operation information (command) of an operator with respect to the controller 28. This operation information is used to operate the robot model, and is also used for the operation of the actual robot 51, which is sent from the game device 2 to the actual robot 51 via the communication network 4. That is, when the operator manipulates the controller 28 while viewing the display screen of the display device 25 to operate the robot model on the display screen, the actual robot 51 at the work site also operates in the same manner. However, as described later, the operation of the robot model is delayed for a predetermined time, so that the actual robot 51 operates.

In the surrounding object model data, static information of the actual surrounding object is included. In the static information of the actual surroundings, structural information of the peripheral device 52, information showing the position and/or posture of the peripheral device 52 before starting work, and shape data of the workpiece W as a work target of the actual robot 51 B. Structure information, and information indicating the position and/or posture of the work W before starting work are included. In addition, the surrounding object model data includes information for predicting the position or posture of the actual surrounding object before a predetermined time. The state information acquisition unit 33 and the simulation unit 34 perform such prediction. In this embodiment, the game device 2 including the state information acquisition unit 33 and the simulation unit 34 corresponds to the "data generating device" of the present invention.

Specifically, the state information acquisition unit 33 acquires state information indicating the state of an actual peripheral object such as the peripheral device 52 or the work W around the actual robot 51. Then, the simulation unit 34 simulates a change in the position or posture of an actual surrounding object over time based on the state information. For example, the state information acquisition unit 33 has information indicating the conveying speed set in the conveying device (conveyor 51a in this example) as the peripheral device 52 at a certain point in time, and the work W conveyed to the conveying device. When information indicating the position of is acquired as state information, the simulation unit 34 can easily calculate the position and posture of the work W ahead of time from the conveying speed information and the work position information. In this way, the simulation unit 34 predicts the actual state of the surrounding object after a predetermined time from the present through the simulation. Then, the simulation unit 34 generates the predicted result as surrounding object model data used for the aptitude of the surrounding object model.

The image display unit 35 displays a work space model created based on the robot model data and the surrounding object model data on the display device 25. For example, the image display unit 35 arranges a virtual camera in a virtual work space in which a robot model created based on the robot model data and the surrounding object model data and the surrounding object model are arranged. An image captured by such a virtual camera is displayed on the display device 25. The position, direction, zoom, and the like of the virtual camera may be determined in advance, and may be changed, for example, according to an operator's operation on the controller 28. The position or direction of the virtual camera in the virtual work space may correspond to the position or direction of the imaging device 52b in the actual work space, respectively.

Details of the communication delay measurement unit 36, the deviation detection unit 37, and the model correction unit 38 will be described later.

5 is a block diagram showing the functional configuration of the control unit 55 of the relay device 53. The control unit 55 of the relay device 53 includes a communication control unit 71 and a robot-side time management unit 72 as a functional configuration. Such a functional unit is functionally realized by cooperation with a predetermined program stored in the control unit 55 and/or the storage unit 57 of the relay device 53.

The communication control unit 71 controls the communication unit 56 to receive operation information generated by an operator operating the controller 28 from the game device 2.

When the operator manipulates the controller 28, the robot-side time management unit 72 determines the time on the robot system 5 side so that the time until the robot 51 operates based on the operation is kept constant. Manage.

(Processing flow)

Next, execution processing in each of the game device 2 and the relay device 53 will be described with reference to Figs. 6-8.

6 is a diagram showing a processing flow of the robot 51 before starting work in each of the game device 2 and the relay device 53. Before starting work, the robot model, the peripheral device model, and the work model in the work space model are created so as to be in the same state as the actual robot 51, the peripheral device 52, and the work W, respectively.

Specifically, when the game device 2 and the relay device 53 are connected by the mediation device 6, the communication control unit 71 of the relay device 53 first, for creating a working space model before the start of work. The information is transmitted to the game device 2 (step S201). The information for creating the work space model before the start of work includes state information indicating the state of the actual surrounding objects in the vicinity of the actual robot 51.

In this embodiment, the state information includes imaging information generated by imaging the peripheral device 52 and the work W by the imaging device 52b installed in the actual work space. Further, the state information includes detection information of the sensor as the peripheral device 52. The detection information of the sensor includes, for example, information indicating whether the work W is at a predetermined position in the work space, information indicating the position or posture of the work W, and the like. Further, the state information includes setting information set in the peripheral device 52. In the setting information, for example, when the robot system 5 includes a conveying device as the peripheral device 52, the conveying speed and conveying interval set for the conveying device may be included. The conveyance interval may be a distance between the workpieces W to be conveyed, and from the time point when any workpiece W is conveyed to a predetermined position in front of the robot 51, the next workpiece W is brought to a predetermined position. The time interval until conveyance, etc. may be sufficient.

In addition, the state information sent from the relay device 53 to the game device 2 in step S201 also includes information indicating the actual state of the robot 51. The information indicating the state of the actual robot 51 may include, for example, posture information or position information of the actual robot 51 stored by the robot controller 51b. Further, the information indicating the actual state of the robot 51 may include imaging information obtained by the imaging device 52b, which is the peripheral device 52, or detection information of the sensor 52c.

On the game device 2 side, the state information acquisition unit 33 acquires the state information received from the relay device 53 (step S101). Then, the simulation unit 34 generates a work space model in the same state as the actual work space before the start of work, based on the state information acquired by the state information acquisition unit 33 (step S102). The state information includes, for example, shape data and structure information of the work W, the position and direction of the work W, location information and structure information of the peripheral device 52, setting information, and the like.

Specifically, the simulation unit 34 includes position information (position coordinates in the coordinate system set for the actual work space) of the actual robot 51 before the start of work, and attitude information (the servo motor included in the robot main body 51a). Based on rotation angle information, etc.), a robot model is created so that the state (position, posture, etc.) of the robot model becomes the same as that of the actual robot 51. Further, the simulation unit 34 creates a surrounding object model so that the surrounding object model is in the same state as the actual surrounding object based on the state information of the actual surrounding object before starting work.

The image display unit 35 generates an image of the work space model created in step S102 and displays it on the display device 25 (step S103). In this way, preparations for commencement of work are made.

7 is a processing flow after the robot 51 starts working in each of the game device 2 and the relay device 53. In addition, FIG. 8 is a diagram for explaining an example of processing executed over time in each of the game device 2 and the relay device 53.

After step S103, the control unit 21 of the game device 2 determines whether or not an operation instructing the start of work has been performed (step S105). When there is no instruction to start work (step S105: No), it becomes a state of waiting until there is an instruction to start work. When an operation for instructing the controller 28 to start work is performed by the operator (step S105: Yes), the communication control unit 31 transmits a work start instruction to the relay device 53 (step S106). )). In addition, the operation-side time management unit 32 stores the transmission time of the operation start instruction as the operation-side reference time t1 in a storage device (RAM 212 or storage unit 23, etc.) of the game device 2 (See step S107, Fig. 8).

On the robot system 5 side, when the communication control unit 71 of the relay device 53 receives a work start instruction (step S202), the communication control unit 71 starts a predetermined time (Δt) from the reception time of the work start instruction. The time after waiting for) is set as the work start time, and a work start instruction is sent to the robot 51 and the peripheral device 52 (step S203). Further, the robot-side time management unit 72 stores a time after a predetermined time Δt from the reception time of the work start instruction as the robot-side reference time t2 (step S204, see Fig. 8).

Thus, as will be described later, in the work space model displayed on the display device 25, the robot model starts from the operation-side reference time t1, while in the actual work space, the robot-side reference time ( The actual robot work is started from t2). In other words, the actual work space is delayed by the difference (t2-t1) of the reference time compared to the work space model, and the work proceeds. For example, the operation start instruction sent from the game device 2 to the robot system 5 may include an instruction for shifting the peripheral device 52 from the stationary state to the operating state. For example, the work model of the work space model starts to be conveyed from the operation-side reference time t1 by the transfer device model, and the actual work W of the actual work space is transferred by the transfer device to the robot-side reference time t2. ) May start to be returned.

In the game device 2, after setting the reference time t1 in step S107, remote operation of the robot 51 becomes possible. That is, when an operation for operating the robot 51 is performed on the controller 28 (step S108: Yes), operation information is generated. The simulation unit 34 simulates the operation of the robot model based on the operation information (step S109). Further, the communication control unit 31 transmits, to the relay device 53, the elapsed time T from the reference time t1 indicating the time elapsed until the operation corresponding to the operation information is performed, together with the operation information. It transmits (step S110).

On the robot system 5 side, when the communication control unit 71 of the relay device 53 receives operation information and time information from the game device 2 as a set, after the elapsed time T from the reference time t2 , An operation command is sent to the robot 51 to execute an operation based on the received operation information (step S205). Here, in step S110, instead of the elapsed time T, time information indicating the time at which the operation was performed in correspondence with the operation information, such as time information indicating the operation time, may be sent.

Further, regardless of whether or not it is determined that there has been an operation in step S108, the simulation unit 34 simulates the actual state of the surrounding objects (step S111). Specifically, the simulation unit 34 predicts a state of an actual surrounding object after a predetermined time from the current point based on the state information, and generates the predicted result as peripheral object model data used to create the surrounding object model. . Here, the state information may not be obtained by the state information acquisition unit 33 in step S101 before starting the work, or may be information acquired after starting the work. That is, the latest information may be sequentially sent from the robot system 5 to the game device 2, or the state information acquisition unit 33 may sequentially acquire the latest information. The simulation unit 34 may predict the actual state of surrounding objects based on the latest information.

The image display unit 35 displays an image representing the work space model on the display device 25 based on the data generated based on the simulation results in steps S109 and S111 (step S112). The image display unit 35 displays a working space model of a situation that is advanced by a difference (t2-t1) between the robot-side reference time t2 and the operation-side reference time t1 with respect to the actual working space situation. In other words, as shown in Fig. 8, with respect to the working space model displayed on the display device 25, the actual working space is the difference between the robot-side reference time t2 and the operation-side reference time t1 (t2 − It is delayed by t1) and the work proceeds.

Until an operation for ending a job is performed on the controller 28 or until a certain job is completed (step S113: No), steps S108 to S112 are repeated.

In the step S203, the robot-side time management unit 72 does not set the reception time of the work start instruction to the robot-side reference time t2, and waits for a predetermined time (Δt) from the reception time of the work start instruction. The time is set as the robot-side reference time t2. In this way, by setting a certain interval between the work-side reference time t1 and the robot-side reference time t2, the communication delay between the game device 2 and the robot system 5 (see Δd1 and Δd2 in Fig. 8) Is absorbing the fluctuations of

Here, the waiting time Δt may be determined based on an actual communication delay between the game device 2 and the robot system 5 (in detail, the relay device 53). The communication delay measurement unit 36 measures the communication delay between the game device 2 and the robot system 5 (relay device 53 in more detail) corresponding thereto. Communication delay is measured by a known method. The robot side time management unit 72 determines the length of the waiting time Δt, that is, the robot side reference time t2, according to the degree of fluctuation of the communication delay measured by the communication delay measurement unit 36 before step S105. The difference (t2-t1) between the and the operation-side reference time t1 may be set.

In addition, in this embodiment, when repeating steps S108 to S112, the work space model is periodically corrected. The shift detection unit 37 detects the degree of shift between the situation of the working space model displayed on the display device 25 at a predetermined time point and the actual work space situation after a predetermined time from the predetermined point in time. In more detail, the deviation detection unit 37 includes the situation of the working space model at a predetermined point in time, and the difference between the robot-side reference time t2 and the operation-side reference time t1 (t2-t1) from the predetermined point in time. The degree of deviation between the actual working space and the situation after that is detected.

For example, the misalignment detection unit 37 predicts the current predicted by the simulation unit 34 based on state information indicating the state of the actual work space at a certain time ta and the state information acquired before the time Ta. The degree of deviation may be detected by comparing the state of the working space model at time Ta. The state information indicating the state of the actual work space compared by the shift detection unit 37 may include state information used by the simulation unit 34 to predict the actual state of surrounding objects. The misalignment detection unit 37 receives, for example, imaging information of the actual work space at the time Ta from the robot system 5, and the actual surrounding objects (peripheral device 52 or work W) from the imaging information. The position or posture of the person may be determined by image recognition. The misalignment detection unit 37 determines the position or posture of the actual surrounding object determined from the imaging information at the time Ta, and a time corresponding to the actual work space at the time ta (that is, time (ta-(t2-t1))). Compare the position or posture of the surrounding object model in the working space model of. That is, the deviation detection unit 37 determines the state of the actual surrounding object (position or posture of the peripheral device 52 or the work W, etc.) determined from the imaging information at the time Ta, and the time (t2-t1) than the time Ta. ) And compare the state of the surrounding object model before.

Then, the model correction unit 38 corrects the work space model so that the deviation is eliminated when the deviation detected by the deviation detection unit 37 exceeds a preset range. For example, the model correction unit 38 may adjust the state information used in the simulation unit 34 by a deviation detected by the deviation detection unit 37. Alternatively, the model correction unit 38 may adjust the peripheral object model data generated by the simulation unit 34 by a deviation detected by the deviation detection unit 37. Here, after the model correction unit 38 performs the correction of the work space model, when the simulation unit 34 simulates the state of the actual surrounding material, the work space model is corrected to simulate the actual state of the surrounding material. do.

As described above, according to the game device 2 and the remote control system 1 as the data generating device of the present embodiment, the state of the actual surrounding objects after a predetermined time from the current point is predicted by the simulation unit 34. , The prediction result is generated as peripheral object model data used to create a peripheral object model displayed on the display device 25. Accordingly, even between the state of the surrounding object model and the actual state of the surrounding object, it is possible to create the same time difference as between the robot model and the actual robot. Accordingly, since the time axis of the robot model and the surrounding object model can be matched, the influence of the communication delay on the operator's operation can be suppressed.

In addition, in the present embodiment, the deviation detection unit 37 detects the degree of deviation between the situation of the working space model displayed on the display device 25 at a predetermined time point and the actual work space situation after a predetermined time from a predetermined point in time. Then, the model correction unit 38 corrects the work space model so that the deviation is eliminated when the deviation detected by the deviation detection unit 37 exceeds a preset range. Accordingly, it is possible to suppress a shift between the situation of the work space model displayed on the display device 25 and the situation of the actual work space.

(Other Examples)

The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention.

For example, in the above embodiment, the game device 2 is exemplified as the operation terminal, but the operation terminal of the present invention may not be the game device 2. The operation terminal may be provided with a manipulator that accepts an operator's operation and a display device that the operator can visually check. For example, the game device 2 may be one of a personal information terminal (PDA (Personal Data Assistant)), a smart phone, a personal computer, a tablet, and a robot-only remote control unit, in addition to various known game devices.

Further, for example, the control unit of the intermediary device 6 or another server device may function as a state information acquisition unit and a prediction unit by executing a predetermined program. That is, the "data generating device" of the present invention may not be an operation terminal operated by an operator, or may be a device that communicates with the operation terminal. For example, the data generating device of the present invention may be a robot controller 51b, a relay device 53, an intermediary device 6, or a server device different from the intermediary device 6. The data generating apparatus of the present invention may not include a part or all of the communication delay measurement unit 36, the deviation detection unit 37, and the model correction unit 38.

In addition, the "data generation program" of the present invention is a robot controller 51b, a relay device 53, an intermediary device 6, and an intermediary device 6 in place of or in addition to the storage device of the game device 2 as an operation terminal. ) May be stored in a storage device provided in at least one of the server devices other than ). In addition, the ``data generation program'' of the present invention is executed on a computer in which at least one of the robot controller 51b, the relay device 53, the mediation device 6, and a server device different from the mediation device 6 is each built-in. , It is good if the computer functions as a state information acquisition unit and a prediction unit. The processing flow of the game device 2 and the relay device 53 described with reference to Figs. 6 and 7 also does not limit the present invention.

Further, the remote operation system 1 may be provided with only one game device 2. Further, the remote operation system 1 may include only one robot system 5. In addition, the remote operation system 1 does not need to be provided with the intermediary device 6. Further, in the robot system 5, the relay device 53 and the robot controller 51b may be integrally configured. That is, the robot system 5 may be provided with one control device that also serves as the respective functions of the relay device 53 and the robot controller 51b.

In addition, in the present invention, the actual robot may not be an industrial robot, but may be a robot that operates according to an operator's operation of an operation terminal. For example, the actual robot of the present invention may be a service robot that provides services such as nursing care, medical care, transportation, cleaning, and cooking. In addition, the actual robot of the present invention may be a humanoid. The actual robot operates based on the operation information generated by the operation of the manipulator by the operator, but the actual robot may operate based on not only the operation information but also a preset task program.

In the above embodiment, the data generation program includes a state information acquisition step of acquiring state information indicating a state of an actual surrounding object, and, based on the state information, predicts the state of the actual surrounding object after the present point, and a prediction result The prediction step of generating the peripheral object model data used to create the peripheral object model displayed on the display is executed by the controller 21 of the game device 2 as an operation terminal, but the data generation of the present invention The program may execute the above-described state information acquisition step and prediction step on another computer. For example, in the data generation program of the present invention, the above-described state information acquisition step and prediction step are performed in a server device different from the robot controller 51b, the relay device 53, the mediation device 6, and the mediation device 6 At least one of them may be implemented in a controller (computer) provided. The data generation program may be distributed and stored in a plurality of storage devices. For example, a part of the data generation program may be stored in the storage device of the operation terminal, and others may be stored in another storage device (for example, the relay device 53). In addition, the data generation program of the present invention detects a degree of deviation between the situation of the work space model displayed on the display at a predetermined time point and the situation of the actual work space after the predetermined time from the predetermined time point. The discrepancy detection step may be performed on the computer. Further, in the data generation program of the present invention, when the deviation detected by the deviation detection step exceeds a preset range, the computer may execute a model correction step for correcting the working space model so that the deviation is eliminated. .

1: remote operation system
2: Game device (operation terminal)
4: communication network
5: robot system
6: intermediary device
25: display device (indicator)
28: controller (manipulator)
33: state information acquisition unit
34: simulation unit (prediction unit)
35: image display unit
36: communication delay measurement unit
37: misalignment detection unit
38: model correction unit
51: robot
51a: robot body
51b: robot controller
52: peripheral device
52a: conveyor
52b: imaging device
52c: sensor
53: relay device
55: control unit

Claims (11)

A remote control system comprising an operation terminal having a manipulator that accepts an operator's operation and an indicator that the operator can visually check, and a real robot installed in an actual work space and connected through a network capable of data communication with the operation terminal In, as a data generating device for generating at least some data used to generate the image displayed on the display,
On the display, a working space model modeling the actual working space is displayed as a video,
The working space model includes a robot model modeling the real robot, and a surrounding object model modeling the real surrounding objects around the real robot,
The robot model is created to operate according to the manipulation of the manipulator with respect to the manipulator,
The data generating device,
A state information acquisition unit that acquires state information indicating the state of the actual surrounding object;
Predicting the actual state of the surrounding object after a predetermined time from the current point based on the state information, and generating the predicted result as surrounding object model data used to create the surrounding object model displayed on the display. A data generating device comprising a prediction unit. The method of claim 1,
The actual surrounding object includes at least one of a work target of the actual robot, a transport device for transporting the work, and a moving device for moving the real robot. The method according to claim 1 or 2,
The status information includes imaging information generated when an imaging device installed in the working space captures the actual surrounding objects. The method according to any one of claims 1 to 3,
The status information includes setting information set in the peripheral device as the actual surrounding object. The method according to any one of claims 1 to 4,
And a deviation detection unit configured to detect a degree of deviation between the situation of the working space model displayed on the display at a predetermined time point and the situation of the actual work space after the predetermined time from the predetermined time point. Data generation device. The method of claim 5,
And a model correction unit for correcting the working space model so that the deviation is eliminated when the deviation detected by the deviation detection unit exceeds a preset range. The method according to any one of claims 1 to 6,
The data generating device, wherein the operation terminal is a game device including a controller as the operation device. The method according to any one of claims 1 to 7,
The operation terminal is at least one of a personal information terminal (PDA (Personal Data Assistant)), a smart phone, a personal computer, a tablet, and a robot-only remote control device. A remote control system comprising an operation terminal having a manipulator that accepts an operator's operation and an indicator that the operator can visually check, and a real robot installed in an actual work space and connected through a network capable of data communication with the operation terminal In, As a data generation method for generating at least a portion of the data used in the generation of the image displayed on the display,
On the display, a working space model modeling the actual working space is displayed as a video,
The working space model includes a robot model modeling the real robot, and a surrounding object model modeling the real surrounding objects around the real robot,
The robot model is created to operate according to the manipulation of the manipulator with respect to the manipulator,
The data generation method,
A state information acquisition step of acquiring state information indicating a state of the actual surrounding object;
Predicting the actual state of the surrounding object after a predetermined time from the current point based on the state information, and generating the predicted result as surrounding object model data used to create the surrounding object model displayed on the display. A data generation method comprising a prediction step. A remote control system comprising an operation terminal having a manipulator that accepts an operator's operation and an indicator that the operator can visually check, and a real robot installed in an actual work space and connected through a network capable of data communication with the operation terminal In, as a data generation program that generates at least a portion of data used for generation of an image displayed on the display by executing it on a computer,
On the display, a working space model modeling the actual working space is displayed as a video,
The working space model includes a robot model modeling the real robot, and a surrounding object model modeling the real surrounding objects around the real robot,
The robot model is created to operate according to the manipulation of the manipulator with respect to the manipulator,
The data generation program,
A state information acquisition step of acquiring state information indicating a state of the actual surrounding object;
Predicting the actual state of the surrounding object after a predetermined time from the current point based on the state information, and generating the predicted result as surrounding object model data used to create the surrounding object model displayed on the display. A data generation program, characterized in that the prediction step is executed by the computer. A remote control system comprising an operation terminal having a manipulator that accepts an operator's operation and an indicator that the operator can visually check, and a real robot installed in an actual work space and connected through a network capable of data communication with the operation terminal As,
On the display, a working space model modeling the actual working space is displayed as a video,
The working space model includes a robot model modeling the real robot, and a surrounding object model modeling the real surrounding objects around the real robot,
The robot model is created to operate according to the manipulation of the manipulator with respect to the manipulator,
The remote control system includes a state information acquisition unit that acquires state information indicating a state of the actual surrounding object, and, based on the state information, predicts the state of the actual surrounding object after a predetermined time from the present point, while predicting And a data generating device including a prediction unit that generates a result of the surrounding object model data used to create the surrounding object model displayed on the display.

Images