JP2022187852A - Robot system and remote control method - Google Patents

Abstract

To properly operate a robot which is controlled remotely.SOLUTION: A robot system for an operator to control a robot comprises: a camera part that photographs a circumference of the robot to obtain an image; an image display part that displays the image obtained by the camera part toward the operator; a visual line detecting part that detects a visual line of the operator; a body motion detecting part that detects a body motion of the operator; and a calculating part which comprises an object obtaining part that obtains an object from the image obtained by the camera part, a motion obtaining part that obtains the body motion of the operator detected by the body motion detecting part and a command generating part that generates a motion command to control a motion of the robot. The calculating part determines whether the object included in the image matches the visual line of the operator, which does not generate the motion command to control the robot which executes operation of motion input when the object does not match the visual line, and generates the motion command to control the robot which executes the operation of motion input when the object matches the visual line.SELECTED DRAWING: Figure 2

Description

The present invention relates to a robot system and a remote control method.

A technique is known in which an operator operates a robot at a remote position. For example, Patent Literature 1 describes a system for remotely operating a robot placed in an area where other operators are present. In addition, by displaying an avatar to the remote operator based on the location information detected by the robot, multiple operators can share 3D position recognition and feel as if they are in the same space. A possible technique is described.

JP 2019-168971 A

When an operator operates a robot from a remote location, the robot may behave unintendedly. For example, when the input for operating the robot is the operator's body motion, if the robot moves by a body motion other than the operation of the robot, there is a risk that the surroundings of the robot will be adversely affected.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a remote control system and a remote control method for appropriately operating a robot that is remotely controlled.

A robot system according to one aspect of the present invention is a robot system in which an operator operates a robot, and includes a camera unit that acquires an image by photographing the surroundings of the robot, and an image acquired by the camera unit that is directed to the operator. a line-of-sight detection unit that detects the line of sight of the operator; a body movement detection unit that detects the body movement of the operator; an object acquisition unit that acquires an object from the image acquired by the camera unit; a motion acquisition unit that acquires the operator's motion detected by the body motion detection unit; It is determined whether or not the object included in the image and the line of sight of the operator match, and if they do not match, the motion command is not generated for the robot executing the operation of the motion input, and they match. If so, the motion command is generated for the robot that executes the operation of the motion input.

A remote control method according to an aspect of the present invention is a remote control method in which an operator operates a robot, comprising: obtaining an image of the surroundings of the robot; obtaining an object from the obtained image; obtaining an operator's motion command based on the operator's body motion; detecting the line of sight of the operator in a state in which an image of the surroundings of the robot is displayed to the operator; It is determined whether the line of sight of the object to be viewed and the line of sight of the operator match. and generating the motion command to the robot that performs the manipulation of the motion input.

ADVANTAGE OF THE INVENTION According to this invention, the robot operated from a remote place can be operated appropriately.

FIG. 1 is a diagram for explaining a configuration example of a robot system according to an embodiment. FIG. 2 is a block diagram showing a configuration example of a remote control device and a robot control device according to the embodiment. FIG. 3 is a flow chart showing an example of processing of the robot operating device of the robot system according to the embodiment.

DETAILED DESCRIPTION OF THE INVENTION Embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that the present invention is not limited by this embodiment, and when there are a plurality of embodiments, a combination of each embodiment is also included. Further, in the following embodiments, the same parts are denoted by the same reference numerals, thereby omitting redundant explanations.

[Robot system]
A robot system according to an embodiment will be described with reference to FIG. FIG. 1 is a diagram illustrating a configuration example of a robot system according to an embodiment. As shown in FIG. 1 , the robot system 10 includes a robot (object to be manipulated) 12 , a remote control device (manipulation terminal) 14 , a robot control device 15 and a network 16 . The robot system 10 transmits an operation to the robot 12 input to the remote control device 14 by the operator to the robot 12 via the network 16 and the robot control device 15 . The robot 12 operates based on the received operation. Network 16 is, for example, the Internet network.

The robot 12 is at least partially physically mobile. The robot 12 may be equipped with a mechanism for moving on the ground, or may be equipped with a mechanism for moving relative to the main body like a robot arm. The robot 12 includes a control section 22 , a drive section 24 , a communication section 26 and an information acquisition section 28 .

The control section 22 controls the operation of each section of the robot 12 . The control unit 22 includes, for example, an arithmetic unit such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit), a RAM (Random Access Memory), a semiconductor memory device such as a flash memory, a hard disk, a solid state drive, or the like. and a storage device of The control unit 22 implements a control function by executing a program (for example, a program according to the present disclosure) stored in a storage device using a RAM or the like as a work area. Also, the control unit 22 may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 22 may be realized by a combination of software and hardware.

Drive 24 is the movable part of robot 12 . The driving unit 24 is a mechanism such as a wheel or a leg that moves the main body, or a mechanism that is installed in the main body such as a robot arm and moves relative to the main body. The driving section 24 is controlled by the control section 22 .

The communication unit 26 is a wireless communication unit that performs wireless communication. Specifically, the communication unit 26 is communicably connected to the robot control device 15 . That is, the robot 12 uses the communication unit 26 to wirelessly communicate with the robot control device 15 . Also, the robot control device 15 is communicably connected to the remote control device 14 via the network 16 .

The information acquisition unit 28 acquires information around the robot 12 . The information acquisition unit 28 includes a position sensor, a camera unit (image sensor), a distance sensor, a microphone, etc., and acquires current position information, surrounding three-dimensional shape, surrounding images, sounds, etc. as surrounding information. As a position sensor, a sensor that acquires the position of absolute coordinates such as GPS, a laser radar that detects the distance between surrounding obstacles (for example, LIDAR: Laser Imaging Detection and Ranging), an infrared irradiation unit and a light receiving sensor and an infrared sensor, a ToF (Time of Flight) sensor, and the like. Examples of the camera section (image sensor) include a CMOS image sensor and a CCD.

FIG. 2 is a block diagram showing a configuration example of the remote control device 14 and the robot control device 15 according to the embodiment, and the illustration of the network 16 is omitted. The remote control device 14 is a device with which an operator obtains information about the robot 12 and inputs an operation to the robot 12 by body motion. The remote control device 14 is, for example, a personal computer, an HMD (Head Mounted Display) device worn by the operator on the head, and a device (camera, sensor) that detects an operation input by body motion. Configurations are exemplified, but not limited to. The remote control device 14 includes an image display section 32 , a communication section 34 , a line-of-sight detection section 36 and a body motion detection section 38 .

The image display unit 32 outputs an image around the robot 12 to the operator. The image display unit 32 displays images acquired by the camera unit of the information acquisition unit 28 of the robot 12 .

The communication unit 34 is a long-distance wireless communication unit that performs long-distance wireless communication. Specifically, the communication unit 34 is communicably connected to the robot control device 15 via the network 16 . That is, the remote control device 14 uses the communication unit 34 to perform long-distance wireless communication with the robot control device 15 .

The line-of-sight detection unit 36 detects the line of sight of the operator. The line-of-sight detection unit 36 detects the positions of the pupils of the operator, and detects the viewing position, specifically which position of the image display unit 32 the operator is viewing. The line-of-sight detection unit 36 outputs the detection result to the robot control device 15 .

The body motion detector 38 detects gesture inputs such as motions of the operator's head, arms, and hands. The body motion detector 38 is a sensor (for example, an acceleration sensor) that is worn by the operator to detect the operator's motion, a sensor (for example, a camera) that detects the operator's motion from a photographed image of the operator, or the like. The body motion detector 38 outputs the body motion of the operator to the robot controller 15 .

A robot control device 15 is arranged in the area where the robot 12 is arranged and controls the operation of the robot. In this embodiment, the robot controller 15 is separate from the robot 12 . The robot control device 15 includes a communication section 50 , a camera section 52 , an ambient information acquisition section 54 , a calculation section 56 and a storage section 58 .

The communication unit 50 is a long-distance wireless communication unit that performs long-distance wireless communication. Specifically, the communication unit 50 is communicably connected to the remote controller 14 via the network 16 and is communicatively connected to the robot 16 by wireless communication. That is, the robot control device 15 uses the communication unit 50 to wirelessly communicate with the robot 12 and the remote control device 14 .

The camera unit 52 photographs an area including the robot 12 . Examples of the camera unit 52 include a CMOS image sensor and a CCD. The surrounding information acquisition unit 54 acquires the circumstances around the robot 12 . That is, the ambient information acquisition unit 54 acquires the information around the robot 12 acquired by the information acquisition unit 28 of the robot 12 via the communication unit 26 and the communication unit 50 .

The computing unit 56 executes a program (for example, a program according to the present disclosure) stored in the storage unit 58 by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like, using a RAM or the like as a work area. It is realized by Further, the calculation unit 56 may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The computing unit 56 may be realized by a combination of software and hardware. The calculation unit 56 includes an object acquisition unit 62 , a motion input acquisition unit 64 , a line-of-sight determination unit 66 , an execution determination unit 68 and an instruction generation unit 69 . Each unit will be described after the storage unit 58 .

The storage unit 58 stores various information. The storage unit 58 can be implemented by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or solid state drive. Storage unit 58 includes control program 70 , execution determination program 72 , instruction generation program 74 , and database 76 .

The control program 70 supervises the entire remote control process of the robot 12 . The control program 70 is a program that implements the operations of the object acquisition unit 62 , the motion input acquisition unit 64 , the line-of-sight determination unit 66 , the execution determination unit 68 , and the command generation unit 69 . The control program 70 also controls the processing of the execution determination program 72 and the instruction generation program 74 . The execution determination program 72 is a program that implements the processing of the execution determination unit 68 . The execution determination program 72 determines whether or not it is permissible to execute the motion command based on the surrounding information of the robot 12 and the line-of-sight information. The instruction generation program 74 is a program that implements the processing of the instruction generation unit 69 . The command generation program 74 generates commands (motion signals) to be output to the robot 12 based on the motion commands.

The database 76 contains reference data for extracting objects from images, reference data for line-of-sight determination, reference data for execution determination, and reference data for command generation. Note that the database 76 may include data necessary for processing of each unit.

Next, each part of the calculation part 56 will be described. The object acquisition unit 62 extracts objects around the robot 12 based on the image acquired by the information acquisition unit 28 of the robot 12 via the communication unit 50 . Objects are objects that affect the operation of the robot 12 . For example, if the robot 12 is a mechanism for moving items, the item to be moved is the object. Also, when the robot 12 moves, an object is an obstacle, for example, an object that needs to be avoided. The object acquisition section 62 may extract the object based on the image acquired by the camera section 52 .

The motion input acquisition unit 64 acquires a motion input to the robot 12 based on the operator's motion detected by the body motion detection unit 38 . The relationship between the operator's motion and the motion input to the robot 12 is stored in the database 76 in advance. The motion input acquiring unit 64 acquires a motion input to the robot 12 based on the data in the database 76 and the detected operator's motion.

The line-of-sight determination unit 66 determines which object the operator is looking at based on the position of the line of sight detected by the line-of-sight detection unit 36 and the position of the object acquired by the object acquisition unit 62 .

The execution determination unit 68 determines whether or not to execute the action based on the relationship between the line of sight and the object detected by the line-of-sight determination unit 66 and the action input acquired by the action input acquisition unit 64 . Specifically, the execution determination unit 68 determines whether or not the object viewed by the operator is the target of motion input. The execution determination unit 68 determines that the motion based on the motion input is to be performed when the object to which the motion input is to be input is viewed. On the other hand, the execution determination unit 68 determines not to execute the motion based on the motion input when the object to which the motion input is applied is not viewed.

When the execution determination unit 68 determines to execute the motion based on the motion input, the command generation unit 69 generates a motion command based on the motion input acquired by the motion input acquisition unit 64 and outputs the motion command to the robot 12 . On the other hand, when the execution determination unit 68 determines not to execute the motion based on the motion input, the command generation unit 69 does not generate the motion command based on the motion input acquired by the motion input acquisition unit 64 . Instead, it generates an instruction to stop the action or an instruction to perform a different action than the action input. The command generation unit 69 transmits the generated motion commands to the robot 12 via the communication unit 50 .

The robot 12 inputs the operation command received via the communication unit 26 to the control unit 22, and the driving unit 24 operates according to the operation command. This allows the operator to operate the robot 12 appropriately.

Next, the operation of the robot controller 15 of the robot system 10 will be described with reference to FIG. FIG. 3 is a flowchart illustrating an example of processing of the robot controller of the robot system according to the embodiment. The robot control device 15 controls the motion of the robot 12 by repeatedly executing the process shown in FIG.

The robot control device 15 acquires information about the surroundings of the robot 12 using the surroundings information obtaining unit 54 (step S12). The robot control device 15 acquires the image acquired by the robot 12 . The robot control device 15 may acquire the data acquired by the information acquisition unit 28 of the robot 12 in addition to the image of the surroundings. Further, the robot control device 15 may acquire an image including the robot 12 acquired by the camera unit 52 as surrounding information.

Next, the robot control device 15 extracts the object by the object obtaining section 62 (step S14). The object acquisition unit 62 analyzes the acquired image and extracts objects around the robot 12 . The number of objects to be extracted is not limited to one, and plural objects may be extracted. Also, if there are no objects around the robot 12, the objects may not be extracted.

Next, the robot control device 15 acquires a motion command by the motion input acquisition section 64 (step S16). The motion input acquisition unit 64 identifies the input motion corresponding to the operator's motion, that is, the operation content of the robot 12 based on the operator's motion acquired by the body motion detection unit 38 and the information in the database 76 . The motion input acquisition unit 64 acquires the identified operation content as a motion command.

Next, the robot control device 15 acquires information on the line of sight of the operator from the line of sight detector 36 (step S18). Information on the line of sight of the operator is information on which position of the image display unit 32 the operator is looking at. It should be noted that the robot control device 15 only needs to be able to extract an object, obtain an action command, and acquire the line of sight of the operator at the same timing as the action command. Therefore, the processing from step S12 to step S18 may be performed in a different order, or may be performed in parallel.

Next, the robot control device 15 uses the line-of-sight determination unit 66 and the execution determination unit 68 to determine whether the operator's line of sight is the object to be operated, that is, whether the operator is looking at the object to be operated (step S20). The robot control device 15 determines which object the operator is looking at based on the line-of-sight determination unit 66, and determines the object viewed by the target of the operation command based on the object and the operation command specified by the execution determination unit 68. It is determined whether or not.

When determining that the line of sight is on the object to be operated, that is, the operator is looking at the object to be operated (Yes in step S20), the robot control device 15 generates a motion command based on the motion input (step S22). ). Specifically, the command generator 69 generates a motion command based on the motion input and outputs it to the robot 12 .

If it is determined that the line of sight is not on the object to be operated, that is, the operator is not looking at the object to be operated (No in step S20), the robot control device 15 does not generate a motion command based on the motion input (step S24). ). Specifically, the command generator 69 does not generate an action command based on the action input. Instead, it generates a command to stop the motion or a command to perform a motion different from the motion input and outputs it to the robot 12 .

The robot control device 15 acquires information about the surrounding situation, motion input, and operator's line of sight from the robot 12, identifies the operator's motion input and the target object, and determines whether the object the operator is looking at is the target of operation. Based on this, an operation command to be sent to the robot 12 is determined.

Thereby, the operator can operate the robot 12 based on the action performed by looking at the object. In other words, even if the operator unintentionally performs a motion input to the robot 12, if the operator does not look at the object, the robot 12 can be placed in a state in which it does not perform the motion corresponding to the motion input. As a result, the robot 12 can be moved by triggering a motion input that the operator independently performs while looking at the object. Thereby, the robot control device 15 can prevent the robot 12 from moving unintended by the operator even when the operator remotely operates the robot 12 .

Further, when executing the motion input acquired by the motion input acquisition unit 64, the command generation unit 69 can calculate the relationship between the object and the robot 12 and generate a motion command including a motion for assisting the motion of the robot 12. preferable. For example, if the operator's body motion is to take an object with the robot arm, and the distance between the object and the robot 12 is greater than or equal to a predetermined distance, and the operator's line of sight is directed toward the object, the robot arm can move the robot arm to the object. It generates an action command to approach the object and an action command to pick up the object with the robot arm. In this case, an action command to bring the robot arm closer to the object is processing that assists the action. Also, the command generation unit 69 can calculate the relative position and distance between the robot 12 and the object using various sensors of the surrounding information acquisition unit 54, and based on the calculation results, assist the operation based on the operator's input operation. can do. In other words, an operation that needs to be complemented to execute an operator's input operation on an object is added as an operation instruction. This allows the operator to perform the intended operation on the object. Instructions for assisting this operation may be stored in the instruction generation program 74 or the database 76 as reference data for instruction generation.

The robot system 10 uses a body motion detection unit 38 that is worn by an operator and includes at least one of a sensor that detects the operator's motion and a sensor that detects the operator's motion from an image of the operator. is the operator's motion, the above processing can prevent the robot 12 from moving due to the operator's unintentional motion, and the operator's intended motion can be performed. be able to. As a result, for example, it is possible to prevent a surgical robot or the like from performing an unintended operation by remote control.

The body motion detection unit 38 may be configured as an input unit for physical input, voice input, gesture input, etc. to the input terminal. Examples of the input unit include a keyboard, mouse, touch panel, microphone, camera, and the like, but are not limited to these. The input unit outputs to the robot control device 15 an operation signal regarding an operator's operation. A motion input acquiring unit 64 of the robot control device 15 acquires a motion input to the robot 12 based on an operator's operation.

The robot control device 15 may use a learned program in which object extraction by the object acquisition unit 62 is learned by machine learning. That is, part of the execution determination program 72 may be a learned program. Specifically, the robot control device 15 performs learning using information obtained by extracting an object from an image as teacher data, and creates a learned program. Here, machine learning is an example of supervised learning, but may be unsupervised learning. The robot control device 15 learns the processing of the object acquisition unit 62 by machine learning, and can easily change by updating the teacher data of the learned program. As a result, it is possible to easily make changes according to differences in the objects to be operated by the robot 12 .

In the robot control device 15, the determination processing of the line-of-sight determination unit 66 and the execution determination unit 68 may also use a learned program learned by machine learning. That is, part of the execution determination program 72 may be a learned program. In this way, the determination process is also performed by machine learning, and the prediction model is re-learned, so that the prediction accuracy can be improved.

Although the robot control device 15 has been described as being separate from the robot 12 , it may be mounted on the robot 12 . In this case, the communication unit 50 of the robot control device 15 and the communication unit 22 of the robot 12 may be shared, and the robot control device 15 and the robot 12 may communicate directly instead of wirelessly. In addition, the camera unit 52 may be arranged in the vicinity of the robot 12 as a separate body from the robot 12 in order to photograph the entire robot 12 , or may be arranged in the vicinity of the robot 12 in order to photograph a specific viewpoint seen from the robot 12 . may be arranged as one with the

The robot control device 15 has been described as including all of the calculation unit 56 and the storage unit 58 . At least some elements of the computing unit 56 and the storage unit 58 may be arranged in a server (not shown) connected to the robot 12 and the network 16 . Furthermore, at least some elements of the storage unit 58 may be distributed over the network 16 . For example, the database 76 may be placed only on the server, or may be placed separately on the robot 12 and the server. The robot 12 periodically refers to the reference data for command generation from the database 76 arranged in the server via the communication unit 26 .

Here, let us consider a case where the robot control device 15 is composed only of the calculation unit 56 and the storage unit 58 . Then, the robot control device 15 is arranged at an arbitrary position on the network 16 (for example, the position of the remote control device 14 in FIG. 1). In this case, the robot control device 15 communicates with each unit via a telecommunications network. Also, the camera unit 52 may be arranged in the vicinity of the robot 12 as a separate unit, or may not be provided.

Although the embodiments of the present invention have been described above, the present invention is not limited by the contents of these embodiments. In addition, the components described above include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those within the so-called equivalent range. Furthermore, the components described above can be combined as appropriate. Furthermore, various omissions, replacements, or modifications of components can be made without departing from the gist of the above-described embodiments.

10 robot system 12 robot (object to be manipulated)
14 remote control device (control terminal)
15 robot control device 16 network 22 control unit 24 drive unit 26 communication unit 28 information acquisition unit 32 image display unit 34 communication unit 36 line of sight detection unit 38 body movement detection unit 50 communication unit 52 camera unit 54 ambient information acquisition unit 56 calculation unit 58 Storage unit 62 Object acquisition unit 64 Motion input acquisition unit 66 Line of sight determination unit 68 Execution determination unit 69 Instruction generation unit 70 Control program 72 Execution determination program 74 Instruction generation program 76 Database

Claims (5)

A remote control system in which an operator operates a robot,
a camera unit for capturing an image by capturing the periphery of the robot;
an image display unit that displays an image acquired by the camera unit toward an operator;
a line-of-sight detection unit that detects the line of sight of the operator;
a body motion detector that detects the body motion of the operator;
An object acquisition unit that acquires an object from the image acquired by the camera unit, a motion input acquisition unit that acquires the operator's motion detected by the body motion detection unit, and a motion command that controls the motion of the robot. a calculation unit comprising an instruction generation unit;
The computing unit determines whether or not the object included in the image and the line of sight of the operator match, and if they do not match, issues the motion command to the robot that executes the motion input operation. A robot system that, if not generated and matched, generates the motion command to the robot that performs the operation of the motion input. 2. The robot according to claim 1, wherein said body movement detection unit is attached to said operator and includes at least one of a sensor for detecting movement of said operator and a sensor for detecting movement of said operator from an image of said operator. system. A peripheral information acquisition unit that acquires a situation around the robot,
The calculation unit measures the distance between the robot and the object based on the information from the surrounding information acquisition unit, and generates the action command for moving the robot toward the object based on the measured distance. 3. The robot system according to claim 1 or 2, wherein: The robot system according to any one of claims 1 to 3, wherein said image display unit is a head-mounted display mounted on said operator's head. A remote control method in which an operator operates a robot,
obtaining an image of the surroundings of the robot;
obtaining an object from the obtained image;
obtaining a motion command of the operator based on the body motion of the operator;
a step of detecting the line of sight of the operator in a state in which an image of the surroundings of the robot is displayed on the operator;
determining whether or not the object included in the image and the line of sight of the operator match; and, if there is a match, generating the motion command to the robot that performs the operation of the motion input.

Images