JP2022157101A - Robot remote operation control device, robot remote operation control system, robot remote operation control method and program - Google Patents

Abstract

To provide a robot remote operation control device, a robot remote operation control system, a robot remote operation control method and a program which enable an operator to easily perform work.SOLUTION: A robot remote operation control device, which performs robot remote operation control by which movements of an operator are recognized and the movements of the operator are transmitted to a robot to operate the robot, comprises: an intention estimating part that estimates motion of the operator on the basis of a robot environment sensor value obtained by an environment sensor installed in the robot or in an environment around the robot and an operator sensor value showing the motion of the operator obtained by an operator sensor; and a control command generating part that generates a control command while reducing a degree of freedom of motion of the operator, by generating a control command suitable for degree of freedom of a portion of the motion of the operator, on the basis of the estimated motion of the operator.SELECTED DRAWING: Figure 2

Description

The present invention relates to a robot remote operation control device, a robot remote operation control system, a robot remote operation control method, and a program.

A control device has been proposed that allows a user to assist the operation of a robot. As such a control device, for example, a first information acquisition unit that acquires first user posture information indicating the posture of a first user who operates the robot, and a a second information acquiring unit for acquiring pre-change posture information indicating a pre-change posture which is the posture of the robot; A control device has been proposed that includes a determination unit that determines a target posture different from the posture of the first user as the posture of the robot based on the first user posture information acquired by the information acquisition unit (see Patent Document 1). ). The system described in Patent Literature 1 changes the posture of the robot to a posture that corresponds to the posture detected by the device worn by the operator.

In addition, in order to perform various tasks using a remote-controlled robot, it is necessary to accurately indicate and control hand targets (positions and postures) with six degrees of freedom in three-dimensional space. For example, when the robot is to open the lid of a plastic bottle in a state in which the vertical direction of the bottle is not perpendicular to the table, the operator performing the remote operation must move the lid while aligning the grip part with the inclination of the bottle in the vertical direction. instruct to turn

Japanese Patent No. 6476358

However, in the conventional technology, in order to realize various tasks using a remote-controlled robot, the operator needs to accurately grasp the situation of the remote robot and the object, and at the same time, advanced operation skills are required. work is limited and its efficiency decreases.
For this reason, in the prior art, it was difficult for the operator to determine and accurately control target values for the six degrees of freedom (position and orientation) in space during remote control, and to perform work.

The present invention has been made in view of the above problems, and provides a robot remote operation control device, a robot remote operation control system, a robot remote operation control method, and a program that facilitate an operator's work. intended to provide

(1) In order to achieve the above objects, a robot remote operation control device according to an aspect of the present invention recognizes a motion of an operator and transmits the motion of the operator to a robot to operate the robot. , based on the robot environment sensor value obtained by the robot or an environment sensor installed in the surrounding environment of the robot and the operator sensor value obtained by the operator sensor, which is the movement of the operator. an intention estimating unit that estimates an operator's motion; and a control command generator that generates a control command by reducing the degree of freedom of operator's motion.

(2) Further, in the robot remote control device according to the aspect of the present invention, the control command generation unit limits the degree of freedom to be controlled by the operator and the controllable range so that the robot can be controlled by the operator. The motion assistance may be performed with respect to the restricted degree of freedom in the motion instruction to the .

(3) Further, in the robot remote operation control device according to the aspect of the present invention, the control command generation unit determines that the distance between the gripping unit of the robot and the target object to be operated by the operator is outside a predetermined range. In the case of , if the distance between the gripping unit provided in the robot and the target object to be operated by the operator is within a predetermined range without reducing the degree of freedom of the operator's motion, Among the actions of the operator, the robot environment sensor values that reduce the degree of freedom of the actions of the operator may include captured image information and depth information.

(4) Further, in the robot remote operation control device according to the aspect of the present invention, the intention estimation unit inputs the robot environment sensor value and the operator sensor value into a learned intention estimation model to The motion of the operator may be estimated.

(5) Further, in the robot remote operation control device according to the aspect of the present invention, the operator sensor values are operator's sight line information and information on the posture and position of the operator's arms. It may be at least one of arm information.

(6) Further, in the robot remote operation control device according to the aspect of the present invention, the robot environment sensor values may include photographed image information and depth information.

(7) In order to achieve the above objects, a robot remote control system according to an aspect of the present invention includes a gripping unit that grips an object, recognizes movements of an operator, and transmits the movements of the operator to a robot. In the robot remote operation for operating the robot, the robot remote operation control device according to any one of the above (1) to (6) and the robot environment installed in the robot or in the surrounding environment of the robot. An environment sensor that detects a sensor value, and an operator sensor that detects the movement of the operator as an operator sensor value.

(8) In order to achieve the above objects, a robot remote control method according to an aspect of the present invention is a robot remote control method for recognizing a motion of an operator and transmitting the motion of the operator to a robot to operate the robot. in which the intention estimating unit obtains a robot environment sensor value obtained by the robot or an environment sensor installed in the robot's surrounding environment, and an operator sensor value obtained by an operator sensor, which is the movement of the operator; , and a control command generation unit, based on the estimated motion of the operator, generates a control command appropriate for a part of the degrees of freedom of the motion of the operator By generating , the degree of freedom of the operator's motion is reduced to generate a control command.

(9) In order to achieve the above object, a program according to an aspect of the present invention provides a program for remote control of a robot that recognizes a motion of an operator and transmits the motion of the operator to a robot to operate the robot. , a robot environment sensor value obtained by the robot or an environment sensor installed in the surrounding environment of the robot, and an operator sensor value obtained by an operator sensor, which is the movement of the operator. By estimating the motion of the operator and generating an appropriate control command for a part of the degrees of freedom of the motion of the operator based on the estimated motion of the operator, Reduce degrees of freedom to generate control commands.

According to (1) to (9), by substituting control target value generation for some of the six degrees of freedom depending on the situation, the operator limits the degrees of freedom to be controlled. makes it easier to do the work.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure which shows the outline|summary of the robot remote operation control system which concerns on embodiment, and the outline|summary of work. 1 is a block diagram showing a configuration example of a robot remote control system according to an embodiment; FIG. FIG. 4 is a diagram showing an example of a state in which an operator wears an HMD and a controller; FIG. 4 is a diagram showing an outline of intention estimation and control command generation processing according to the embodiment; FIG. 10 is a diagram showing a case where the operator's intention is to open the cap of the PET bottle; It is a figure which shows the case where an operator's intention is to grab a box. It is a figure which shows the example of the information which the memory|storage part which concerns on embodiment memorize|stores. FIG. 10 is a diagram showing an example of correction of a hand target of a gripping position with the assistance of a robot remote control device; 4 is a flow chart of an example of a processing procedure of the robot and the robot remote control device according to the embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, in the drawings used for the following description, the scale of each member is appropriately changed so that each member has a recognizable size.

[Overview]
First, an outline of work and processing performed by the robot remote control system will be described.
FIG. 1 is a diagram showing an outline of a robot remote control system 1 and an outline of work according to this embodiment. As shown in FIG. 1, the operator Us is wearing an HMD (head mounted display) 5 and a controller 6, for example. Environmental sensors 7 (7a, 7b) are installed in the work environment. In addition, the environment sensor 7c may be attached to the robot 2 . The environment sensors 7 (7a, 7b) include, for example, an RGB camera and a depth sensor as described later. The operator Us remotely operates the robot 2 by moving the hand or fingers wearing the controller 6 while watching the image displayed on the HMD 5 . In the example of FIG. 1, the operator Us remotely operates the robot 2 to grip the PET bottle obj on the table Tb and, for example, open the cap of the PET bottle. In the remote operation, the operator Us cannot directly view the motion of the robot 2, but can indirectly view the image of the robot 2 through the HMD 5. FIG. In this embodiment, based on the information acquired by the controller 6 worn by the operator and the information acquired by the environment sensor 7, the operator's intention (information about the target object, work content) is estimated and estimated. Based on the results, the degree of freedom to be controlled by the operator is limited to support the operation.

[Configuration example of robot remote control system]
Next, a configuration example of the robot remote control system 1 will be described.
FIG. 2 is a block diagram showing a configuration example of the robot remote control system 1 according to this embodiment. As shown in FIG. 2 , the robot remote control system 1 includes a robot 2 , a robot remote control device 3 , an HMD 5 , a controller 6 and an environment sensor 7 .

The robot 2 includes, for example, a control unit 21, a drive unit 22, a sound pickup unit 23, a storage unit 25, a power supply 26, and a sensor 27.
The robot remote control device 3 includes, for example, an information acquisition unit 31, an intention estimation unit 33, a control command generation unit 34, a robot state image creation unit 35, a transmission unit 36, and a storage unit 37.

HMD5 is provided with the image display part 51, the line-of-sight detection part 52 (operator sensor), the control part 54, and the communication part 55, for example. In addition, HMD5 may be equipped with the sensor which detects the inclination of an operator's head, etc., for example.

The controller 6 includes, for example, a sensor 61 (operator sensor), a control section 62, a communication section 63, and feedback means 64.

The environment sensor 7 includes, for example, an imaging device 71, a sensor 72, and a communication section 73.

The robot remote control device 3 and the HMD 5 are connected via a wireless or wired network, for example. The robot remote control device 3 and the controller 6 are connected via a wireless or wired network, for example. The robot remote control device 3 and the environment sensor 7 are connected via a wireless or wired network, for example. The robot remote control device 3 and the robot 2 are connected via a wireless or wired network, for example. Note that the robot remote control device 3 and the HMD 5 may be directly connected without going through a network. The robot remote control device 3 and the controller 6 may be directly connected without going through a network. The robot remote control device 3 and the environment sensor 7 may be directly connected without going through a network. The robot remote control device 3 and the robot 2 may be directly connected without going through a network.

[Function example of robot remote control system]
Next, an example of functions of the robot remote control system will be described with reference to FIG.
The HMD 5 displays the robot state image received from the robot remote control device 3 . The HMD 5 detects the movement of the operator's line of sight, etc., and transmits the detected line-of-sight information (operator sensor value) to the robot remote control device 3 .

The image display unit 51 displays the state image of the robot received from the robot remote control device 3 under the control of the control unit 54 .

The line-of-sight detection unit 52 detects the line-of-sight of the operator and outputs the detected line-of-sight information to the control unit 54 .

The control unit 54 transmits line-of-sight information based on the line-of-sight information detected by the line-of-sight detection unit 52 to the robot remote control device 3 via the communication unit 55 . The control unit 54 causes the image display unit 51 to display the state image of the robot transmitted by the robot remote control device 3 .

The communication unit 55 receives the robot state image transmitted by the robot remote control device 3 and outputs the received robot state image to the control unit 54 . The communication unit 55 transmits the operator state information to the robot remote control device 3 under the control of the control unit 54 .

The controller 6 is, for example, a tactile data glove worn on the operator's hand. The controller 6 detects operator arm information, which is information about the posture and position of the operator's arm, such as orientation, movement of each finger, and movement of the hand, using the sensor 61, and stores the detected operator arm information (operation sensor value) to the robot remote control device 3. The operator arm information (operator sensor values) includes the position and posture information of the hand, the angle information of each finger, the position and posture information of the elbow, and the information tracking the movement of each part. It contains a wide range of information.

The sensor 61 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic force sensor, or the like. Note that the sensor 61, which includes a plurality of sensors, tracks the movement of each finger using, for example, two sensors. The sensor 61 detects operator arm information and outputs the detected operator arm information to the controller 62 .

The control unit 62 transmits operator arm information based on the detection result detected by the sensor 61 to the robot remote control device 3 via the communication unit 63 . The controller 62 controls the feedback means 64 based on the feedback information.

The communication unit 63 transmits operator arm information to the robot remote control device 3 under the control of the control unit 62 . The communication unit 63 acquires the feedback information transmitted by the robot remote control device 3 and outputs the acquired feedback information to the control unit 62 .

The feedback means 64 feeds back feedback information to the operator under the control of the control section 62 . According to the feedback information, the feedback means 64 includes, for example, means for applying vibration (not shown), means for applying air pressure (not shown), and means for restraining hand movement (not shown) attached to the grip part 222 of the robot 2. (not shown), means for feeling temperature (not shown), means for feeling hardness or softness (not shown), or the like is used to feed back sensations to the operator.

The environment sensor 7 is installed, for example, at a position where the work of the robot 2 can be photographed and detected. The environment sensor 7 may be provided in the robot 2 or may be attached to the robot 2 . Alternatively, a plurality of environment sensors 7 may be provided, and may be installed in the work environment and attached to the robot 2 as shown in FIG. The environment sensor 7 transmits the photographed image and the detected detection result (environment sensor value) to the robot remote control device 3 .

The imaging device 71 is, for example, an RGB camera. The photographing device 71 transmits the photographed image to the robot remote control device 3 via the communication unit 73 . In addition, in the environment sensor 7, the positional relationship between the imaging device 71 and the sensor 72 is known.

Sensor 72 is, for example, a depth sensor. The sensor 72 transmits the detected sensor value to the robot remote control device 3 via the communication unit 73 . Note that the imaging device 71 and the sensor 72 may be distance sensors.

The communication unit 73 transmits the captured image and the sensor values detected by the sensor 72 to the robot remote control device 3 as environmental sensor information. The environment sensor 7 may detect the position information of the object using the photographed image and the sensor value, and transmit the detection result to the robot remote control device 3 as the environment sensor information. The data transmitted by the environment sensor 7 may be, for example, a point cloud having position information.

The behavior of the robot 2 is controlled according to the control of the control unit 21 when it is not remotely operated. When the robot 2 is remotely operated, the behavior of the robot 2 is controlled according to the grasping plan information generated by the robot remote control device 3 .

The control unit 21 controls the driving unit 22 based on control commands output from the robot remote control device 3 . Based on the information stored in the storage unit 25, the control unit 21 uses the image captured by the environment sensor 7 and the detection result to notify the three-dimensional position, size, shape, etc. of the target object in the captured image. obtained by the method of Note that the control unit 21 may perform speech recognition processing (speech segment detection, sound source separation, sound source localization, noise suppression, sound source identification, etc.) on the acoustic signal collected by the sound collection unit 23 . If the result of voice recognition includes an action instruction for the robot, the control unit 21 may control the action of the robot 2 based on the action instruction by voice. The control unit 21 generates feedback information and transmits the generated feedback information to the controller 6 via the robot remote control device 3 .

The driving section 22 drives each section (arms, fingers, legs, head, torso, waist, etc.) of the robot 2 under the control of the control section 21 . The drive unit 22 includes, for example, actuators, gears, artificial muscles, and the like.

The sound pickup unit 23 is, for example, a microphone array including a plurality of microphones. The sound pickup unit 23 outputs the collected sound signal to the control unit 21 . The sound pickup unit 23 may have a speech recognition processing function. In this case, the sound pickup unit 23 outputs the speech recognition result to the control unit 21 .

The storage unit 25 stores, for example, programs and threshold values used for control by the control unit 21, and temporarily stores voice recognition results, image processing results, control commands, and the like. Note that the storage unit 37 may also serve as the storage unit 25 . Alternatively, the storage unit 37 may also serve as the storage unit 25 .

A power supply 26 supplies power to each part of the robot 2 . Power source 26 may comprise, for example, a rechargeable battery or charging circuit.

The sensor 27 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic force sensor, each joint encoder, or the like. The sensor 27 is attached to each joint, head, etc. of the robot 2 . The sensor 27 outputs the detected result to the control unit 21 , the intention estimation unit 33 , the control command generation unit 34 , and the robot state image generation unit 35 .

The robot remote control device 3 detects the operator based on the operator detection value detected by the operator sensors (the line-of-sight detection unit 52 of the HMD 5 and the sensor 61 of the controller 6) and the environment sensor value detected by the environment sensor 7. A motion intention is estimated and a control command for the robot 2 is generated.

The information acquisition unit 31 acquires line-of-sight information from the HMD 5, acquires operator arm information from the controller 6, and sends the acquired line-of-sight information and operator arm information to the intention estimation unit 33 and the robot state image creation unit 35. Output. The information acquisition unit 31 acquires environment sensor information from the environment sensor 7 and outputs the acquired environment sensor information to the intention estimation unit 33 and the robot state image creation unit 35 . The information acquisition unit 31 acquires the detection result detected by the sensor 27 and outputs the acquired detection result to the intention estimation unit 33 and the robot state image creation unit 35 .

Based on the environment sensor information acquired by the information acquisition unit 31, the intention estimation unit 33 obtains information about the target object (robot environment sensor values) (name of the target object, position of the target object, inclination of the target object in the vertical direction, etc.). to estimate Note that the intention estimating unit 33 performs image processing (edge detection, binarization processing, feature amount extraction, Image enhancement processing, image extraction, pattern matching processing, etc.) are performed to estimate the name of the target object. The intention estimation unit 33 estimates the position of the target object, the tilt angle in the vertical direction, and the like based on the result of the image processing and the detection result of the depth sensor. In addition, the intention estimation unit 33 estimates the target object, for example, also using the operator's line-of-sight information detected by the HMD 5 . The intention estimation unit 33 estimates the motion intention of the operator based on the line-of-sight information and the operator arm information acquired by the information acquisition unit 31 and the information on the target object. The intention of the operator and the estimation method will be described later.

The control command generation unit 34 generates a control command for gripping an object based on the result estimated by the intention estimation unit 33, the detection result detected by the sensor 27, and the image captured by the environment sensor 7 and the detection result. As will be described later, the control command generator 34 limits the degree of freedom to be controlled by the operator and the controllable range, that is, reduces the degree of freedom of motion of the operator to generate the control command. The control command generator 34 outputs the determined control command information to the controller 21 .

The robot state image creation unit 35 creates a robot state image to be displayed on the HMD 5 based on the control command information generated by the control command generation unit 34 .

The transmission unit 36 transmits the robot state image created by the robot state image creation unit 35 to the HMD 5 . The transmission unit 36 acquires feedback information output by the robot 2 and transmits the acquired feedback information to the controller 6 .

The storage unit 37 stores the positional relationship between the imaging device 71 and the sensor 72 of the environment sensor 7 . The storage unit 37 stores information that restricts the degree of freedom to be controlled by the operator, that is, the controllable range, for each task to be assisted. The storage unit 37 stores model images to be compared in pattern matching processing of image processing. The storage unit 37 stores programs used for controlling the robot remote control device 3 . Note that the program may be on the cloud or network.

[Example of state in which the operator wears HMD 5 and controller 6]
Next, a state example in which the operator wears the HMD 5 and the controller 6 will be described.
FIG. 3 is a diagram showing an example of a state in which an operator wears the HMD 5 and the controller 6. As shown in FIG. In the example of FIG. 3, the operator Us wears the controller 6a on his left hand, the controller 6b on his right hand, and the HMD 5 on his head. Note that the HMD 5 and the controller 6 shown in FIG. 3 are examples, and the mounting method, shape, and the like are not limited to these.

[Example of intention estimation and control command generation]
Next, an outline of intention estimation and control command generation processing will be described.
FIG. 4 is a diagram showing an overview of intention estimation and control command generation processing according to the present embodiment.
As shown in FIG. 4 , the robot remote control device 3 acquires line-of-sight information from the HMD 5 , operator arm information from the controller 6 , environment sensor information from the environment sensor 7 , and detection results detected by the sensor 27 .

The information obtained from the robot 2 is information such as detection values of motor encoders provided at each joint of the robot 2 .
The information obtained from the environment sensor 7 includes images captured by the RGB camera, detection values detected by the depth sensor, and the like.

The intention estimation unit 33 estimates the operator's action intention based on the acquired information and the information stored in the storage unit 37 . The intention of the operator is, for example, the object to be operated, the content of the operation, and the like. The content of the operation is, for example, opening the lid of a plastic bottle.

Based on the information obtained as a result of the estimation by the intention estimation unit 33, the control command generation unit 34 limits the degree of freedom to be controlled by the operator and the controllable range according to the target object and work content. Then, the control command generator 34 generates a control command based on the result estimated by the intention estimator 33 .

[Description of examples of restrictions on the degree of freedom to be controlled by the operator and the controllable range]
Next, examples of restrictions on the degree of freedom to be controlled by the operator and the controllable range will be described with reference to FIGS. 5 and 6. FIG. Note that the xyz axes in FIGS. 5 and 6 are the xyz axes in the robot world. The robot remote control device 3 of the present embodiment performs the following restrictions on degrees of freedom and correction of hand targets based on xyz in the robot world.

(First operation content example)
FIG. 5 is a diagram showing a case where the operator's intention is to open the cap of the PET bottle.
The intention estimation unit 33 estimates the target object Obj1 based on the tracking result of the operator's hand, the operator's line of sight information, the image captured by the environment sensor 7, and the detection result. As a result, the intention estimation unit 33 estimates that the operation object Obj1 is the "PET bottle" based on the acquired information. Further, the intention estimation unit 33 detects, for example, the vertical direction of the PET bottle and the tilt of the PET bottle in the vertical direction, for example, the z-axis direction, based on the image captured by the environment sensor 7 and the detection result.

Next, the intention estimation unit 33 estimates the content of the operation based on the tracking result of the operator's hand, the operator's line of sight information, the image captured by the environment sensor 7, and the detection result. As a result, based on the acquired information, the intention estimation unit 33 estimates that the operation content is an action of “opening the cap of the PET bottle”.

In the case shown in FIG. 5, in the conventional method, the operator needs to remotely operate the gripper 221 of the robot 2 so that it is in the vertical direction of the PET bottle and also to remotely operate it to open the lid. there were. As described above, the conventional method has a problem that the operator himself/herself is required to have a high level of operation skill.

On the other hand, in this embodiment, the robot remote control device 3 replaces the control target value generation for some of the six degrees of freedom according to the situation, thereby limiting the degrees of freedom to be controlled by the operator. . In the example of FIG. 5, the robot remote control device 3 controls and assists the gripper 221 of the robot 2 so that the gripper 221 stays in the vertical direction. As a result, the operator can concentrate on instructing the rotation of the cap without worrying about the inclination of the PET bottle.

(Second operation content example)
FIG. 6 is a diagram showing a case where the operator's intention is to grab the box.
The intention estimation unit 33 estimates that the operation target Obj2 is a "box" based on the acquired information.
Next, based on the acquired information, the intention estimation unit 33 estimates that the operation content is the action of “grabbing a box”.

In the case shown in FIG. 6, the robot remote control device 3, for example, assists the position of the operation target Obj2 on the xy plane and the rotation around the xyz axis by controlling the gripper 221. FIG. This allows the operator to concentrate on instructing the translational movement of the z-axis position.

Note that the target object, work content, assistance content (degree of freedom), and the like shown in FIGS. 5 and 6 are examples, and the target object, work content, and assistance content are not limited to these. The content of the assistance may be anything that corresponds to the target object or the content of the work.

Here, an example of information stored in the storage unit 37 used for assisting work will be described.
FIG. 7 is a diagram showing an example of information stored in the storage unit 37 according to this embodiment. As shown in FIG. 7, the storage unit 37 stores the target object, the work content, the degree of freedom restricted to the operator (the degree of freedom reinforced by the robot 2), and the degree of freedom operable by the operator. are stored in association with each other. Note that the example shown in FIG. 7 is just an example, and other information may also be associated and stored.

Further, since the coordinate system differs between the environment operated by the operator and the operating environment of the robot, for example, the operating environment of the operator and the operating environment of the robot may be calibrated when the robot 2 is activated.

When gripping, the robot remote control device 3 determines the gripping position based on the gripping force of the robot 2, the frictional force between the object and the gripping part, etc., and considering the error of the gripping position at the time of gripping. You may make it

[Modification of hand target of grasping position]
Next, an example of correction of the hand target of the gripping position with the assistance of the robot remote control device 3 will be described.
FIG. 8 is a diagram showing an example of correction of the hand target of the gripping position with the assistance of the robot remote control device 3. In FIG. In the example of FIG. 8, the shape of the target object g21 to be grasped is a substantially rectangular parallelepiped. When the robot remote control device 3 assists the operator's input g11 by restricting the degree of freedom so that it is easy to grasp, the robot remote control device 3 changes the hand target to a position that is easy to grasp (position g12 after correction). When the vertical direction of the target object is tilted with respect to the z-axis direction, for example, as shown in FIG. 5, the change of the hand target is corrected so that the angle of the operator's input in the z-axis direction is aligned with the vertical direction of the PET bottle. to generate a control command for the grip part 221 . More specifically, first, the control command generator 34 estimates a pattern for assisting each object. The estimation method is performed, for example, by matching with a database stored in the storage unit 37 or by machine learning. After estimating the auxiliary pattern, the control command generator 34 corrects the direction by performing vector calculation (cross product/inner product) between the coordinate system of the object and the hand coordinate system of the robot. In addition, the control command generation unit 34, for example, corrects the direction by vector calculation, and generates the command value by an end-to-end machine learning method that directly generates the correction command value from the sensor information. can be As a result, according to the present embodiment, the robot remote control device 3 estimates the target object and the work content, and corrects the hand target to a position that is easier to grip, so that the operator can easily give instructions.

The intention estimating unit 33 or the storage unit 37 inputs, for example, line-of-sight information, operator arm information, environmental sensor information, and detection results detected by the sensors of the robot 2, and learns the target object and operation details as teacher data. The intention estimation model may be provided. The intention estimation model may be provided on the cloud. The intention estimation unit 33 may input the acquired information into the intention estimation model to estimate the operator's intention. Alternatively, the intention estimation unit 33 may input the acquired information into the intention estimation model to estimate at least one of the target object and the work content.

Alternatively, the intention estimation unit 33 may estimate the intention of the operator by probabilistic inference using the acquired information. Alternatively, the intention estimation unit 33 may estimate at least one of the target object and the work content by probabilistic inference using the acquired information.

[Example of processing procedure of robot 2 and robot remote control device 3]
Next, a processing procedure example of the robot 2 and the robot remote control device 3 will be described.
FIG. 9 is a flow chart of an example of a processing procedure of the robot 2 and the robot remote control device 3 according to this embodiment.

(Step S<b>1 ) The information acquisition unit 31 acquires line-of-sight information from the HMD 5 and operator arm information from the controller 6 .

(Step S<b>2 ) The information acquisition unit 31 acquires environment sensor information from the environment sensor 7 .

(Step S3) The intention estimation unit 33 estimates the operator's intention including the object to be grasped and the work content based on the acquired line-of-sight information, operator arm information, and environment sensor information.

(Step S4) Based on the estimated intention of the operator and the information stored in the storage unit 37, the control command generation unit 34 generates the degree of freedom assisted by the robot remote control device 3, that is, the operator's operation instruction. determines the limits on the degrees of freedom of Note that the control command generation unit 34 generates a control command suitable for a portion of the degree of freedom of the operator's motion, thereby reducing the degree of freedom of the operator's motion and generating the control command.

(Step S5) The control command generator 34 corrects the target hand position to be gripped based on the arm movement information included in the obtained operator arm information and the degree of freedom to be restricted determined in step S4. .

(Step S6) Based on the operator arm information acquired by the information acquisition unit 31, the result estimated by the intention estimation unit 33, and the gripping position corrected by the control command generation unit 34, the robot state image creation unit 35 , create a robot state image to be displayed on the HMD 5 .

(Step S7) The control command generator 34 generates a control command based on the corrected target hand position to be gripped.

(Step S8) Based on the control command generated by the control command generating unit 34, the control unit 21 controls the driving unit 22 to drive the grasping unit of the robot 2 and the like. After the processing, the control unit 21 returns to the processing of step S1.

The processing procedure shown in FIG. 9 is an example, and the robot remote control device 3 may process the above-described processing in parallel.

Also, the image displayed on the image display unit 51 of the HMD 5 is an image such as that shown in FIG. 8, for example. Note that the robot state image may not include the image of the operator's input g11. In addition, the robot state image creating unit 35, for example, presents an assist pattern with characters or displays a rectangle, an arrow, or the like on the HMD 5 to limit the degree of freedom and present the details of assistance by the robot to the operator. may As a result, when the operator gives a work instruction while looking at the robot state image displayed on the HMD 5, for example, when opening the lid of a PET bottle, the operator can operate the robot without worrying about aligning the gripping part in the vertical direction of the PET bottle. You can concentrate on the work instructions for opening the lid.

As described above, in this embodiment, the robot remote control device 3 determines the work content and the target object through sensor information installed in the robot 2, the environment, and the operator. By substituting the control target value generation of the part, the degree of freedom to be controlled by the operator is limited. That is, in the present embodiment, the degree of freedom to be controlled by the operator is reduced by automatically generating appropriate control commands for some of the degrees of freedom.

As a result, according to the present embodiment, it is possible to reduce the burden on the operator, diversify the work, and improve the efficiency of the remote control robot at the same time.

In addition, the example of restriction|limiting of the degree of freedom mentioned above is an example, and it is not restricted to this. A limited degree of freedom does not mean that the operator cannot completely control the degree of freedom, but also includes a limited operable area.
For example, the robot remote control device 3 does not limit the degree of freedom when the distance from the target object is outside the predetermined range, but limits the degree of freedom when the distance from the target object is within the predetermined range. good too. The predetermined range is, for example, a range of ±1 m of the translational x-axis.

In addition, in the above-described example, an example of grasping the target object by remote control has been described, but the present invention is not limited to this. The robot remote control device 3 limits the degree of freedom according to the work content, even if the work content is different.
Moreover, when there are a plurality of work contents, the robot remote control device 3 may change the restriction of the degree of freedom each time the work contents change. For example, when a plurality of objects are placed on a table, and one PET bottle is grasped from among them and then the lid of the PET bottle is opened, the first degree of freedom restriction in the gripping step is to open the lid. The robot remote control device 3 sets a limit of the second degree of freedom when allowing the robot to move.

In addition, in the case of a work content that uses both grips, the robot remote control device 3 limits the degree of freedom and corrects the hand target for each grip.

Further, the above-described robot 2 may be, for example, a bipedal walking robot, a stationary reception robot, or a working robot.

Also, in the above-described example, an example in which the robot 2 is caused to grip by remote control has been described, but the present invention is not limited to this.

Also, in the above example, an example in which the operator wears the HMD 5 has been described, but the present invention is not limited to this. Detection of line-of-sight information and provision of the robot state image to the operator may be performed by, for example, a combination of a sensor and an image display device.

In addition, a program for realizing all or part of the functions of the robot 2 and all or part of the functions of the robot remote control device 3 in the present invention is recorded on a computer-readable recording medium. All or part of the processing performed by the robot 2 and all or part of the processing performed by the robot remote control device 3 may be performed by loading the recorded program into the computer system and executing the program. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices. In addition, the "computer system" shall include a system built on a local network, a system built on the cloud, and the like. The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. In addition, "computer-readable recording medium" means a volatile memory (RAM) inside a computer system that acts as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. , includes those that hold the program for a certain period of time.

Further, the above program may be transmitted from a computer system storing this program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in a transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the program may be for realizing part of the functions described above. Further, it may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

As described above, the mode for carrying out the present invention has been described using the embodiments, but the present invention is not limited to such embodiments at all, and various modifications and replacements can be made without departing from the scope of the present invention. can be added.

DESCRIPTION OF SYMBOLS 1... Robot remote control system, 2... Robot, 3... Robot remote control device, 5... HMD, 6... Controller, 7... Environmental sensor, 21... Control part, 22... Drive part, 23... Sound pickup part, 25 Memory unit 26 Power supply 27 Sensor 221 Grasping unit 31 Information acquiring unit 33 Intention estimating unit 34 Control command generating unit 35 Robot state image creating unit 36 Transmitting unit 37 ... storage section 51 ... image display section 52 ... line-of-sight detection section 54 ... control section 55 ... communication section 61 ... sensor 62 ... control section 63 ... communication section 64 ... feedback means 71 ... photographing device 72...Sensor, 73...Communication unit

Claims (9)

In a robot remote operation that recognizes the movement of an operator and transmits the movement of the operator to a robot to operate the robot,
Based on a robot environment sensor value obtained by the robot or an environment sensor installed in the surrounding environment of the robot, and an operator sensor value obtained by an operator sensor, which is the movement of the operator, the operator an intention estimation unit that estimates the motion of
By generating an appropriate control command for a part of the degrees of freedom of the operator's motion based on the estimated motion of the operator, the degree of freedom of the motion of the operator is reduced and the control command is generated. a control command generator that generates
A robot remote control device comprising: The control command generation unit limits the degree of freedom to be controlled by the operator and the controllable range, and assists the movement with respect to the limited degree of freedom in the operation instruction to the robot by the operator.
The robot remote control device according to claim 1. The control command generator is
When the distance between the grip portion provided in the robot and the target object to be operated by the operator is outside a predetermined range, without reducing the degree of freedom of the operator's motion among the operator's motions,
When the distance between the grip portion provided in the robot and the target object to be operated by the operator is within a predetermined range, reducing the degree of freedom of the operator's motion among the operator's motions.
the robot environment sensor values have captured image information and depth information;
The robot remote control device according to claim 1 or 2. The intention estimation unit
inputting the robot environment sensor value and the operator sensor value into a learned intention estimation model to estimate the operator's motion;
The robot remote control device according to any one of claims 1 to 3. The operator sensor value is at least one of the operator's line of sight information and the operator's arm information, which is information about the posture and position of the operator's arm,
The robot remote control device according to any one of claims 1 to 4. the robot environment sensor values have captured image information and depth information;
The robot remote control device according to any one of claims 1 to 5. In a robot remote operation that recognizes the movement of an operator and transmits the movement of the operator to a robot to operate the robot,
The robot remote control device according to any one of claims 1 to 6;
a gripping part that grips an object;
an environment sensor installed in the robot or in the surrounding environment of the robot and detecting a robot environment sensor value;
an operator sensor that detects the movement of the operator as an operator sensor value;
A robot remote control system. In a robot remote operation that recognizes the movement of an operator and transmits the movement of the operator to a robot to operate the robot,
The intention estimating unit obtains a robot environment sensor value obtained by the robot or an environment sensor installed in the surrounding environment of the robot, and an operator sensor value obtained by an operator sensor, which is the movement of the operator. estimating the motion of the operator based on
A control command generation unit generates a control command appropriate for a degree of freedom of a part of the operator's motion based on the estimated operator's motion, thereby reducing the operator's freedom of motion. generate control commands with reduced degrees,
Robot remote control method. In a robot remote operation that recognizes the movement of an operator and transmits the movement of the operator to a robot to operate the robot,
to the computer,
Based on a robot environment sensor value obtained by the robot or an environment sensor installed in the surrounding environment of the robot, and an operator sensor value obtained by an operator sensor, which is the movement of the operator, the operator to estimate the motion of
By generating an appropriate control command for a part of the degrees of freedom of the operator's motion based on the estimated motion of the operator, the degree of freedom of the motion of the operator is reduced and the control command is generated. to generate
program.

Images