JP2022157123A - Robot remote operation control device, robot remote operation control system, robot remote operation control method and program - Google Patents

Abstract

To provide a robot remote operation control device, a robot remote operation control system, a robot remote operation control method and a program which can estimate what kind of action an operator is doing from information obtained by a sensor mounted on the body of the operator.SOLUTION: A robot remote operation control device, which performs robot remote operation control by which motion of an operator is recognized and the motion of the operator is transmitted to a robot to operate the robot, comprises: an information obtaining part that obtains an operator sensor value which is information showing sensed motion of the operator; and an intention estimating part that estimates motion of the operator which is motion instructed to the robot by using a learnt model from the operator sensor value.SELECTED DRAWING: Figure 2

Description

The present invention relates to a robot remote operation control device, a robot remote operation control system, a robot remote operation control method, and a program.

A control device has been proposed that allows a user to assist the operation of a robot. As such a control device, for example, a first information acquisition unit that acquires first user posture information indicating the posture of a first user who operates the robot, and a a second information acquiring unit for acquiring pre-change posture information indicating a pre-change posture which is the posture of the robot; A control device has been proposed that includes a determination unit that determines a target posture different from the posture of the first user as the posture of the robot based on the first user posture information acquired by the information acquisition unit (see Patent Document 1). ). The system described in Patent Literature 1 changes the posture of the robot to a posture that corresponds to the posture detected by the device worn by the operator.

In such a system, when an operator manipulates an object by remote control, it is possible to determine what the operator does from sensors attached to the operator's body (sensors that acquire line of sight, head rotation, hand position and angle, etc.). It is necessary to estimate the work content of remote instructions to the robot by performing such actions.

Patent No. 6476358

However, in the prior art, when an operator manipulates an object by remote control, the sensor attached to the operator's body (for example, a sensor that acquires the line of sight, head rotation, hand position and angle, etc.) It was not possible to directly estimate what the operator was doing from the sensor value information for each time.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and provides a robot remote controller that can estimate what actions the operator is taking from information obtained from sensors attached to the operator's body. An object of the present invention is to provide an operation control device, a robot remote operation control system, a robot remote operation control method, and a program.

(1) In order to achieve the above objects, a robot remote operation control device according to an aspect of the present invention recognizes the movement of an operator and transmits the operator's movement to a robot to operate the robot. an information acquisition unit that acquires an operator sensor value that is information indicating the detected movement of the operator; and the operator that is an operation instruction to the robot using a model learned from the operator sensor value. and an intention estimating unit that estimates the motion of the

(2) In the robot remote control device according to the aspect of the present invention, the information acquisition unit acquires an environment sensor value obtained by an environment sensor installed in the robot or in the surrounding environment of the robot, The intention estimating unit uses a model that has been learned from the environment sensor values and the operator sensor values to determine whether the operator is trying to align the gripped object gripped by the robot by operating the robot. It is also possible to estimate whether or not, and to generate a control command value for driving the robot based on the estimated result.

(3) In the robot remote operation control device according to the aspect of the present invention, the information acquisition unit acquires an environment sensor value obtained by an environment sensor installed in the robot or in the surrounding environment of the robot, The intention estimating unit uses the environment sensor value and the operator sensor value to calculate the likelihood that the operator is going to align the gripped object in what kind of posture, thereby supporting alignment. may be performed.

(4) Further, in the robot remote operation control device according to the aspect of the present invention, the intention estimating unit estimates from the operator sensor values, and after the processing for stabilizing the values, the operator is expected to hold the gripped object. , the likelihood may be calculated.

(5) Further, in the robot remote operation control device according to the aspect of the present invention, the information acquisition unit acquires an environment sensor value obtained by an environment sensor installed in the robot or in the surrounding environment of the robot, The intention estimating unit may input the environment sensor value into the learned model to estimate the operator's motion, which is the motion instruction to the robot.

(6) Further, in the robot remote operation control device according to the aspect of the present invention, the intention estimating unit extracts a feature amount from the operator sensor value, and issues an operation instruction to the robot based on the extracted feature amount. It may be estimated by classifying the motion of a given operator.

(7) In order to achieve the above objects, a robot remote operation control system according to an aspect of the present invention includes the operator motion estimation during remote operation according to any one of (1) to (6). A device, a robot remote control for recognizing a movement of an operator and transmitting the movement of the operator to a robot to operate the robot, a gripping part for gripping an object, and a gripping part installed in the robot or in the surrounding environment of the robot. , an environment sensor for detecting a robot environment sensor value, an operator sensor for detecting the movement of the operator as the operator sensor value, and an image display device for displaying an image of the robot environment in the field of view of the operator. Prepare.

(8) In order to achieve the above objects, a robot remote control method according to an aspect of the present invention is a robot remote control method for recognizing a motion of an operator and transmitting the motion of the operator to a robot to operate the robot. an information acquisition unit that acquires an operator sensor value that is information indicating the detected movement of the operator; The operator's motion, which is an instruction, is estimated.

(9) In order to achieve the above object, a program according to an aspect of the present invention provides, in remote control of a robot for recognizing a motion of an operator and transmitting the motion of the operator to a robot to operate the robot, a computer, An operator sensor value for detecting the movement of the operator is acquired, and a learned model is used from the operator sensor value to estimate the operator's movement, which is a movement instruction to the robot.

According to (1) to (9), it is possible to estimate what the operator is doing from the information obtained from the sensor attached to the operator's body.
According to (2), the time required for alignment can be reduced by detecting the alignment intention of the operator and proactively supporting alignment.
According to (3) and (4), by probabilistically calculating at what angle the operator places the gripped object, Place support is executed to reduce the time and stress required for the operator to align. lead to reduction.
According to (4), the influence of camera shake can be reduced.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure which shows the outline|summary of the robot remote operation control system which concerns on embodiment, and the outline|summary of work. 1 is a block diagram showing a configuration example of a robot remote control system according to an embodiment; FIG. It is a figure which shows the example of the kind of action classified by the classification|category part which concerns on 1st Embodiment. FIG. 5 is a diagram showing an example of a classification method of a classification unit according to the first embodiment; 4 is a flow chart of a processing procedure of the robot remote control device according to the first embodiment; FIG. 11 is a diagram for explaining support for alignment of a gripped object according to the second embodiment; 9 is a flow chart of a processing procedure of a robot remote control device according to a second embodiment; FIG. 11 is a diagram illustrating an example of a classification method of a classification unit according to the third embodiment; 9 is a flow chart of a processing procedure of a robot remote control device according to a second embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, in the drawings used for the following description, the scale of each member is appropriately changed so that each member has a recognizable size.

[Overview]
First, an outline of work and processing performed by the system for estimating the operator's motion during remote operation will be described.
FIG. 1 is a diagram showing an outline of a robot remote control system 1 and an outline of work according to this embodiment. As shown in FIG. 1, the operator Us is wearing an HMD (head mounted display) 5 and a controller 6, for example. An environment sensor 7a and an environment sensor 7b are installed in the work space. Note that the environment sensor 7 may be attached to the robot 2 . The robot 2 also includes a gripper 222 (222a, 222b). The environment sensors 7 (7a, 7b) include, for example, an RGB camera and a depth sensor as described later. The operator Us remotely operates the robot 2 by moving the hand or fingers wearing the controller 6 while watching the image displayed on the HMD 5 .

In this embodiment, for example, a machine learning technique is used to estimate what kind of action the operator is taking from a plurality of sensor values detected by a plurality of sensors worn by the operator. Further, in this embodiment, alignment support is realized by calculating how the operator is likely to align (behavior) with respect to the object. In addition, in this embodiment, for example, using a machine learning method, from a plurality of sensor values and the state of the operator motion estimation device system during remote operation, before the operator aligns with the object, let's perform alignment. Estimate whether or not

[Configuration example of robot remote control system]
Next, a configuration example of the robot remote control system 1 will be described.
FIG. 2 is a block diagram showing a configuration example of the robot remote control system 1 according to this embodiment. As shown in FIG. 2 , the robot remote control system 1 includes a robot 2 , a robot remote control device 3 , an HMD 5 (image display device), a controller 6 and an environment sensor 7 .

The robot 2 includes, for example, a control unit 21, a drive unit 22, a sound pickup unit 23, a storage unit 25, a power supply 26, and a sensor 27.
The robot remote control device 3 includes, for example, an information acquisition unit 31, an intention estimation unit 33, a control command generation unit 34, an image generation unit 35, a transmission unit 36, and a storage unit 37.

The intention estimation unit 33 includes a motion estimation unit 331 , an alignment support unit 332 , and an alignment availability prediction unit 333 .
The motion estimation unit 331 includes a feature quantity extraction unit 3311 and a classification unit 3312 .

HMD5 is provided with the image display part 51, the line-of-sight detection part 52 (operator sensor), the sensor 53 (operator sensor), the control part 54, and the communication part 55, for example. Note that the HMD 5 may include a sensor that detects, for example, the movement of the operator's line of sight.

The controller 6 includes, for example, a sensor 61 (operator sensor), a control section 62, a communication section 63, and feedback means 64.

The environment sensor 7 includes, for example, an imaging device 71 , a sensor 72 , an object position detection section 73 and a communication section 74 .

The robot remote control device 3 and the HMD 5 are connected via a wireless or wired network, for example. The robot remote control device 3 and the controller 6 are connected via a wireless or wired network, for example. The robot remote control device 3 and the environment sensor 7 are connected via a wireless or wired network, for example. The robot remote control device 3 and the robot 2 are connected via a wireless or wired network, for example. Note that the robot remote control device 3 and the HMD 5 may be directly connected without going through a network. The robot remote control device 3 and the controller 6 may be directly connected without going through a network. The robot 2 and the environment sensor 7 may be directly connected without going through a network. The robot remote control device 3 and the robot 2 may be directly connected without going through a network.

[Function example of robot remote control system]
Next, an example of functions of the robot remote control system will be described with reference to FIG.
The HMD 5 displays the robot state image received from the robot remote control device 3 . The HMD 5 detects the movement of the operator's line of sight, the movement (rotation, tilt) of the operator's head, and the like, and transmits the detected operator sensor values to the robot remote control device 3 .

The image display unit 51 displays the state image of the robot received from the robot remote control device 3 under the control of the control unit 54 .

The line-of-sight detection unit 52 detects the line-of-sight of the operator and outputs the detected operator sensor value to the control unit 54 . Note that the line-of-sight information is a line-of-sight vector.

The sensor 53 detects the tilt and rotation of the operator's head, and outputs the detected operator sensor values to the controller 54 .

The control unit 54 transmits operator sensor values detected by the line-of-sight detection unit 52 and the sensor 53 to the robot remote control device 3 via the communication unit 55 . The control unit 54 causes the image display unit 51 to display the robot state image transmitted by the robot remote control device 3 .

The communication unit 55 receives the robot state image transmitted by the robot remote control device 3 and outputs the received robot state image to the control unit 54 . The communication unit 55 transmits the operator sensor values to the robot remote control device 3 under the control of the control unit 54 .

The controller 6 is, for example, a tactile data glove worn on the operator's hand. The controller 6 uses the sensor 61 to detect the orientation, motion of each finger, and motion of the hand, and transmits detected hand motion information (operator sensor values) to the robot remote control device 3 .

The sensor 61 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic force sensor, or the like. Note that the sensor 61, which includes a plurality of sensors, tracks the movement of each finger using, for example, two sensors. The sensor 61 detects the motion of each finger and the motion of the hand, and outputs the detected hand motion information (operator sensor value) to the control unit 62 .

The control unit 62 transmits hand motion information detected by the sensor 61 to the robot remote control device 3 via the communication unit 63 . The controller 62 controls the feedback means 64 based on the feedback information.

The communication unit 63 transmits operator action information to the robot remote control device 3 under the control of the control unit 62 . The communication unit 63 acquires the feedback information transmitted by the robot remote control device 3 and outputs the acquired feedback information to the control unit 62 .

The feedback means 64 feeds back feedback information to the operator under the control of the control section 62 . According to the feedback information, the feedback means 64 includes, for example, means for applying vibration (not shown), means for applying air pressure (not shown), and means for restraining hand movement (not shown) attached to the grip part 222 of the robot 2. (not shown), means for feeling temperature (not shown), means for feeling hardness or softness (not shown), or the like is used to feed back sensations to the operator.

The environment sensor 7 is installed, for example, at a position where the work of the robot 2 can be photographed and detected. The environment sensor 7 may be provided in the robot 2 or may be attached to the robot 2 . Alternatively, the number of environmental sensors 7 may be two or more. The environment sensor 7 detects object position information based on the captured image and the detection result detected by the sensor, and transmits the detected object position information (environment sensor value) to the robot remote control device 3 .

The imaging device 71 is, for example, an RGB camera. The photographing device 71 outputs the photographed image to the object position detection section 73 . In addition, in the environment sensor 7, the positional relationship between the imaging device 71 and the sensor 72 is known.

Sensor 72 is, for example, a depth sensor. The sensor 72 outputs the detection result to the object position detection section 73 . Note that the imaging device 71 and the sensor 72 may be distance sensors.

The object position detection unit 73 detects the three-dimensional position, size, shape, etc. of the target object in the photographed image by a well-known method based on the photographed image and the detection result detected by the sensor. The object position detection unit 73 refers to the pattern matching model or the like stored in the object position detection unit 73, and performs image processing (edge detection, binarization processing, feature amount extraction, image enhancement processing, image extraction, pattern matching processing, etc.) to estimate the position of the object. Note that when a plurality of objects are detected from the captured image, the object position detection unit 73 detects the position of each object. The object position detection unit 73 transmits the detected object position information (environment sensor value) to the robot remote control device 3 via the communication unit 74 .

The communication unit 74 transmits object position information to the robot remote control device 3 . The data transmitted by the environment sensor 7 may be, for example, a point cloud having position information.

The behavior of the robot 2 is controlled according to the control of the control unit 21 when it is not remotely operated. When the robot 2 is remotely controlled, its action is controlled according to the action command generated by the robot remote control device 3 .

The control unit 21 controls the driving unit 22 based on control commands output from the robot remote control device 3 . Note that the control unit 21 may perform speech recognition processing (speech segment detection, sound source separation, sound source localization, noise suppression, sound source identification, etc.) on the acoustic signal collected by the sound collection unit 23 . The control unit 21 generates feedback information and transmits the generated feedback information to the controller 6 via the robot remote control device 3 .

The driving section 22 drives each section of the robot 2 (grasping section 222 , arms, fingers, feet, head, torso, waist, etc.) under the control of the control section 21 . The drive unit 22 includes, for example, actuators, gears, artificial muscles, and the like.

The sound pickup unit 23 is, for example, a microphone array including a plurality of microphones. The sound pickup unit 23 outputs the collected sound signal to the control unit 21 . The sound pickup unit 23 may have a speech recognition processing function. In this case, the sound pickup unit 23 outputs the speech recognition result to the control unit 21 .

The storage unit 25 stores, for example, programs and threshold values used for control by the control unit 21, and temporarily stores voice recognition results, image processing results, control commands, and the like. Note that the storage unit 37 may also serve as the storage unit 25 . Alternatively, the storage unit 37 may also serve as the storage unit 25 .

A power supply 26 supplies power to each part of the robot 2 . Power source 26 may comprise, for example, a rechargeable battery or charging circuit.

The sensor 27 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic force sensor, each joint encoder, or the like. The sensor 27 is attached to each joint, head, etc. of the robot 2 . The sensor 27 outputs the detected result to the control unit 21 , the intention estimation unit 33 , the control command generation unit 34 and the image generation unit 35 .

Based on the operator sensor values detected by the HMD 5 and the operator sensor values detected by the controller 6, the robot remote control device 3 estimates the behavior of the operator to remotely control the robot 2. , to generate control commands for the robot 2 .

The information acquisition unit 31 acquires operator sensor values from the HMD 5 , acquires operator sensor values from the controller 6 , and acquires environment sensor values from the environment sensor 7 . The information acquisition unit 31 outputs the acquired operator sensor values and environment sensor values to the intention estimation unit 33 .

A feature quantity extraction unit 3311 extracts a feature quantity based on the obtained operator sensor value and environment sensor value.

The intention estimation unit 33 estimates the object to be worked on and its position based on the acquired environmental sensor values.

The classification unit 3312 classifies the behavior of the operator based on the extracted feature amount. In addition, the behavior to be classified will be described later.

The alignment support unit 332 implements alignment support by calculating how likely the operator aligns the gripped object with respect to the alignment reference object.

The alignment feasibility prediction unit 333 uses the information obtained from the operator sensor value to determine in advance whether or not the operator is about to align when the operator holds the object in his/her hand and reaches an arbitrary object. Predict. Note that the alignment feasibility prediction unit 333 may make the prediction at a time before the object is aligned, or at any time before the release of the object is completed.

The control command generator 34 generates, for example, a control command for gripping an object based on the result estimated by the intention estimator 33, the detection result detected by the sensor 27, and the environment sensor value detected by the environment sensor 7. . The control command generator 34 outputs the generated control command information to the controller 21 .

The image creation unit 35 creates a robot state image to be displayed on the HMD 5 based on the control command information generated by the control command generation unit 34 .

The transmission unit 36 transmits the robot state image created by the image creation unit 35 to the HMD 5 . The transmission unit 36 acquires feedback information output by the robot 2 and transmits the acquired feedback information to the controller 6 .

The storage unit 37 stores predetermined values and learned models used by the intention estimating unit 33 . The storage unit 37 stores the positional relationship between the imaging device 71 and the sensor 72 of the environment sensor 7 . The storage unit 37 stores programs used for controlling the robot remote control device 3 . Note that the program may be on the cloud or network.

<First embodiment>
In this embodiment, what kind of action the operator is taking is estimated from a plurality of sensor values detected by a plurality of sensors worn by the operator.

[Examples of types of behavior classified]
First, examples of behavior types classified by the classification unit 3312 will be described.
FIG. 3 is a diagram showing examples of types of actions classified by the classification unit 3312 according to this embodiment. As shown in FIG. 3, the classification unit 3312 classifies, for example, into four classes (first class to fourth class). The classified operator actions are, for example, Reach, Grasp, Move, and Release. Note that the reaching motion is a motion of bending the arm toward a target point. A grasp is the action of grasping an object. A move action is an action to move or carry an object. A release operation is an operation of releasing or placing a gripped object. Note that the classification unit 3312 may also classify other actions such as pinching.

[Classification example]
Next, an example of the classification method of the classification unit 3312 will be described.
FIG. 4 is a diagram showing an example of a classification method of the classification unit 3312 according to this embodiment. As shown in FIG. 4, the classification unit 3312 inputs at least the operator sensor values acquired from the HMD 5 and the controller 6 to the learned model, and obtains the output of the classification result of the operator's actions. In this case, the feature quantity extraction unit 3311 does not have to extract the feature quantity from the operator sensor value. Note that the learned model is a model created by learning with a machine learning method (for example, a neural network, etc.) using operator sensor values as input and teacher data of classification results of the operator's actions. Also, the learned model is stored in the classification unit 3312 or the storage unit 37 . It should be noted that the machine learning method may be other than the neural network (for example, a decision tree-based method, etc.).

In the example shown in FIG. 4, the classification unit 3312 inputs the operator sensor values to the learned model 3321 and obtains the output of the classification result of the operator's behavior. However, the present invention is not limited to this. For example, the feature amount extraction unit 3311 may extract feature amounts from the operator sensor values, and the classification unit 3312 may classify the operator's actions based on the extracted feature amounts.

Also, in the above example, an example in which operator sensor values are used in learning and estimation has been described, but the present invention is not limited to this. Environmental sensor values may also be used as inputs for learning and estimation.

[Processing procedure]
Next, the processing procedure of the robot remote control device 3 will be described.
FIG. 5 is a flow chart of the processing procedure of the robot remote control device 3 according to this embodiment.

(Step S<b>1 ) The information acquisition unit 31 acquires an operator sensor value (line-of-sight vector) from the HMD 5 , an operator sensor value from the controller 6 , and an environment sensor value from the environment sensor 7 .

(Step S2) The feature amount extraction unit 3311 extracts feature amounts from the operator sensor values.

(Step S3) The classification unit 3312 classifies the behavior of the operator based on the extracted feature amount.

(Step S4) The control command generator 34 generates a control command based on the estimated operator's behavior.

(Step S5) Based on the control command generated by the control command generating unit 34, the control unit 21 controls the driving unit 22 to drive the grasping unit of the robot 2 and the like. After the processing, the control unit 21 returns to the processing of step S1.

As described above, in the present embodiment, what kind of action the operator is taking is classified, for example, by a learned model or the like, based on a plurality of sensor values.

Thus, according to the present embodiment, it is possible to determine what action (Reach/Grasp/Move/Release) the operator is performing based on the information obtained from the sensors (HMD 5, controller 6) attached to the operator's body. can be estimated. As a result, according to this embodiment, it is possible to proactively support alignment. Thus, according to this embodiment, the time required for alignment can be reduced.
Note that it is also possible to utilize the action estimation result for alignment. For example, when it is predicted that alignment will be performed in the alignment possibility estimation and the result of the action estimation is Move, the robot remote control device 3 performs control to support alignment during the move (e.g., control of a specific axis of the robot arm). fixing the movement) may be implemented.

<Second embodiment>
In remote control, it is difficult to align (hereinafter referred to as alignment) a gripped object next to an arbitrary object (hereinafter referred to as alignment reference object). Furthermore, from the information obtained from sensors attached to the operator's body (sensors that acquire line of sight, head rotation, hand position and angle, etc.), it is possible to determine how the operator aligns the gripped object with respect to the object. is difficult to estimate directly.

Therefore, in the present embodiment, alignment support is realized by calculating the likelihood (probability) of aligning the gripped object gripped by the operator with respect to the alignment reference object.

FIG. 6 is a diagram for explaining support for alignment of a grasped object according to this embodiment. The alignment reference object KObj is an object already placed on the table Tb. The first placement candidate Obja is an example in which the longitudinal direction of the grasped object Obj is at the same angle as the alignment reference object KObj (angle of 0 degrees). The second placement candidate Objb is an example in which the longitudinal direction of the grasped object Obj is 90 degrees with respect to the alignment reference object KObj. Note that the alignment reference object KObj may be a virtual object.

It should be noted that the robot remote control device 3 generates and holds these placement method candidates in advance. Further, the placement candidate shown in FIG. 6 is an example, and the present invention is not limited to this.
The robot remote control device 3 calculates the likelihood (probability) of how the gripped object Obj currently gripped is placed with respect to this alignment reference object KObj. Then, the robot remote control device 3 supports alignment according to the calculation result.

[Processing procedure]
Next, the processing procedure of the robot remote control device 3 will be described.
FIG. 7 is a flow chart of the processing procedure of the robot remote control device 3 according to this embodiment.

(Step S<b>101 ) The information acquisition unit 31 acquires an operator sensor value (line-of-sight vector) from the HMD 5 , an operator sensor value from the controller 6 , and an environment sensor value from the environment sensor 7 .

(Step S102) The alignment support section 332 detects the position and placement of the alignment reference object KObj based on the acquired environmental sensor values.

(Step S<b>103 ) The alignment support unit 332 aligns the gripped object gripped by the robot 2 by the operator's remote control based on the obtained operator sensor value, the environment sensor value, and the detection result detected by the sensor 27 . Detect condition.

(Step S104) The alignment support unit 332 generates candidates for how to place the gripped object.

(Step S105) The alignment support unit 332 calculates the likelihood (probability) of each placement candidate from the relative relationship between all the placement candidates and the state of the gripped object. The likelihood (probability) l is calculated by the following equation (1) for the relative angle d (−180°≦d≦180°) of the grasped object with respect to any placement candidate, for example.

l=exp(−k·|d|) (1)

In equation (1), k is a constant that adjusts how large the likelihood (probability) is to be calculated with respect to the difference in angle (magnitude of relative angle). (1), the likelihood (probability) increases as the relative angle to the orientation candidate of the gripped object decreases.

Note that the alignment support unit 332 calculates the likelihood (probability) based on, for example, the transition probability from the current gripped object to each placement candidate. Note that the alignment support unit 332 performs processing (for example, a Kalman filter) for stabilizing the values by estimating from the acquired sensor values in order to correct the shake of the operator's hand, and then calculates the likelihood (probability). You may make it calculate.

The posture of a grasped object is greatly affected by the posture of the hand. By estimating the posture of the grasped object using, for example, a Kalman filter, it is possible to output a posture that is more stable (less affected by camera shake, etc.) than the actual observed value. In this embodiment, the value may be used to generate the control command.

(Step S106) The control command generation unit 34 generates a control command so as to support the placement method of the placement method candidate with the highest calculated likelihood (probability).

(Step S<b>107 ) Based on the control command generated by the control command generation unit 34 , the control unit 21 controls the drive unit 22 to drive the grasping unit of the robot 2 and the like.

(Step S108) The control unit 21 determines whether or not the gripped object is placed on, for example, a table, according to the operator's remote control. If the control unit 21 determines that the gripped object has been placed, for example, on the table (step S108; YES), the process ends. When the control unit 21 determines that the gripped object is not placed on, for example, the table (step S108; NO), the process returns to step S101.

As described above, in the present embodiment, the likelihood (probability) of how the operator aligns the gripped object with respect to the alignment reference object is calculated to realize alignment support. Note that the processing procedure described above is an example, and is not limited to this. For example, the processing order of steps S102 to S104 does not have to be this order, and the processing order can be changed. As for these processes, for example, the process of step S104 may be performed after the process of step S102, and then the process of step S103 may be performed. As for these processes, for example, the process of step S102 may be performed after the process of step S103, and then step S104 may be performed.

Thus, according to the present embodiment, by probabilistically calculating at what angle the operator releases (or places) the gripped object, Place support is executed, and the operator adjusts the alignment. It can save you time and stress.

Note that this embodiment can also be applied to the first embodiment described above.

<Third Embodiment>
A sensor attached to the operator's body when a grasped object is approaching an arbitrary object (existing on a desk, etc.), such as when picking and placing an object by remote control (when grasping and reaching the object). From the information (at each time) obtained from (sensors that acquire line of sight, head rotation, hand position and angle, etc.), it is possible to determine whether the operator is trying to align the gripped object with an arbitrary object. The system cannot directly estimate whether

For this reason, in the present embodiment, when releasing an object gripped by the operator based on a plurality of sensor values and the state of the robot remote operation control system, before starting the releasing process, before alignment is performed with respect to the gripped object. First, it estimates whether or not alignment is to be performed.

[Classification example]
Next, an example of the classification method of the classification unit 3312 will be described.
FIG. 8 is a diagram showing an example of the classification method of the classification unit 3312 according to this embodiment. As shown in FIG. 8, the classification unit 3312 inputs at least the operator sensor values acquired from the HMD 5 and the controller 6 to the learned model 3321A, and obtains the output of the classification result as to whether or not to perform alignment. In this case, the feature quantity extraction unit 3311 of the intention estimation unit 33 does not have to extract the feature quantity from the operator sensor value. Note that the learned model 3321A is a model generated by learning by a machine learning method (for example, a neural network, etc.) using operator sensor values as input and teacher data indicating whether or not to perform alignment. Also, the learned model 3321A is stored in the classification unit 3312 or the storage unit 37 . It should be noted that the machine learning method may be other than the neural network (for example, a decision tree-based method, etc.).

In the example shown in FIG. 8, the classifying unit 3312 inputs the operator sensor values to the learned model 3321A and outputs the classification result as to whether or not to perform alignment. do not have. For example, the feature quantity extraction unit 3311 may extract a feature quantity from the operator sensor values, and the classification unit 3312 may classify whether or not to perform alignment based on the extracted feature quantity.

Also, in the above example, an example in which operator sensor values are used in learning and estimation has been described, but the present invention is not limited to this. Environmental sensor values may also be used as inputs for learning and estimation.

[Processing procedure]
Next, the processing procedure of the robot remote control device 3 will be described.
FIG. 9 is a flow chart of the processing procedure of the robot remote control device 3 according to this embodiment.

(Step S<b>201 ) The information acquisition unit 31 acquires an operator sensor value (line-of-sight vector) from the HMD 5 , an operator sensor value from the controller 6 , and an environment sensor value from the environment sensor 7 .

(Step S202) Based on the obtained operator sensor value, the environment sensor value, and the detection result detected by the sensor 27, the alignment availability prediction unit 333 determines whether the gripped object gripped by the robot 2 is remotely controlled by the operator. to detect the state of

(Step S203) The alignment feasibility prediction unit 333, for example, inputs the acquired operator sensor values into a learned learning model to perform pre-prediction of alignment (whether or not alignment is to be performed).

(Step S204) The control command generator 34 generates a control command based on the prediction result as to whether alignment is to be performed.

(Step S205) Based on the control command generated by the control command generating unit 34, the control unit 21 controls the driving unit 22 to drive the grasping unit of the robot 2 and the like.

(Step S206) The control unit 21 determines whether or not the gripped object is placed on, for example, a table according to the operator's remote control. If the control unit 21 determines that the gripped object has been placed, for example, on the table (step S206; YES), the process ends. When the control unit 21 determines that the gripped object is not placed on, for example, the table (step S206; NO), the process returns to step S201.

As described above, in the present embodiment, information obtained from the sensor attached to the body of the operator is used to perform alignment when the operator is holding an object in his hand and reaching an arbitrary object. I tried to predict in advance whether or not there would be. Note that the alignment feasibility prediction unit 333 may make the prediction at a time before the object is aligned, or at any time before the release of the object is completed.

As a result, according to the present embodiment, the robot remote control device 3 detects the operator's alignment intention and proactively supports alignment, thereby reducing the time required for alignment.

It should be noted that this embodiment can also be applied to at least one of the above-described first embodiment and second embodiment.

Note that the intention estimation unit 33 may predict in advance the future trajectory of the hand intended by the operator based on the operator state information and the state information of the robot 2 .

In addition, since the coordinate system differs between the environment operated by the operator and the operating environment of the robot, the robot remote control device 3 calibrates the operating environment of the operator and the operating environment of the robot when the robot 2 is activated, for example. may

Further, when gripping, the robot remote control device 3 corrects an error in the gripping position at the time of gripping based on the gripping force of the robot 2, the frictional force between the object and the gripping part 222, etc., and adjusts the gripping position. You may decide.

Further, the above-described robot 2 may be, for example, a bipedal walking robot, a stationary reception robot, or a working robot.

Also, in the above-described example, an example in which the robot 2 is caused to grip by remote control has been described, but the present invention is not limited to this.

Also, in the above example, an example in which the operator wears the HMD 5 has been described, but the present invention is not limited to this. Detection of line-of-sight information and provision of the robot state image to the operator may be performed by, for example, a combination of a sensor and an image display device.

A program for realizing all or part of the functions of the robot remote control device 3 of the present invention is recorded on a computer-readable recording medium, and the program recorded on this recording medium is read by the computer system. , all or part of the processing performed by the robot remote control device 3 may be performed. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices. In addition, the "computer system" shall include a system built on a local network, a system built on the cloud, and the like. The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. In addition, "computer-readable recording medium" means a volatile memory (RAM) inside a computer system that acts as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. , includes those that hold the program for a certain period of time.

Further, the above program may be transmitted from a computer system storing this program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in a transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the program may be for realizing part of the functions described above. Further, it may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

As described above, the mode for carrying out the present invention has been described using the embodiments, but the present invention is not limited to such embodiments at all, and various modifications and replacements can be made without departing from the scope of the present invention. can be added.

DESCRIPTION OF SYMBOLS 1... Robot remote control system, 2... Robot, 3... Robot remote control device, 5... HMD, 6... Controller, 7... Environmental sensor, 21... Control part, 22... Drive part, 23... Sound pickup part, 25 Memory unit 26 Power supply 27 Sensors 222, 222a, 222b Grasping unit 31 Information acquiring unit 33 Intention estimating unit 331 Motion estimating unit 3311 Feature amount extracting unit 3312 Classifying unit , 332... Alignment support unit, 333... Alignment availability prediction unit, 34... Control command generation unit, 35... Image creation unit, 36... Transmission unit, 37... Storage unit, 51... Image display unit, 52... Line of sight detection unit, 53 ... sensor 54 ... control section 55 ... communication section 61 ... sensor 62 ... control section 63 ... communication section 64 ... feedback means 71 ... photographing device 72 ... sensor 73 ... object position detection section 74 ... communication department

Claims (9)

In a robot remote operation that recognizes the movement of an operator and transmits the movement of the operator to a robot to operate the robot,
an information acquisition unit that acquires an operator sensor value that is information indicating the detected movement of the operator;
an intention estimating unit for estimating a motion of the operator, which is a motion instruction to the robot, using a model that has been learned from the operator sensor values;
A robot remote control device comprising: The information acquisition unit acquires an environment sensor value obtained by an environment sensor installed in the robot or in the surrounding environment of the robot,
The intention estimating unit uses a model that has been learned from the environment sensor values and the operator sensor values to determine whether the operator is trying to align the gripped object gripped by the robot by operating the robot. estimating whether or not, and generating a control command value for driving the robot based on the estimated result;
The robot remote control device according to claim 1. The information acquisition unit acquires an environment sensor value obtained by an environment sensor installed in the robot or in the surrounding environment of the robot,
The intention estimating unit uses the environment sensor value and the operator sensor value to calculate the likelihood that the operator is going to align the gripped object in what kind of posture, thereby supporting alignment. I do,
The robot remote control device according to claim 2. The intention estimation unit estimates from the operator sensor value and calculates the likelihood with respect to the gripped object by the operator after processing for stabilizing the value.
The robot remote control device according to claim 3. The information acquisition unit acquires an environment sensor value obtained by an environment sensor installed in the robot or in the surrounding environment of the robot,
The intention estimating unit also inputs the environment sensor value to the learned model to estimate the operator's motion, which is a motion instruction to the robot.
The robot remote control device according to claim 1. The intention estimation unit extracts a feature amount from the operator sensor value, and estimates by classifying the operator's motion, which is an operation instruction to the robot, based on the extracted feature amount.
The robot remote control device according to claim 1 or 5. The robot remote control device according to any one of claims 1 to 6;
In a robot remote operation that recognizes the movement of an operator and transmits the movement of the operator to a robot to operate the robot,
a gripping portion that grips an object;
an environment sensor installed in the robot or in the surrounding environment of the robot and detecting an environment sensor value;
an operator sensor that detects the movement of the operator as an operator sensor value;
an image display device for displaying an image of the robot environment within the field of view of the operator;
A robot remote control system. In a robot remote operation that recognizes the movement of an operator and transmits the movement of the operator to a robot to operate the robot,
an information acquisition unit acquires an operator sensor value for detecting the movement of the operator;
an intention estimating unit estimating a motion of the operator, which is a motion instruction to the robot, using a learned model from the operator sensor values;
Robot remote control method. In a robot remote operation that recognizes the movement of an operator and transmits the movement of the operator to a robot to operate the robot,
to the computer,
obtaining an operator sensor value, which is information indicating the detected movement of the operator;
estimating a motion of the operator, which is a motion instruction to the robot, using a learned model from the operator sensor values;
program.

Images