JP3482228B2 - Manipulator control system by gaze detection - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a manipulator control system based on line-of-sight detection for remotely operating a manipulator. 2. Description of the Related Art Some conventional remote control means for a manipulator use a master-slave system. That is, in order to operate the slave manipulator installed in the actual work environment, the operator directly holds and operates a master manipulator having a structure similar to the slave manipulator. In the master-slave method, so-called bilateral control is used, in which the external force acting on the slave manipulator is transmitted by the force exerted by the master manipulator to the operator in order to compensate for the lack of realism which is a problem of the remote control means. Have been. In order to further enhance the visual realism, a stereoscopic image of an object is also projected on an installation environment of a master manipulator. However, in the above-described bilateral control, since the master manipulator has no actual state of the object, the master manipulator is inadvertently operated, and as a result, the slave manipulator is not operated. When contacting the object, the object or the slave manipulator may be subjected to an excessive force.
That is, at least one of them may be damaged. Even if an action to avoid this is added to the master manipulator under bilateral control, it is possible that the action force will not be transmitted to the slave manipulator due to the delay in signal transmission and arithmetic processing. There is. [0004] In addition, the master manipulator can be operated without giving the operator a visual sense of realism by means of a three-dimensional image and without looking at the three-dimensional image. If the operator inadvertently operates the master manipulator without seeing the stereoscopic image and the slave manipulator comes into contact with the object, the object or the slave manipulator may still receive excessive force. was there. SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and is based on gaze detection for preventing an excessive force from acting on an object or a slave manipulator due to an erroneous operation not intended by an operator in remote control of a manipulator. It aims at providing a manipulator control system. SUMMARY OF THE INVENTION The present invention provides a stereoscopic observation means for picking up an image of an object as right and left images having a convergence angle, and a left and right image obtained by the stereoscopic observation means. Stereoscopic display means for displaying observable, means for detecting an operator's line of sight to the stereoscopic display means for observing the stereoscopic display means, and an action manipulator for exerting a physical action on the object, Remote operation input means for giving an instruction for remotely operating the action manipulator, and the position of the observation target is calculated based on the line of sight of the operator detected by the detection means, and the position of the observation target and the remote operation are calculated. An arithmetic processing means for comparing a position of an operation point to be taken by the operation manipulator indicated by the input means to calculate a positional deviation; and a positional deviation calculated by the arithmetic processing means is defined. Control means for controlling the drive so that the position of the action point of the action manipulator becomes the position of the action point specified by the remote operation input means only when the value is smaller than the value. In the structure of the present invention, the position of the observation target is calculated by the arithmetic processing means based on the operator's line of sight detected by the detection means, and the position of the observation target and the remote operation input means are calculated. The instructed action manipulator compares the position of the action point to be taken with the position of the action point to calculate a positional deviation, and the control means controls the operation of the action manipulator only when the positional deviation calculated by the arithmetic processing means is smaller than a specified value. Drive control is performed such that the position of the point becomes the position of the action point specified by the remote operation input means. Therefore, according to the configuration of the present invention, when the positional deviation is larger than the specified value, that is, when the operator's line of sight deviates greatly from the operation point of the operation manipulator, for example, there is no instruction from the remote operation input means. However, since the action manipulator does not operate, it is possible to prevent an excessive force acting on the action manipulator or the object due to an erroneous operation. An embodiment of the present invention will be described below with reference to the drawings. 1 to 7 relate to a first embodiment of the present invention, FIG. 1 is an overall configuration diagram of a manipulator control system, FIG. 2 is a schematic configuration diagram of a head mount display and a stereoscopic scope, and FIG. Configuration diagram of detection means,
FIG. 4 is a configuration diagram of the distal end of the master arm and the operation manipulator, FIG. 5 is an explanatory diagram of the operation of the master arm, FIG. 6 is an explanatory diagram showing a use mode of the operation manipulator, and FIG. 7 is a flowchart showing the operation of the control device. The manipulator control system 3 shown in FIG.
Numeral 0 has a slave manipulator 1 as an action manipulator which is inserted into a body cavity and exerts a physical action on an object. As shown in FIG. 4A, the manipulator 1 includes a multi-joint arm 2 having a plurality of joints, a gripper 3 provided at a distal end of the multi-joint arm 2 and acting as an operating unit for gripping an organ or the like. And a tactile sensor 4 provided on the gripper 3. In addition, this manipulator 1
As shown in FIG. 1, the multi-joint arm 2 has a drive unit 5 that can be driven to drive the articulated arm 2 to change the bending angle of each joint. Further, the manipulator 1 is integrally provided with a stereoscopic scope 6 as a stereoscopic observation means for enabling stereoscopic observation of the inside of a body cavity, as shown in FIG. The manipulator control system 3
0 controls the operation of the manipulator 1 and
A control device 7 that processes signals of the left and right images captured by the stereoscopic scope 6, and a head mount display 8 that displays the signals processed by the control device 7 on the left and right images, respectively.
(Hereinafter referred to as HMD) and the manipulator 1
And a master arm 9 as a remote control input means for supplying an instruction amount for operating. The number of master arms 9 shown in FIGS. 4B and 5 is prepared corresponding to the number of the articulated arms 2 of the manipulator 1. The operator 10 has the master arm 9
Is operated by hand, the operation target position of the articulated arm 2 is determined under the control of the control device 7. Further, as shown in FIG. 5, the operator 10
The user operates the master arm 9 while viewing a stereoscopic image of the inside of the body cavity projected on the HMD 8 mounted on his own head. As shown in FIG. 1B, the stereoscopic scope 6
Are left and right objective optical systems 9a and 9b arranged at an angle of convergence with respect to an object serving as a subject;
CC that captures the left and right optical images formed by a and 9b
D10a and D10b are arranged on the tip side. In the stereoscopic scope 6, the left and right optical images formed by the objective optical systems 9a and 9b are photoelectrically converted by the CCDs 10a and 10b, and the converted electric signals are output to the control device 7. The stereoscopic scope 6 has a configuration that includes a variable focus optical system (not shown) in each of the left and right objective optical systems 9a and 9b. This varifocal optical system has one optical axis and divides the left and right optical images by pupil division.
A lens system that forms images at 0a and 10b may be used. The variable focus optical system performs focus adjustment under the control of the control device 7. As shown in FIG. 1, the control device 7 has a stereoscopic signal processing circuit 11 for inputting the electric signals of the CCDs 10a and 10b. The electrical signal is processed into a standard video signal. The HMD 8 is, as shown in FIG.
Left and right display units 8a and 8b are provided, and the left and right images are separately displayed on the display units 8a and 8b in response to the video signal. Thereby, the operator 10 can recognize the three-dimensional image of the object 32. That is, as shown in FIG. 2B, the real image 32 of the target object is observed with parallax by the objective optical systems 9a and 9b. However, as shown in FIG. These are displayed separately in the sections 8a and 8b. Therefore, the object 32 is observed with a similar parallax to the eyes 12a and 12b of the observer 10, and is three-dimensionally recognized as a virtual image 32 'of the object. As shown in FIG. 2A, the HMD 8 is provided with sight line detection sensors 13a and 13b for detecting the sight line direction of the eyeballs 12a and 12b. The visual line detection sensors 13a and 13b are connected to an arithmetic processing unit 14 of the control device 7, as shown in FIG. The arithmetic processing unit 14 obtains each line of sight of the eyeballs 12a and 12b based on signals detected by the line of sight detection sensors 13a and 13b, and calculates an object position calculation circuit 15 for calculating the position of the object. A position deviation calculating circuit that compares the position of the object obtained by the circuit 15 with the indicated value of the master arm 9 and calculates the deviation between the position of the object and the position of the operating point of the gripper 3 indicated by the indicated value. 16. The object position calculation circuit 15 also performs so-called autofocus control for driving and controlling the variable focus optical system of the stereoscopic scope 6 based on the obtained position of the object. Further, the control device 7 calculates the position deviation calculated by the position deviation calculation circuit 16 of the arithmetic processing unit 14 according to the following:
By controlling the driving unit 5 of the slave manipulator 1, the control circuit 17 as a control means for controlling the operation point of the gripper 3 to reach the operation point designated by the master arm 6 by moving the articulated arm 2 is provided. Have.
The control circuit 17 stops the operation of the manipulator 1 when the contact pressure between the gripper 3 and the object detected by the tactile sensor 4 is equal to or higher than a set value. In the control, the gripping operation may be stopped and the gripper 3 may be expanded in order to further enhance safety. As a specific example of the line-of-sight detection sensors 13a and 13b, as shown in FIG.
There is a method of observing the eyeballs 12a and 12b, and there is a method of detecting a line of sight detected from the relative relationship between the position of the eyeball and the center of the pupil by image processing. As another configuration, as shown in FIG. 3B, infrared light is emitted from the light emitting diode 19 to the eyes 12a and 12b,
There is known a method of detecting the direction of the line of sight by detecting the light reflected by a and 12b with a PSD (position sensing device) 20. Both methods are known and will not be described in detail. For example, in the photo industry (January 1993, P63-P64, P104-P105 / published by Photo Industry Co., Ltd.), the pupil and the cornea of the eye are used. A method is described in which the positional relationship between the reflected infrared light and a reflected image is calculated at a high speed by a microcomputer to determine the rotation angle of the eyeball, and to calculate where the user is looking. In FIG. 2B, when the operator 10 is observing the object, the line of sight of the left and right eyes intersects on the virtual image 32 'of the object. 13a and 13b, the virtual image 3 of the object
A 2 ′ three-dimensional position is obtained. Further, the HM
Since the relationship between the optical system of D8 and the stereoscopic scope 6 is known, the three-dimensional position of the real image of the object with respect to the stereoscopic scope 6 is obtained from the three-dimensional position of the virtual image of the object in the HMD 8. . The relationship between the optical systems is, for example, the eyeballs 12 a and 12 b and the display unit 8 in the HMD 8.
a, 8b, the positional relationship between the objective optical systems 9a, 9b and the CCDs 10a, 10b in the stereoscopic scope 6, the magnification of each optical system, and the like. Ideally, if the relationship between the optical systems is similar, θL = ψL shown in FIG.
θR = ψR holds. However, even if there is no similarity, since the relationship of the optical system is known, the three-dimensional position of the real image can be easily obtained. Note that ψL and ψR are declination angles with respect to the optical axes of the objective optical systems 9a and 9b and the CCDs 10a and 10b when the objective optical systems 9a and 9b are focused on the object 32. ΘL, θR are the left and right eyeballs 12a, 1
It is the angle between the central axes A and B of the cornea 21a and 21b of 2b, that is, the directions of the left and right lines of sight and the display units 8a and 8b. Next, the operation of the control device 7 will be described with reference to the usage pattern shown in FIG. 6, the flowchart shown in FIG. 7, and FIGS. 2 and 3. FIG. 7A is a flowchart for obtaining the three-dimensional position of the target object by detecting the line of sight and performing the autofocus of the stereoscopic scope 6 using the result. FIG. 7B is a flowchart for operating the slave manipulator 1 using the three-dimensional position of the object obtained by the line-of-sight detection as a target point. FIG. 7 (c)
Is a flowchart relating to control for preventing erroneous operation when remotely operating the slave manipulator 1 by using the three-dimensional position of the object obtained by the line of sight detection. This flow is a control method different from the control method shown in FIG. 7B, in which the movement of the master arm 9 is performed only when the operation command from the master arm 9 and the position of the observation target are within the specified range. The slave manipulator 1 operates according to the above. FIG. 6 shows a state in which an operation in the body cavity of the subject 31 is performed by remote control in the present system. The distal end of the slave manipulator 1 is inserted into the body cavity 33 via the trocar 22 that cuts the subject 31. The slave manipulator 1 is provided with a stereoscopic scope 6, and this image is transmitted to a controller 7.
Is displayed on the HMD 8. The operator 10 has the HMD 8
When the master arm 9 is operated while watching the image displayed on the slave manipulator 1 inserted into the body cavity 33,
Operates in accordance with the movement of the master arm 9, and the body cavity 3
Surgery can be performed at 3. As shown in FIG. 2A, the HMD 8 is provided with sensors 13a and 13b for detecting the direction of the line of sight, so that the operator 10 can recognize the object to be observed by using both eyes. The lines of sight 12a and 12b need to be directed to the observation target (see SS1 and SS2 in FIG. 7A). Therefore, conversely, the position of the observation target is detected by detecting the line-of-sight directions of the eyes 12a and 12b. That is, the object position calculation circuit 15 corrects the geometrical difference of the optical system between the HMD 8 and the stereoscopic scope 6 with respect to the left and right visual line directions θL and θR, and Azimuth angle ψL, ψR to observation object
Is obtained, and the position of the observation target is detected in the same manner as in the triangulation (see steps S3 to S5 in FIG. 7A). At this time, as shown in step S6, the object position calculation circuit 15 performs autofocus control so that the observation object of interest is always focused. In FIG. 2 and FIG. 7A, the problem is treated as a plane for ease of explanation. However, the position of the observation target can be detected in space according to the same principle. As a control method of the control device 7, FIG.
There is a flow shown in FIG. In this control, the control circuit 17 uses the three-dimensional (line of sight) position obtained by the object position calculation circuit 15 as a target point so that the action point of the slave manipulator 1 reaches the multi-joint arm via the drive unit 5. 2 is drive-controlled. On the other hand, another control method of the control device 7 is shown in the flow chart of FIG. As shown in FIGS. 4 (a) and 4 (b), the position of the action point designated by the master arm 9 is P, and the three-dimensional position of the object obtained by detecting the line of sight is Q.
Further, the deviation amount as a specified value indicating the movable proper range of the gripper 3 due to the operation of the articulated arm 2 is defined as R. In FIG. 4B, the point Q coincides with the operation point of the gripper 3. Under the control of the control circuit 17, | P-
Only under the condition of Q | <R, the articulated arm 2 is allowed to follow the position P of the action point specified by the master arm 9. In other words, the control circuit 17 restricts the remote operation only when the operator 10 is watching the range of the radius R centering on the point of action of the articulated arm 2. 4A and 4B, the action point of the slave manipulator 1 and the designated point P of the master arm 9 are shown.
Consider a situation where If the operator 10 shifts the viewpoint to the point Q1 in FIG. 4B, FIG.
In step S8, since the set value R is smaller than | P−Q |, which is the positional deviation between the position of the target object and the gazing point, the control circuit 17 proceeds to step S9 and enters the operation permission state. Here, the control circuit 17 controls the slave manipulator 1 to follow the position P by moving the designated position P of the master arm 9 which is an input means of the remote operation. When the operator 10 shifts his or her viewpoint to the point Q2 in FIG. 4B, the set value R is larger than | PQ | in step S8 in FIG. The control circuit 17 proceeds to step S10, and the operation of the articulated arm 2 is prohibited. In this embodiment, if the observer 10 does not observe at least the vicinity of the action point of the slave manipulator 1 or, specifically, does not observe a point within the set value R, the operation by the master arm 9 is performed. Even if there is a command, the slave manipulator 1 does not operate. For this reason, in the present embodiment, it is possible to prevent the object or the multi-joint arm from being damaged due to an erroneous operation not intended by the operator, and it is possible to enhance safety. This embodiment is not limited to the case where the operator shifts his / her gaze from the current point of action of the slave manipulator to another, but in general, the case where the consciousness is shifted to another and the line of sight is also shifted from the point of action. Can avoid an accident due to an erroneous operation of the master manipulator even in a situation where the concentration of the operator is interrupted and the consciousness is neglected. FIGS. 8 and 9 relate to a second embodiment of the present invention. FIG. 8 is a block diagram of a manipulator control system.
FIG. 9 is an explanatory diagram of the operation of the system shown in FIG. The manipulator control system of this embodiment has a separate operation manipulator and a manipulator for observation, and is configured to remotely control the operation manipulator by a separate master manipulator. The control system is configured to remotely control an observation manipulator having a stereoscopic observation observation unit in accordance with the movement of the HMD. In addition, the same configurations and operations as those of the first embodiment are denoted by the same reference numerals, and description thereof is omitted. The system 35 of this embodiment shown in FIG. 8 has two treatment manipulators 24a and 24b used for performing treatment on a subject and an observation manipulator 25. In the present system 35, these manipulators can be inserted into a body cavity to perform a necessary operation in the body cavity. At the tip of the observation manipulator 25, a stereoscopic scope 25a shown in FIG.
Is provided. The HMD 8A has the HMD shown in FIG.
In addition to the configuration of FIG. 8, a three-dimensional position sensor 26 capable of detecting information of six degrees of freedom regarding the position and inclination of the HMD 8A is provided. The three-dimensional position sensor 26 of this HMD8A
Is composed of, for example, three detection coils 26a wound in three orthogonal directions shown in FIG. 9B and a fixed coil (not shown) having a similar structure and fixed at a fixed position. By detecting the intensity of the magnetic field emitted from the fixed coil by the detection coil 26a, six degrees of freedom information on the position and the inclination can be obtained. The amount of displacement of the HMD 8A detected by the three-dimensional position sensor 26 is supplied to a control circuit 27 of the control device 7. The control circuit 27 controls the direction of the visual field of the stereoscopic scope 25a in accordance with the amount of displacement detected by the three-dimensional position sensor 26. The master manipulators 28a, 28
8b is connected to the position deviation calculation circuit 16A of the control device 7. The control circuit 27 controls the treatment manipulator 24 according to each deviation between the position indicated by the master manipulators 28a and 28b and the position of the object position detection circuit position 5 from the observation target.
a and 24b are respectively controlled. With the above configuration, the operator can operate the master manipulators 28a, 28a while the HMD 8A is mounted.
The treatment manipulators 24a and 24b are operated by causing the user to hold the hand 8b. The control circuit 27 includes a 3D position sensor 26
, Information of six degrees of freedom relating to the position and inclination of the HMD 8 is obtained, and the operating position of the observation manipulator 25 is determined according to the obtained information. Further, the three-dimensional position obtained from the line-of-sight detection sensors 13a and 13b by the method of the first embodiment described above and the treatment manipulator 24
Only when the action points of the a and 24b are within the range of the specified value, the control circuit 27 controls the master manipulator 28a,
28b according to the operation of the treatment manipulator 24a,
4b is operating. According to this embodiment, at the same time as viewing the stereoscopic image, the positional relationship between the hand and the eye can be operated while maintaining the positional relationship between the treatment manipulator and the observation manipulator in the body cavity. Realism is obtained.
Further, in this embodiment, the operation can be safely performed by the line of sight detection as in the first embodiment. As described above, according to the present invention,
In the remote control of the manipulator, it is possible to prevent an unreasonable operation by the operator from applying an unreasonable force to the target object or the action manipulator, thereby providing an effect of enabling a safer operation.

BRIEF DESCRIPTION OF THE DRAWINGS FIGS. 1 to 7 relate to a first embodiment, and FIG. 1 is an overall configuration diagram of a manipulator control system. FIG. 2 is a schematic configuration diagram of a head-mounted display and a stereoscopic scope. FIG. 3 is a configuration diagram of a visual line detection unit. FIG. 4 is a front end configuration diagram of a master arm and an action manipulator. FIG. 5 is an operation explanatory view of a master arm. FIG. 6 is an explanatory view showing a use mode of the action manipulator. FIG. 7 is a flowchart showing the operation of the control device. 8 and 9 relate to a second embodiment, and FIG. 8 is a configuration diagram of a manipulator control system. FIG. 9 is an explanatory diagram of the operation of the system shown in FIG. 8; [Description of Signs] 30 ... Manipulator control system 1 ... Slave manipulator 2 ... Articulated arm 3 ... Gripper 6 ... Stereoscopic scope 7 ... Control device 8 ... HMD 9 ... Master arms 13a and 13b ... Gaze position detection sensor 14 ... Calculation processing Unit 15: Object position calculation circuit 16: Position deviation calculation circuit 17: Control circuit

──────────────────────────────────────────────────続 き Continued on the front page (58) Fields surveyed (Int.Cl. ⁷ , DB name) B25J 3/00 B25J 13/00 B25J 19/00 A61B 1/00

Claims (1)

(57) [Claims 1] Stereoscopic observation means for imaging an object as right and left images having a convergence angle, and left and right images obtained by the stereoscopic observation means can be stereoscopically observed. A means for detecting a line of sight of the operator observing the stereoscopic display means on the stereoscopic display means, an action manipulator for exerting a physical action on the object, and the action Remote operation input means for giving an instruction for remotely manipulating the manipulator; calculating the position of the observation target based on the line of sight of the operator detected by the detection means; and calculating the position of the observation target and the remote operation input means An arithmetic processing means for comparing a position of an action point to be taken by the action manipulator to calculate a positional deviation, wherein the positional deviation calculated by the arithmetic processing means is smaller than a specified value Control means for controlling the operation of the operation manipulator so that the position of the operation point of the operation manipulator becomes the position of the operation point specified by the remote operation input means. system