WO2015014668A1

WO2015014668A1 - Input device for gesture control, having a protection device

Info

Publication number: WO2015014668A1
Application number: PCT/EP2014/065708
Authority: WO
Inventors: Bernd Gombert
Original assignee: gomtec GmbH
Priority date: 2013-07-30
Filing date: 2014-07-22
Publication date: 2015-02-05
Also published as: DE102013108114B4; DE102013108114A1

Abstract

The invention relates to an input device (10) for controlling a robot system (3), having an image processing system (2, 9) for detecting gestures, which are defined by an operator by moving at least one arm (6a, 6b) and/or at least one hand, and having a control unit (9), which recognizes certain gestures and converts them into corresponding control signals for controlling the robot system (3). According to the invention, an arm-operated protection device (1, 9) is provided, by way of which the control of the robot system (3) can be activated or deactivated.

Description

Input device for gesture control with protective device

The invention relates to an input device for controlling a robot by means of gestures according to the preamble of patent claim 1, and to a method for controlling a robot according to the preamble of patent claim 13.

Known robot systems, to which reference will now be made, include an input device for gesture control of a robot having an image processing system for detecting gestures generated by a robot

User z. B. by moving his arms or hands are given. The image processing system includes a camera and a control unit which processes the image data captured by the camera and converts it into corresponding control signals. The control unit comprises for this purpose a software algorithm that is able to detect certain gestures of the user, and finally generates control signals with which the actuators of the robot or z. B. a tool mounted on the robot can be controlled, so that the robot or the tool performs the actions desired by the user. More complex robot systems, such. B. Surgery robots are often operated simultaneously with two hands. In such systems, the user z. B. with a first hand a first control function such. B. a movement of

Robot head, and with a second hand, a second control function, such. As an operation of a surgical instrument perform.

The operation of such a robot system by means of gesture control is relatively difficult and requires a certain training time. In industrial

In applications, maloperation of the robot may result in damage to the machined workpiece or component of the robotic system. In the case of a surgery robot, a misoperation can lead to injury to the patient and thus far more serious consequences. In the field of surgery robots therefore protective devices are used, which should prevent one from unwanted by the operator or accidentally executed gesture or control specification is converted into a corresponding action of the robot.

From US 8 332 072 B1, for example, a manually operated

Input device for controlling a surgery robot is known, which is operated in a similar manner as a joystick. The control specifications manually executed by the operator are converted into corresponding control commands for the robot. To avoid erroneous operation of the robot, while operating the input device, the operator must simultaneously operate a switch integrated with a handle of the input device. When the operator grasps the handle, the switch is automatically activated. In doing so, the operator confirms that the inputs he has made are intentional and valid. However, when the operator releases his hand from the grip, the input device becomes

automatically deactivated. Unwanted tax requirements, such. B. accidental bumping against the input device, are thus not in appropriate

Control commands implemented for the robot and therefore have no effect.

From US 201 1/0 001 813 A1 is also an input device for

Gesture control of a robot known that can recognize different hand gestures and converts into corresponding control commands for the robot. The input device comprises an image processing system with which the gestures are recorded. In such an input device, however, erroneous operations of the robot may occur when the operator moves the arm or the hand, but does not want to control the robot. Since both hands are usually required to carry out certain control commands or gestures, a handle with a switch integrated therein, as is known from US Pat. No. 8,232,072 B1, would be a hindrance in this case.

It is therefore an object of the present invention to provide an input device for gesture control of a robot system, in the incorrect operation of the robot system by unintentionally executed control instructions in

Are essentially excluded. This object is achieved according to the invention by the features specified in claim 1 and in claim 13. Further

Embodiments of the invention will become apparent from the dependent claims. According to the invention, an input device for controlling a

Robot system proposed by gestures comprising an image processing system for detecting gestures, which are given by an operator by moving at least one arm and / or at least one hand, and further comprising a control unit that detects certain gestures and converts them into corresponding control commands for controlling the robot system , According to the invention, the input device further comprises a protective device which is actuated by means of an arm, wherein the protective device actively switches the gesture control when it is actuated, and deactivated when it is not actuated. Since the protective device according to the invention comprises an arm-operated sensor, the operator of the robot has his hands free and can thus use the robot

steer freely. By lifting the arm away from the guard, it may indicate that its subsequent arm or hand movements are no longer to be considered control inputs, and thus may affect the gesture control of the robotic system, at least with respect to that arm or hand

deactivate.

For the purposes of this document, a "robot system" is understood in particular to be a technical system with one or more robots, which may also comprise one or more robot-operated tools and / or one or more additional machines may be equipped, for example, one or more

Robotic, each equipped with a surgical instrument or other tool, and include an electrically adjustable operating table, wherein all components of the robot system can be preferably controlled by gesture control.

In the context of this document, a "robot" is understood in particular to mean any machine having one or more articulated arms which are movable by means of one or more actuators, such as electric motors. According to the invention, the input device comprises an image acquisition system which converts the hand or arm movements of an operator into corresponding ones

Converts control signals with which then a machine is controlled. Such a system has z. For example, a camera that can record various gestures of a user within an image capture area. The recorded gestures can z. B. be compared with a database in the

various gestures are stored. If a match of the

Operator's gesture with a stored gesture, an associated control command is triggered.

According to the invention, the protective device comprises at least one sensor which can be actuated by means of an arm. Since the sensor is operated by means of the arm, the operator has his hands free and can precisely control the robot by means of gesture control.

In the above-mentioned sensor, it may, for. B. one

touch-sensitive sensor, in particular a switch or button, or a non-contact sensor, such. As a proximity sensor or a camera act. In the case of the camera, a particular position or position of the arm could then be detected, as well as deformations of a

Armrest on which the arms can support.

The sensor is preferably arranged so that it can be actuated by means of the elbow or forearm.

According to a particular embodiment of the invention, the at least one sensor z. B. be integrated in an armrest. The armrest can

for example, be the armrest of a chair. Alternatively, but could also be provided a separate armrest, the z. B. may be attached to the chair or other device of the robot system. According to another

Embodiment of the invention could also be provided a kind of table on which the operator is supported with his two quick bows. In this case, one or more sensors for activating / deactivating the table are in the table Gesture control integrated. The armrest could also be integrated in a control panel.

According to a preferred embodiment of the invention, a sensor is provided for both arms of the operator. When the inventive

Protective device includes at least two sensors ("two sensors" should also be understood a sensor that is sensitive to the touch by two arms), the robot system can be activated or deactivated depending on the operating state of the sensors according to different methods:

According to a first embodiment of the invention, the protection device is designed such that the input device or gesture control is completely deactivated if at least one of the sensors is not actuated. The operator must therefore operate both sensors by means of his arms in order to be able to control the robot system. If he just removes one of his arms from the sensor, the input device or gesture control is completely deactivated.

According to a second embodiment of the invention, the protective device is designed such that the input device is deactivated only for the arm which does not actuate the associated sensor. As long as the operator but with at least one arm actuates the associated sensor, he can

Continue to control the robot system with this arm or its hand.

Movements that the operator makes with one arm or his hand without activating the associated sensor are not considered

Control input interpreted and thus not implemented in a corresponding action of the robot system.

According to a third embodiment of the invention, the protection device can also be designed such that the input device is activated for both arms when at least one of the sensors is actuated, and is completely deactivated when none of the sensors is actuated. This means that the operator only has to operate at least one of the sensors in order to be able to control the robot system with both arms or hands. By means of the input device according to the invention basically any components of a robot system can be controlled, such. B.

Robots of any design, robot-operated tools, but also any other machines or individual parts of the robot system. The left and right arms and / or the left or right hand can be used to control the same or the same actuators or different actuators of a robot and / or a tool and / or another component of the

Serve robot system. So z. For example, the position of a robot head by means of the first arm and the first hand, and one carried by the robot

Tool are controlled by means of the second arm and the second hand. The input device or its image acquisition system is in this case preferably designed such that it can distinguish the gestures executed by the left and right arms or left and right hands. According to a special embodiment of the invention, the protective device is preferably designed to be redundant. For this purpose, the image acquisition system checks whether at least one sensor is actuated by an arm, and whether the signal output by the sensor coincides with the result of the image processing. When the sensor or its output signal indicates an actuated state and the image processing system also recognizes an actuated state, the input device is active at least for the relevant arm. However, if the detected by the sensor operating state of the of

Detected imaging state system actuation state, this state is detected by the protection device as an error and the input device is preferably completely disabled.

The invention also relates to a method for controlling a robot by means of gesture control, in which the gestures predetermined by an operator by moving his arms or hands are detected by means of an image processing system and the image data recorded by the image processing system are processed and converted into corresponding control commands for controlling the robot system. According to the invention, it is further determined whether a safety device operable by the operator by means of an arm is actuated. if the

Protective device is actuated, the operator can at least the robot system steer with one arm and / or one hand. However, if the safety device is not actuated, the input device is deactivated. The operator can do that

Robot system then at least with one of his arms and / or hands no longer control.

Brief description of the drawings

The invention will now be described by way of example with reference to the accompanying drawings. Show it:

Fig. 1 is a schematic representation of a robot system with a

An image acquisition system serving as an input device for controlling a robot; Fig. 2 shows a first embodiment of a protective device for

Activating / deactivating the robot system of Fig. 1; and

Fig. 3 shows a second embodiment of a protective device for

Activating / deactivating the robot system of FIG. 1.

Fig. 4 shows a development of the robot system, which provides an optical feedback to the user of the protective device.

Embodiments of the invention

1 shows a robot system with a robot 3 for minimally invasive surgery. The robot 3 in this embodiment has two articulated arms, which are each connected to one another via one or more joints. The individual joints are each driven by a motor, here an electric motor (not shown), and can perform a pivoting and / or rolling movement of the articulated arms.

The robot 3 has a freely movable end, which also acts as a robot head

can be referred to, as well as a fixed end, here at an operating table 12th is mounted. On the robot head, a surgical instrument 1 1 is attached, which is tailored to the particular application of the robot. For the

Surgical use of the robot 3, the robot head, for example, with an endoscope, a laparoscopic instrument, a cutting, gripping, holding, connecting, sewing or other surgical tool or

Instrument 1 1 be equipped.

The robot system shown in FIG. 1 is controlled by means of an input device 10, which in this exemplary embodiment has a camera 2 and a camera

Control unit 9 includes. The camera 2 is on the operator of the robot system, such. As a doctor, directed, the two arms 6 a, 6 b are shown schematically here. The operator can by executing certain control instructions by means of his arms 6a, 6b or hands the robot 3 and / or the on

Robot head attached tool and / or other components of the

Robot system, such. B. the operating table 12, control.

During operation, the camera 2 continuously records images that are processed by an image processing software included in the control unit 9. The image processing software is capable of interpreting movements made with the forearms, hands or fingers as a control input and generating corresponding control signals. To detect a specific movement or gesture, the images taken by the camera 2 z. B. are compared with the stored in a database movement patterns or gestures. Upon coincidence of a detected gesture with a gesture stored in the database, an associated control signal is generated by which, finally, the robot 3 or another component of the robotic system, e.g. the instrument 1 1, is controlled. The input device 10 is preferably configured so that both static gestures, such. B.

Hand signals, as well as dynamic gestures, such. B. Waving, can be detected.

By executing certain tax specifications can thus z. B. the position of the robot head and / or the position of the surgical instrument 1 1 can be influenced. But it could also be individual actuators of the robot 3 or the surgical instrument 1 1 are controlled separately. So could

for example, by a translational movement of a first arm, z. B. 6a, the robot moves accordingly and operated by a running with the other hand gesture the tool or surgical instrument 1 1. The action taken by the robotic system in response to a particular control default is preferably freely configurable.

In order to prevent the patient from being injured by accidental or accidental arm and / or hand movements when carrying out a surgical procedure, the robot system shown in FIG. 1 further comprises a protective device 1, 9 with which the gesture control is selectively deactivated can be. By triggering the protective device 1, 9, the operator can indicate to the robot system that subsequent hand or

Arm movements should no longer be interpreted as control specifications and can thus disable the gesture control of the robot system. in the

illustrated embodiment, the protective device comprises an armrest 4, in which for each arm 6a, 6b, an associated sensor 5a, 5b is integrated. The armrest 4 can z. B. be part of a chair or seat back. The sensors 5a, 5b may be, for example, touch-sensitive sensors, such. As switches or buttons, or acting on the capacitive or inductive principle sensors. Alternatively, non-contact sensors, such. B. Proximity switch, be provided, which already react when approaching the sensor surface.

As can be seen in Fig. 1, the operator relies on controlling the

Robot system with his elbows 8a, 8b each on one of the sensors 5a, 5b. The forearms 7a, 7b and the hands remain free to move. The operator can thus by means of his forearms 7a, 7b and / or hands

perform various movements and thus the robot system

control accordingly. If the operator takes one of his arms 6a, 6b from the associated sensor 5a, 5b, he can thereby control the gesture of the

Disable robot system. The protective device 1, 9 functions like a switch with which activates or deactivates the control of the robot system can be. The operator places his elbow 8a, 8b on one of the

Sensors 5a, 5b, he indicates that of him in the following

gestures are to be considered as taxable In this case, the gestures detected by the image acquisition system 10 are converted into corresponding ones

Control signals implemented. Would the operator like his arm or

Hand movements are not interpreted as a control, so he can lift his arm from the respective sensor 5a, 5b and thus disable the control.

If, as in the illustrated embodiment, the placement of both arms 6a, 6b is sensory detected, there are in principle several ways that

Protective device 1, 9 to configure. According to a first embodiment of the invention, the protection device 1, 9 is configured such that both sensors 5a, 5b must be actuated in order to activate the control. If only one of the sensors 5a, 5b is not actuated, the control is completely deactivated and the robot system can no longer be controlled by input of gestures.

According to a second embodiment of the invention, such as. For example, as shown in Fig. 3, the protection device is configured such that the control is activated for an arm 6a, 6b which actuates one of the sensors 5a, 5b. An arm 6a, 6b which does not actuate a sensor 5a, 5b, on the other hand, is ignored by the input device 10 and can not be used to control the robot system. Furthermore, the gesture recognition is designed to distinguish the right arm / hand and the left arm / hand of a user from each other and evaluate separately. An appropriate algorithm may be implemented in gesture recognition.

According to a third embodiment of the invention, the protection device 1, 9 can also be configured such that the robot system can be controlled by both arms 6a, 6b or hands, if at least one of the sensors 5a, 5b is actuated. That the operator only has to operate at least one of the sensors 5a, 5b in order to be able to control the robot system with both arms. If, on the other hand, none of the sensors 5a, 5b is actuated, the gesture control will be activated

completely disabled. To activate or deactivate the gesture control, the input device 10 z. B. have a hardware or software switch, the z. B. the forwarding of the control signals to the robot 3 interrupts. Instead of two separate sensors 5a, 5b, of course, a single sensor could be provided, which is able to distinguish an operation by means of one or two arms 6a, 6b.

According to a preferred embodiment of the invention is the

Protective device 1, 9 designed redundantly. In this case, that will

Imaging system 10 used to check whether at least one of the sensors 5a, 5b by an arm 6a, 6b, is actuated. The signal output by the sensors 5a, 5b is then compared with the result of the optical image recognition. If both signals indicate in unison that one of the sensors 5a, 5b is actuated, the actuated state of the sensor 5a, 5b is assumed to be safe. Depending on the embodiment of the protective device then the gesture control can be activated or deactivated accordingly.

According to one embodiment of the invention, the operation of the

Protective device can be freely selected by the operator. This can be the

Robot system z. B. have a switch or other input means with which can be switched between the different modes. So z. B. experienced users the less restrictive mode of the

Activate protection, which is sufficient when one of the sensors is operated, and inexperienced users can select the more restrictive mode in which both sensors 5a, 5b must be pressed to activate the gesture control.

Fig. 2 shows an embodiment of a protective device with a sensor 5, which is integrated in an armrest of a chair. The protective device is activated or deactivated in this case by placing a forearm 7b.

3 shows an embodiment of a protective device with two sensors 5a, 5b, which are integrated in an armrest 4. For each arm 6a, 6b is a separate sensor 5a and 5b provided. As can be seen, in this embodiment, the sensors are actuated by supporting the rapid bends 8a, 8b. Depending on the design of the edition the user will be different

Degrees of freedom. In the case of the armrest of Fig. 2, the user can only move his hands freely. The representation of gestures for controlling the robot system 3 is thus limited to hand or finger movements. In the case of the elbow pad of Fig. 1 or Fig. 3, the user can move his forearms 7a, 7b. That the user can thus also include a movement of his forearms 7a, 7b to indicate certain gestures, so that z. B. a specific forearm movement can be interpreted as a gesture. FIG. 4 shows a development of the invention which provides an optical feedback to the user of the protective device 1. Thus, the user can be displayed to what extent the robot 3, the instrument 1 1 and / or the operating table 12 are switched to be active or inactive by the protective device 1. For this purpose, the protective device 1 is connected to a monitor 13, which represents the user-to-use elements. The status of the element to be controlled can be highlighted in color on the monitor. For example, all active elements can be displayed green and all inactive elements red on the monitor.

For example, instrument 1 1 is operated and displayed on the monitor 13. The instrument 1 1 has a plurality of axes 14, 15 and 16 to the parts of the

Instruments 1 1 can be swiveled. For example, gesture recognition can be used to control individual parts of the instrument. Thus, with the arm 6a, the shaft 17 of the instrument 11 can be rotated about the axis 14 and the arm 6b of the end effector 18 can be actuated about the axis 16.

However, since only the arm 6b operates the protector 1, depending on

Embodiment of the invention, different elements of the instrument 1 1 active. For example, the protective device 1 operates after the second

Embodiment in which only elements of that arm are controlled can, who operates the protection device 1. According to FIG. 4, only arm 6b actuates the protective device 1, so that consequently only the end effector 18 could be moved about the axis 16. Gestures of the arm 6a would be ignored and would not trigger movement of the instrument 1 1 about the axis 14. Accordingly, axis 16 would be active and axis 14 inactive. This condition could be indicated to the user on the monitor by a corresponding color marking. For example, the end effector 18 and / or its axis 16 may be green and the shaft 17 and / or its axis 14 may be red. Thus, the user immediately receives a status feedback on which elements are currently active and which are not.

The development can be transferred analogously to any of the hands or arms controllable elements. Thus, gesture-recognition operation is not limited to just one instrument, but can include multiple instruments. Furthermore, instead of an instrument, a robot and / or the operating table can also be controlled. For example, the user can control two instruments simultaneously by means of gestures of his hands or arms, whereby he can control an instrument 19 with arm 6a and another instrument 11 with arm 6b. Analogous to the above embodiment but only one instrument can be operated, namely instrument 1 1, since only arm 6b operates the protection device 1. Ie. Instrument 19 would be inactive. In order to display this state in turn on the monitor 13 in color, the instrument 19 with a red

Marker and instrument 1 1 are highlighted with a green marker.

Claims

claims

1 . Input device (10) for controlling a robot (3) by means of gestures, with an image processing system (2, 9) for detecting gestures given by a user by moving at least one arm (6a, 6b) and / or one hand, and with a control unit (9), which recognize certain gestures and in corresponding control signals for

Control of the robot (3), an instrument (1 1) and / or an operating table (12), characterized by an arm-operated protection device (1, 9), which switches the gesture control of the robot (3) active when they is pressed, and deactivated when it is not operated.

Second input device (10) according to claim 1, characterized in that the protective device (1, 9) comprises at least one arm-operated sensor (5a, 5b).

3. Input device (10) according to claim 2, characterized in that the protective device (1, 9) at least one touch-sensitive sensor (5a, 5b), in particular a switch or push button, and / or at least one non-contact sensor (5a, 5b), especially one

Proximity sensor includes.

4. input device (10) according to claim 2 or 3, characterized in that the at least one sensor (5a, 5b) in an armrest (4) is integrated.

5. input device (10) according to claim 4, characterized in that the armrest (4) is part of a chair and in particular an armrest.

6. input device (10) according to any one of the preceding claims, characterized in that the protective device (1, 9) in such a way

is formed so that it can be operated by means of an elbow (8a, 8b) or by means of a forearm (7a, 7b).

7. input device (10) according to one of the preceding claims,

characterized in that the protective device (1, 9) for both arms (6a, 6b) of a user each having a sensor (5a, 5b).

8. input device (10) according to claim 7, characterized in that the protective device (1, 9) is designed such that the gesture control of the robot (3), the instrument (1 1) and / or the operating table (12) is deactivated for both arms, if one of the sensors (5a, 5b) is not operated.

9. input device (10) according to claim 7, characterized in that the protective device (1, 9) is designed such that the gesture control of the robot (3), the instrument (1 1) and / or the operating table (12) is deactivated only for those arm (6a, 6b) which does not actuate the associated sensor (5a, 5b).

10. Input device (10) according to claim 7, characterized in that the protective device (1, 9) is designed such that the gesture control of the robot (3), the instrument (1 1) and / or the operating table (12). for both arms (6a, 6b) is active when at least one of the sensors (5a, 5b) is actuated.

1 1. Input device (10) according to one of the preceding claims,

characterized in that the image processing system (2, 9) is adapted to distinguish gestures made by the left and right arms (6a, 6b) and the left and right hands, respectively.

12. Input device (10) according to one of the preceding claims, characterized in that the image processing system (2, 9) checks whether the at least one sensor (5a, 5b) is actuated, and the

Protective device (1, 9) is designed such that the gesture control is only active if the at least one sensor (5a, 5b) indicates that it is actuated and the image processing system (2, 9) detects that the sensor (5a , 5b) is actuated.

13. Input device (10) according to one of the preceding claims,

characterized in that the input device (10) comprises means for

Selecting a particular mode has, so that the

Input device (10) according to the selected mode either according to claim 8, 9 or 10 can be operated.

14. Input device (10) according to one of the preceding claims,

characterized in that the input device (10) comprises means for optically displaying (13) the respective status of the elements (3, 11, 12, 17, 18, 19) controllable by means of gestures of the hands and / or arms (6a, 6b) Dependence of the operation of the protective device (1).

15. A method for controlling a robot (3) by means of gesture control, comprising the following steps:

- Detecting gestures, which are given by a user by moving his arms (6a, 6b) and / or hands, by means of a

Image processing system (2, 9); and

- Controlling the robot (3), the instrument (1 1) and / or the operating table (12) according to the gestures given by the user; further characterized by the following steps: Determining whether the user actuates an arm-operated protective device (1, 9),

Activating the gesture control when it has been detected that the user operates the protection device (1, 9); and

Deactivate the gesture control if it has been detected that the

User does not operate the protective device (1, 9).