WO2013164622A1

WO2013164622A1 - System and method for manipulating objects using one or more active effectors controlled by one or more haptic manipulators

Info

Publication number: WO2013164622A1
Application number: PCT/GB2013/051135
Authority: WO
Inventors: Mario Ettore Giardini
Original assignee: University Court Of The University Of St Andrews
Priority date: 2012-05-04
Filing date: 2013-05-02
Publication date: 2013-11-07
Also published as: GB201207877D0

Abstract

A system for manipulating an object using a haptic manipulator coupled with an active effector, the system comprising: sensing means for sensing a parameter of the object being manipulated and/or of the environment surrounding the object, wherein the sensing means comprise an optical sensor, and means for generating a signal to cause the haptic manipulator to exert a force and/or torque on a user, wherein the force exerted depends on the optically sensed parameter.

Description

SYSTEM AND METHOD FOR MANIPULATING OBJECTS USING ONE OR MORE ACTIVE

EFFECTORS

CONTROLLED BY ONE OR MORE HAPTIC MANIPULATORS

Introduction

The present invention relates to a system and method for manipulating objects using one or more active effectors controlled by one or more haptic manipulators.

Background

Haptic perception is the combination of cutaneous feeling and position of body limbs and joints. Haptic perception is key to manipulation of an object by an operator, as it greatly facilitates appropriate handling of the object, e.g. by allowing the application of appropriate forces to the object, the correct handling speed, the application of appropriate torques to the object, a prompt response to object deformations due to object stiffness, etc. Ultimately, this allows movement efficacy and precision to be achieved with no damage to the object being handled.

When an operator controls an effector, such as a robot, via a real-time controller, such as a 3-D joystick, hand-wheel or similar, haptic perception of the effector manipulation, e.g. of the objects being manipulated by the robot end-effector, is absent, unless specific provisions are made. Examples of arrangements for providing haptic perception to the operator require sensing to determine the force and/or torque enacted by the effector. Such sensing may be performed either through direct force/torque sensors embedded in the effector, or by an analysis of the effector control parameters (such as motor voltages, motor currents, gas or oil pressure in pressure-based effectors). Alternatively, image analysis of the effector operating field could be used, for example by analysing the deformation of imaged manipulated objects and reconstructing the forces and torques based on such deformations.

Once the forces and torques are sensed, they are conveyed to the effector operator. To this end, a controller with haptic feedback can be used. This is a controller that can exert forces and/or torques on the operator. Examples are force-feedback joysticks, and exoskeletons with haptic capabilities (also known as "sensing limbs"), i.e. frames applied to the operator limbs, which sense the limb positions and can apply forces to the limbs defined by an external computing unit connected to the frame. Alternatively, sensory substitution may be employed, in which the forces are conveyed via sounds, brightness and/or colour of objects, other visual information such as position of dials, height of display bars, shape of indicator diagrams etc.

Similar approaches are employed when haptic information is delivered from the simulated ("virtual") effector such as it happens, for example, in force-feedback- enabled videogames, where virtual hands, limbs and/or effectors operate in a virtual world visualized to the operator through a computer screen, and the mechanical interaction of the virtual hand, limb and/or effector with the virtual world is relayed to the operator via a force-feedback joystick. Another example exists in virtual surgery, where a virtual surgical robot, simulated by appropriate computer software, is emulated on a computer screen while working on a virtual patient. The mechanical interaction between the virtual robot and the virtual patient is relayed to the operator via real haptics-enabled robot controls. In general, known techniques are directed towards replacing the lost, remotised or virtualised haptic perception either by relaying it to the operator via appropriate force- feedback and/or torque-feedback manipulators or by implementing an appropriate sensory substitution mechanism. This enables the tasks to be performed by the effector, and/or increases the accuracy at which they are performed. In other cases, such as videogames, this enhances the operator psychological experience in performing the tasks in the videogame.

Summary of the invention

According to one aspect of the invention, there is provided a system for controlling manipulation of an object by an active effector, such as, for example, a robotic arm, by using a haptic manipulator, the system comprising: sensing means for sensing a parameter of the object being manipulated or of the environment surrounding the object, where such parameter is not the force or the torque exerted by the effector; and means for generating a signal to cause the haptic manipulator to exert a force and/or torque on the manipulator operator, wherein the force and/or torque exerted depends on the sensed parameter.

By detecting one or more n on -force/torque signals, and conveying such signals to the operator in the form of a haptic signal, the operator is able to perform otherwise impossible or difficult tasks, and/or to enhance their performance, such as, for example, by increasing movement accuracy..

The sensing means may be operable to sense one or more parameters not related to haptic perception (i.e. not force or torque), such as, for example, light intensity, optical properties (colour, Raman spectrum, backscattering properties, fluorescence properties, optical absorption, refractive index etc.), temperature, chemical composition. These parameters are advantageously conveyed to the operator via the translation of such quantities into haptic information.

The sensing means may comprise a colour sensor. In this case, the colour of the object being manipulated may be used to determine the level of force and/or torque to be exerted on the operator via the haptic manipulator. The sensing means may comprise a temperature sensor. Measured temperatures may be used to determine the level of force and/or torque to be exerted on the operator via the haptic manipulator.

The sensing means may comprise a refractive index sensor. Measured refractive index may be used to determine the level of force and/or torque to be exerted on the user operator via the haptic manipulator.

The sensing means may comprise a magnetic field sensor. Measured magnetic field may be used to determine the level of force and/or torque to be exerted on the operator via the haptic manipulator.

The sensing means may comprise a fluorescence sensor. Measured fluorescence intensity and/or spectrum may be used to determine the level of force and/or torque to be exerted on the operator via the haptic manipulator.

The sensing means may comprise a backscattering sensor. Measured backscattered intensity and/or spectrum may be used to determine the level of force and/or torque to be exerted on the operator via the haptic manipulator. The sensing means may comprise a sensor of a physiological parameter such as, for example, tissue oxygen saturation, or blood glucose content. Measured values of the physiological parameter may be used to determine the level of force and/or torque to be exerted on the operator via the haptic manipulator.

The sensing means may comprise a pH sensor. Measured pH value may be used to determine the level of force and/or torque to be exerted on the operator via the haptic manipulator. The sensing means may comprise a Raman sensor. The Raman spectrum may be used to determine the level of force and/or torque to be exerted on the operator via the haptic manipulator

The sensing means may comprise a light intensity sensor. Light intensity value may be used to determine the level of force and/or torque to be exerted on the user via the haptic manipulator.

In any of the above cases, the forces and/or torques exerted by the haptic manipulator may transmit to the operator the feeling of a vibrating object.

In any of the above cases, the forces and/or torques exerted by the haptic manipulator may transmit to the operator the feeling of a magnetic attraction.

In any of the above cases, the forces and/or torques exerted by the haptic manipulator may transmit to the operator the feeling of a magnetic repulsion.

In any of the above cases, the forces and/or torques exerted by the haptic manipulator may transmit to the operator the feeling of an elastic response. In any of the above cases, the forces and/or torques exerted by the haptic manipulator may transmit to the operator the feeling of a soft material.

In any of the above cases, the forces and/or torques exerted by the haptic manipulator may transmit to the operator the feeling of a hard material. In any of the above cases, the forces and/or torques exerted by the haptic manipulator may transmit to the operator the feeling of a boundary between materials.

In any of the above cases, the forces and/or torques exerted by the haptic manipulator may transmit to the operator the feeling of attraction towards a certain region of space.

In any of the above cases, the forces and/or torques exerted by the haptic manipulator may transmit to the operator the feeling of repulsion from a certain region of space. Brief Description of the Drawings

Various aspects of the invention will now be described by way of example only and with reference to the accompanying drawings, of which:

Figure 1 (a) is a schematic diagram of a surgical robot for controlling a scalpel for cutting tissue;

Figures 1 (b) and 1 (c) show trajectories of the scalpel of the robot of Figure 1 (a) when cutting through a test phantom;

Figure 2 shows a surgical robotic arm 1 holding an electro-surgical cutting tool

2;

Figure 3 shows neurosurgical robot implanting a stimulation electrode in a patient's brain;

Figure 4 shows a robotic arm 1 holding an arc welding tool 2, and

Figure 5 shows a remote operating vehicle (ROV) operating underwater.

Detailed Description of the Drawings

Figure 1 (a) shows a surgical robotic arm 1 that is carrying a scalpel blade 2. The robotic arm is controlled by a console 5 and copies the movements of a haptic manipulator 6 (a force-feedback 3d joystick). The scalpel blade 2 is co-aligned with an optical fibre 7. Coupled with the fibre 7 is a light source for emitting sensing light. A second fibre 8 co-aligned with the scalpel 2 is provided to detect light reflected or backscattered from tissue. The detected light intensity is acquired by an acquisition system (e.g., an analog-to-digital converter) on the console 5.

In Figure 1 (a), the arm 1 is shown moving the scalpel blade 2 towards a tissue layer 3 between two areas of bulk tissue 4 to create an incision through that layer. The layer 3 is highly reflective/backscattering tissue and the areas of darker bulk tissue 4 are relatively non-reflective. The light source is selected to be highly reflected by the layer 3 and less reflected by the bulk tissue 4.

The console 5 computes and sets on the haptic manipulator 6 a force to be exerted by the manipulator on the operator directed against the manipulator movement direction. The force exerted on the operator is proportional to the manipulator speed and inversely proportional to the acquired light intensity. This means that, for any given manipulator speed, the force is higher when the detected light intensity is lower and that for any given intensity, the force is higher when the speed is higher.

In use, when the scalpel 2 is cutting through the highly reflective layer 3, the intensity of the sensing light reflected is relatively high and so the force exerted on the operator is relatively low. Hence, the operator perceives the scalpel 2 as moving through a medium with low viscosity / density. In contrast, when the direction of the scalpel blade 2 is such that the blade 2 starts to move outside the layer 3, the sensing light is incident on the bulk tissue 4, which has a lower reflectivity than the layer 3. Consequently, the intensity of the sensing light reflected is relatively low and so the force exerted on the operator is relatively high. Hence, the operator perceives the scalpel 2 as moving through a medium with high viscosity / density. In this way, the haptic feedback allows the operator to keep the scalpel 2 moving inside the layer 3, and stops the scalpel when it deviates outside the layer 3.

The system of Figure 1 (a) has been tested using a Denso VP-6242G (Japan) robotic arm to hold two optical fibres and a scalpel. The robotic arm was connected and controlled by a Novint Falcon (USA) haptic manipulator. A layered tissue phantom was used as a test material. The phantom was made of a transparent gel layer (aqueous solution of Laponite (Rockwell Additives, USA) at a concentration of 3.5%) in which optical backscattering was enhanced by the addition of 4% aluminium oxide powder between two gel layers with no added powder, and hence with much lower backscattering.

Figure 1 (b) shows a trajectory of the scalpel as cutting takes place with the support of the light-to-haptic feedback described above. Figure 1 (c) shows a trajectory of the scalpel as cutting takes place without the light based haptic feedback. It can be seen that Figure 1 (b) shows a clear increase in the precision of the cutting path with respect to Figure 1 (c). Figures 1 (b) and 1 (c) show scalpel trajectories as measured, during an experiment, using the principle described above. The dashed lines in the figures represent typical scalpel trajectories seen from the surface into which the scalpel is cutting. The cut is intended to remain within the white area (layer), and to avoid the shaded areas (bulk tissue).

Figure 2 shows a system that has a surgical robotic arm 1 holding an electro-surgical cutting tool 2. The cutting tool 2 is shown cutting though a specimen 4. The arm is connected to a control console 3. The console 3 is also connected to the electro- surgical tool. A sensor (not shown) is connected to a working end of the tool to allow sensing of a parameter of the material being cut. In this case, the sensor could be an optical sensor as described with reference to Figure 1 . Alternatively or additionally, a temperature sensor and/or a pH sensor may be provided. The operator controls the arm by using a haptic manipulator, effectively a force-feedback 3D joystick 6. The surgical tool copies the movement of the 3D joystick. The tool is arranged to help an operator cut through tissue that has, for example, a higher parameter value than bulk tissue.

The console 3 computes and sets on the haptic manipulator 6 a force to be exerted on the operator directed against the manipulator movement direction. The force exerted by the manipulator on the operator is proportional to the manipulator speed and inversely proportional to the sensed parameter. Therefore, for any given manipulator speed, the force is higher when the detected parameter is lower, and for any given parameter value, the force is higher when the speed is higher. Where multiple sensors are used, the sensed parameters from each may be used to determine the force to be exerted. The sensed parameters may be combined. The combination may be a weighted combination.

In use, the operator intends to cut through a tissue layer 5 having a much higher parameter value than the bulk of the specimen. When the cutting tool 2 is cutting through the layer 5, the parameter sensed by electro-surgical tool is relatively high and so the force exerted on the operator is relatively low. Hence, the operator perceives the cutting tool 2 as moving through a medium with low viscosity / density. In contrast, when the direction is such that the cutting tool 2 starts to move outside the layer 5, the parameter sensed is lower than for the layer 5. Consequently, the force exerted on the operator is relatively high. Hence, the operator perceives the cutting tool 2 as moving through a medium with high viscosity / density. In this way, the haptic feedback allows the operator to keep the cutting tool 2 moving inside the layer 5, and stops the operator from moving cutting tool outside the layer 5.

Figure 3 shows a system that has a neurosurgical robot 1 arranged to implant a stimulation electrode 2 in a patient's brain 3. A colour sensor 4 senses the tissue colour through an optical fibre bundle 5 according to methods known in the field of colorimetry. The fibre bundle is inserted coaxially in the electrode, so that the colour is sensed at the electrode tip, while the electrode is being advanced by the robot. The colour sensor is attached to a robot control console 6. The operator controls the robot advancing the electrode-sensor unit by using a haptic manipulator 7.

The console 6 computes and sets on the haptic manipulator 7 a force to be exerted on the operator directed against the manipulator movement direction. The force exerted on the operator is proportional to the manipulator speed and inversely proportional to a measure of similarity between the sensed colour and a pre-determined colour, for example a measure of the closeness of the sensed colour to the colour of blood. For any given manipulator speed, the force is higher when the colour detected is closer to the predetermined colour. For any given intensity, the force is higher when the speed is higher.

In use, when set up to detect the colour of blood, the haptic manipulator 7 exerts on the operator a force which is proportional to the tool speed, directed against the direction of movement, and increasingly higher as the colour sensor detects a colour more and more similar to the colour of blood (e.g., inversely proportional to the distance between the CIE 1931 XYZ colour coordinates of the measured colour and the CIE 1931 XYZ coordinates of blood). In this way, the operator perceives the electrode as advancing through a medium that offers more resistance when there is more blood, and can effectively stop the electrode advancement if the perceived resistance of the manipulator to movement becomes too high due to the close proximity of the electrode tip of a large blood vessel. This avoids the operator hitting large blood vessels with the electrode, which would cause brain haemorrhage. This reduces the risks of the surgical procedure. Figure 4 shows a robotic arm 1 holding an arc welding tool 2. The tool is to be used to welding two butt-jointed metal plates 3 and 4. Again, the arm is connected to a console 5, and copies the movements of a haptic manipulator 6 operated by the welding robot operator. An optical position sensor 7 (e.g., based on a quadrant photodiode and appropriate amplifier, as well known in the state of the art) detects the distance x of the tool from the common surface of the plates (X direction), and the deviation z with respect to the line joining the two plates (Z direction) oriented as in 8. The x and z signals are acquired by the console. Given that x₀ is the optimal distance for the welding, the console computes and sets on the haptic manipulator a force, to be exerted on the operator, proportional to -z in the Z direction and to -(x-x₀) in the X direction. In this way, the operator perceives an elastic force pulling the welding tool towards the optimal distance from the plates, and towards the centre of the optimal cutting line, thus being facilitated in the welding operation and/or increasing its accuracy.

Figure 5 shows an arrangement similar to that of Figure 4, but in this case, the robot arm 1 is replaced by a Remote Operating Vehicle (ROV) 9 operating underwater to weld the two metal plates, e.g. in an oil extraction plant. This shows how the principle is independent of the nature of the robotic end-effector. Other end-effectors could be, for example, endoscopes, endoscopic capsules, devices controlled via a radio, wireless or other wireless data transmission method, devices controlled via a magnetic link.

Various modifications and variations to the described embodiments of the invention will be apparent to those skilled in the art without departing from the scope and spirit of the invention. Although the invention has been described in connection with specific preferred embodiments, it should be understood that the invention as claimed should not be unduly limited to such specific embodiments. For example, although the sensors used in the specific embodiments are an optical sensor, and a colour sensor, a combination of such sensors could be used. Equally, it will be appreciated that many other sensor types could be used either separately or in combination, for example a temperature sensor, a refractive index sensor, a magnetic field sensor, a fluorescence sensor, a backscattering sensor, a physiological parameter sensor, a pH sensor, a Raman sensor, a radio wave sensor (antenna based). As another example, non-force- related parameters related to the manipulation and not measured directly on the object or in close proximity to it may be employed. For example, the forces and/or torques exerted by the haptic manipulator may be determined using the light reflected by the object being manipulated and./or the ambient light in the ambient containing the effector and the object, or by the Raman spectrum measured on the object and/or the sound intensity in the ambient. Indeed, various modifications of the described modes of carrying out the invention obvious to those skilled in the art are covered by the present invention.

Claims

1 . A system for manipulating an object using a haptic manipulator coupled with an active effector, the system comprising: sensing means for sensing a parameter of the object being manipulated and/or of the environment surrounding the object, wherein the sensing means comprise an optical sensor, and means for generating a signal to cause the haptic manipulator to exert a force and/or torque on a user, wherein the force exerted depends on the optically sensed parameter.

2. A system as claimed in claim 1 wherein the optical sensor is operable to detect intensity of light reflected or backscattered by the object.

3. A system as claimed in claim 1 wherein the the optical sensor comprises a colour sensor and the colour detected is used to determine the force and/or torque to be exerted on the user via the haptic manipulator.

4. A system as claimed in claim 1 or claim 2 wherein the sensing means comprise a temperature sensor and the measured temperature is used to determine the force and/or torque to be exerted on the user via the haptic manipulator.

5. A system as claimed in any of the preceding claims wherein the the optical sensor comprises a refractive index sensor and measured refractive index is used to determine the level of force and/or torque to be exerted on the user operator via the haptic manipulator.

6. A system as claimed in any of the preceding claims wherein the sensing means comprise a magnetic field sensor and measured magnetic field is used to determine the level of force and/or torque to be exerted on the operator via the haptic manipulator.

7. A system as claimed in any of the preceding claims wherein the optical sensor comprises a fluorescence sensor and measured fluorescence intensity and/or spectrum may be used to determine the level of force and/or torque to be exerted on the operator via the haptic manipulator.

8. A system as claimed in any of the preceding claims wherein the sensing means comprise a backscattering sensor and measured backscattered intensity and/or spectrum is used to determine the level of force and/or torque to be exerted on the operator via the haptic manipulator.

9. A system as claimed in any of the preceding claims wherein the sensing means comprise a sensor of a physiological parameter such as, for example, tissue oxygen saturation, or blood glucose content and measured values of the physiological parameter are used to determine the level of force and/or torque to be exerted on the operator via the haptic manipulator.

10. A system as claimed in any of the preceding claims wherein the sensing means comprise a pH sensor and the measured pH value is used to determine the force and/or torque to be exerted on the user via the haptic manipulator.

1 1 . A system as claimed in any of the preceding claims wherein the optical sensor comprises a Raman sensor and the Raman signal is used to determine the force and/or torque to be exerted on the user via the haptic manipulator.

12. A system for manipulating an object using a haptic manipulator coupled with an active effector, the system comprising: sensing means for sensing a parameter of the object being manipulated and/or of the environment surrounding the object, wherein the sensing means comprise a temperature sensor, and means for generating a signal to cause the haptic manipulator to exert a force and/or torque on a user, wherein the force exerted depends on the sensed temperature.

13. A system for manipulating an object using a haptic manipulator coupled with an active effector, the system comprising: sensing means for sensing a parameter of the object being manipulated and/or of the environment surrounding the object, wherein the sensing means comprise a PH sensor, and means for generating a signal to cause the haptic manipulator to exert a force and/or torque on a user, wherein the force exerted depends on the sensed PH.

14. A system as claimed in any of the preceding claims wherein the force and/or torque exerted by the haptic manipulator transmits to the operator at least one of: the feeling of a vibrating object or the feeling of a magnetic attraction; the feeling of a magnetic repulsion; the feeling of an elastic response; the feeling of a soft material; the feeling of a hard material; the feeling of a boundary between materials; the feeling of attraction towards a certain region of space; the feeling of repulsion from a certain region of space.

15. A system as claimed in any of the preceding claims wherein one or more components are virtual, except the haptic manipulator, which is real.

16. A system as claimed in any of the preceding claims wherein the effector is a robotic arm or a remotely operated vehicle or an endoscope or an endoscopic capsule.

17. A system as claimed in any of the preceding claims where the effector is operated via a radio or other wireless data transmission method

18. A system as claimed in any of the preceding claims where the effector is operated via a magnetic link.