EP3909515A1

EP3909515A1 - A robot system arranged for autonomously obtaining biological surface samples from a patient

Info

Publication number: EP3909515A1
Application number: EP20174694.8A
Authority: EP
Inventors: Thiusius Rajeeth Savarimuthu; Henrik Gordon Petersen; Nicolai Iversen; Aljaz KRAMBERGER; Iñigo Iturrate San Juan; Jakob WILM; Michael Kjær Schmidt; Miha Denisa; Anders Prier Lindvig; Anders Glent Buch; Trine Straarup Winther; Christoffer SLOTH
Original assignee: Syddansk Universitet
Current assignee: Syddansk Universitet
Priority date: 2020-05-14
Filing date: 2020-05-14
Publication date: 2021-11-17
Also published as: WO2021228813A1

Abstract

The invention relates to a method for controlling an apparatus (100) for taking biological surface samples of a test surface of a patient, the apparatus comprise a motorized multi-degree of freedom motion system (101) which is connectable with a tool holder (102), wherein the tool holder is configured to hold a sample probe (103) for obtaining the biological sample, a vision system arranged to provide image data, and a control system for controlling the motorized multi-degree of freedom motion system based on the image data, the method comprises setting the sample probe in an initial position, obtaining an image of the test surface during an identification period, determining probe motion data based on the image data of the image, controlling the motorized multi-degree of freedom motion system to move the sample probe until contact with the test surface is determined, and controlling the motorized multi-degree of freedom motion system to move the sample probe while maintaining contact with the test surface based on the probe motion data.

Description

FIELD OF THE INVENTION

The present invention relates to an apparatus and a method for obtaining a biological surface sample from a patient.

BACKGROUND OF THE INVENTION

The spread of virus through the global populous is recurrent in history, the most resent being the coronavirus which emerged in the Asian continent in late 2019 where it was identified in early 2020 and was later named COVID-19. The virus spread rapidly to other areas of the word in the following period. The COVID-19 outbreak was soon thereafter declared for a pandemic.
Previously in history, the world has witnessed other virus based pandemics, such as the "Spanish flu" pandemic of 1918, the avian Influenza A H5N1 virus ("Bird flu") epidemic of 2004, and the H 1N 1 viral strain ("Swine flu") pandemic of 2009. These pandemics cause global problems, both economically and medically. Besides these virus-based pandemics, influenza virus epidemics are frequently recurring.
In order to determine if a human being is infected with a virus, such as for instance the COVID-19, heath care staff perform a throat swab for collecting a sample swab from the throat. The aim of the swabbing procedure is to collect a high number of virus particles from the patient's throat. There is evidence that sampling should occur from the posterior parts of the oral cavity and the posterior nasopharyngeal wall of the mouth of the patient.
The inhalation of air contaminated by harmful virus and/or other micro-organisms is a common route for infection of human beings, particularly health care staff and others caused to work with infected humans or animals. Air exhaled by infected patients is a source of contamination. Masks are used as a barrier to prevent species-to-species transmission of the virus. However, especially health care staff taking throat samples from patients are particularly exposed and although they are wearing masks and similar barriers, there is a risk of cross contamination between heath care staff and patients in the sampling taking operation.
In order to control a pandemic outbreak a vast amount of swab tests must be taken. This increases the risk of contamination but it can be reduced by using protective clothes, masks, protective eyewear and the like. This however, is costly as all of these protective measures are for single use or must be sterilised between each use.
Furthermore, the health system may be under a great pressure during a virus outbreak with the risk that an insufficient number of health care workers are available for performing the sample takings of patients in order to be able to detect infected patients to enable control of the spread of the virus.
Accordingly, it is a problem that the manual sample taking may lead to cross contamination and thereby increase spreading of the virus. It is another problem, that the health care system is not geared to of perform the desired number of sample takings.

SUMMARY OF THE INVENTION

It is an object of the invention to solve the above-mentioned problems with cross contamination and limited capacity of the health system to perform a sufficient number of sample takings.
In a first aspect of the invention there is provided a method for controlling an apparatus for taking biological surface samples of a test surface of a patient, the apparatus comprises

a motorized multi-degree of freedom motion system which is connectable with a tool holder, wherein the tool holder is configured to hold a sample probe for obtaining the biological sample,
a vision system arranged to provide image data,
a control system for controlling the motorized multi-degree of freedom motion system based on the image data, the method comprises:
- setting the sample probe in an initial position,
- obtaining an image of the test surface during an identification period,
- determining probe motion data based on the image data of the image,
- controlling the motorized multi-degree of freedom motion system to move the sample probe until contact with the test surface is determined, and
- controlling the motorized multi-degree of freedom motion system to move the sample probe while maintaining contact with the test surface based on the probe motion data.

The apparatus may be a robot system, i.e. a system comprising a controllable handling system such as a motorized multi-degree of freedom motion system.
Hereby an automated method for taking biological samples is provided eliminating or at least substantially reducing the risk of contamination of the people involved in the operation, i.e. both the patient and the health care personnel.
Furthermore, the method is not limited with respect to the sample taking capacity in the same way as manual sample taking performed by health care staff. E.g. the number of robot system can be increased, they can work day-and-night, and cannot be prevented from working, like the health care staff which can be prevented from working due to infections or suspected infections.
Accordingly, the robot system may be particularly advantageous for detecting infected patients or people in general in case of a pandemic virus outbreak wherein the capacity of the healthcare workers is insufficient for testing a high number of patients. However, the automated capability of the robot system can also find use in other situations, e.g. during normal seasonal virus spreading and other virus outbreaks.
Advantageously, the probe motion data such as a trajectory for the motion of the sample probe is obtained initially based on imaging data of a region containing the test surface so that the probe motion data is pre-determined before starting moving the sample probe towards the test surface which reduces risks of performing the sample taking incorrectly.
The test surface of the patient may be a surface of an intra-cavity of the patient, such as a throat, an oral cavity, a rectum, an auditory canal, a nasal cavity, a vagina, or other. Alternatively, the test surface may be a skin surface, an eye surface or other.
According to an embodiment, the method comprises determining a location of the test surface based on the image data and pre-stored data obtained based on training images, wherein the probe motion data is determined based at least on the location of test surface.
The probe motion data such as a trajectory may be determined when the location of the test surface is known. The location may be the position in a reference coordinate system such as reference coordinate system of the motorized multi-degree of freedom motion system. The probe motion data may be determined based on predetermined trajectories, e.g. a circle with a given radius, or the probe motion data may further be determined based on a determined extent of the test surface.
According to an embodiment, wherein the controlling of the motorized multi-degree of freedom motion system to move the sample probe until contact with the test surface is determined, the determination of contact is based on images obtained during the motion of the sample probe.
Advantageously, the use of image data obtained during the motion to the test surface may prevent contact with other body-parts and thereby reducing the risk of contamination of the sample probe.
According to an embodiment, the controlling of the motorized multi-degree of freedom motion system to move the sample probe until contact with the test surface is determined based on force data indicative of a force acting on the sample probe, and/or the controlling the motorized multi-degree of freedom motion system to move the sample probe while maintaining contact with the test surface is based on the probe motion data and the force data.
Advantageously, controlling of the motorized multi-degree of freedom motion system to move the sample probe until contact with the test surface is determined based on force data which provides a reliable method for determining contact.
According to an embodiment, wherein the controlling of the motorized multi-degree of freedom motion system to move the sample probe until contact with the test surface is determined, the controlling is further subject to a calculated distance between the sample probe and the test surface.
Advantageously, when the distance to test surface is large, the sample probe may be moved with a relative high velocity, and when the distance is low, the velocity may be reduced.
According to an embodiment, the force data indicative of a force acting on the sample probe is used for determining an unintended contact with biological surfaces of the patient other than the test surface.
Advantageously, force data may be generated by unintended contact and e.g. used for controlling the motion of the sample probe.
According to an embodiment, controlling the motorized multi-degree of freedom motion system to move the sample probe while maintaining contact with the test surface is further based on the image data obtained during the motion of the sample probe.
Advantageously, image data obtained during the sample taking may be used e.g. to avoid contact with a moving tongue, to compensate patient motion, or to improve trajectory control.
According to an embodiment, the method comprises providing visual feedback to the user during the identification period, and optionally during a period where the probe motion data is determined.
Advantageously, the visual feedback may be used to guide the patient to improve the quality of obtained images.
Furthermore, visual feedback may be provided to the user during a period of moving the sample probe away from the test surface of the patient or out of the intra-cavity, thereby limiting the risk of unintended contact with other body surfaces.
According to an embodiment, when the initial position is obtained, the method further comprises initiating an autonomous mode wherein the position of the tool holder is controlled based on the image data.
Advantageously, when the tool holder is connected to the vision system or the tool holder comprises the vision system, the tool holder will be moved, dependent on the image data, to obtain a more optimal viewing angle.
According to an embodiment, the method comprises controlling the motorized multi-degree of freedom motion system to move the sample probe away from the test surface and/or out of the intra-cavity.
According to an embodiment, the controlling for moving the sample probe away from the test surface and/or out of the intra-cavity is based on the image data obtained during the motion of the sample probe and/or the force data.
Advantageously, image data may be used during retraction to avoid unintended contact with other body surfaces.
According to an embodiment, subsequent to moving the sample probe away from the test surface of the patient or out of the intra-cavity, the method comprises controlling the motorized multi-degree of freedom motion system to move the sample probe into a sample container.
Advantageously, no human interaction is required for inserting the sample probe in the sample container.
According to an embodiment, subsequently to move the sample probe into the sample container, the method further comprises controlling the motorized multi-degree of freedom motion system to move the sample probe in a direction transverse to a longitudinal axis of the sample probe so as to break or snap an end-portion of the sample probe into the sample container.
According to an embodiment, the method further comprises performing a safety procedure, such as prohibiting motion of the tool holder, dependent on an alarm signal generated by a security means dependent on a force acting on the sample probe.
Advantageously, if a force acting on the sample probe increases to a level which is above an acceptable level of the patient, e.g. a level which is comfortable for the patient, certain actions may be initiated such as stopping further motion of the sample probe or tool holder.
According to an embodiment, setting the sample probe in an initial position, comprises initially setting the motorized multi-degree of freedom motion system in a user guided mode wherein the user can position the sample probe in the initial position.
Advantageously, in some situations a user guide mode may preferred, e.g. where the patient feels uncomfortable with the initial part autonomous motion of the tool holder.
According to an embodiment, the method further comprises controlling the motorized multi-degree of freedom motion system to rotate the sample probe around a longitudinal axis thereof and repeat or continue the moving the sample probe with contact to the test surface while a different portion of a sample-end contacts the test surface.
Advantageously, by rotating the sample probe such as a swab, it is possible to obtain more sample material and/or obtain sample material from a larger area.
A second aspect of the invention relates to an apparatus for taking biological surface samples of a patient, the apparatus comprises

a motorized multi-degree of freedom motion system which is connectable with a tool holder, wherein the tool holder is configured to hold a sample probe for obtaining the biological sample,
a vision system arranged to provide image data,
a control system for controlling the motorized multi-degree of freedom motion system based on the image data according to the first aspect.

According to an embodiment, the force adjustment system comprises a spring device arranged to displace in response to a force acting on the sample probe.
In general, the spring device may be an elastic device arranged to displace elastically, e.g. axially along the coaxial, or substantially coaxial, longitudinal axes of the tool holder and the sample probe, to allow the probe-end to displace axially, e.g. to adapt to surface height variations or variations in the controlled position of the tool holder, in order to reduce variations in the axial force acting on the probe-end.
Advantageously, the spring device may ensure that forces levels acting on the test surface do not exceed a maximum force given by the spring element.
According to an embodiment, the force adjustment system comprises a force feedback system arranged to provide the force data, wherein the control system is arranged for controlling the motorized multi-degree of freedom motion system dependent on the force data.
According to an embodiment, the apparatus further comprises a security means arranged to generate an alarm signal dependent on the force data.
Advantageously, if the force acting on the sample probe exceeds a maximum force, the motorized multi-degree of freedom motion system may be controlled dependent on the alarm signal to perform a safety action.
A third aspect of the invention relates to a computer program product comprising software code adapted to control an apparatus when executed on a data processing system, the computer program product being adapted to perform the method of the first aspect.
In general, the various aspects and embodiments of the invention may be combined and coupled in any way possible within the scope of the invention. These and other aspects, features and/or advantages of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will be described, by way of example only, with reference to the drawings, in which

Fig. 1A shows a robotic sampling apparatus for obtaining a biological sample from a patient,
Fig. 1B shows an example of the tool holder and the sample probe,
Fig. 2 schematically illustrates a robot system which comprises the vision system,
Fig. 3 shows an example of the automatic sample taking process,
Fig. 4A shows an example of the test surface in an intra-cavity,
Fig. 4B illustrates an example of determining the probe motion data,
Fig. 5 shows an example of a process for determining the probe motion data
Fig. 6 shows an example of a process for controlling the motorized multi-degree of freedom motion system to move the sample probe until contact with the test surface, and
Fig. 7 shows an example of a process for controlling the motorized multi-degree of freedom motion system to move the sample probe while maintaining contact.

DETAILED DESCRIPTION

Fig. 1A shows a robotic sampling apparatus 199 for obtaining a biological sample from a patient. The robotic sampling apparatus 199 comprises a preparation compartment 181 for preparing a test kit including a sample probe 103 for obtaining the biological sample. The robotic sampling apparatus 199 further comprises a sanitation compartment 182, which is adapted to receive the test kit from the preparation compartment 181 via a transfer arrangement 183 from the preparation compartment 181. The preparation compartment 181 and the sanitation compartment 182 are separated by a wall which comprises an opening 183 that can be closed by a closing means like a door. The sampling apparatus 199 also comprises an opening leading to a patient area (not shown - but located outside the apparatus 199), wherein the patient area is for providing the patient in a predetermined position which is accessible to a second handling system which is arranged to move said test kit to an active position. The test kit comprises the sample probe 103 and a sample container for storing a part of the sample probe 103 containing the biological sample from the patient. According to an embodiment, the second handling system which may in the form of a motorized multi-degree of freedom motion system 101, such as a robot, is further capable of autonomously obtaining the biological sample from a patient and returning the biological sample to the sample container.
Hereby, an automated taking of biological samples is provided eliminating or at least substantially reducing the risk of contamination of the people involved in the operation, i.e. both the patient and the health care staff.
Advantageously, the preparation compartment may be environmentally controlled and a sterile environment is kept therein. This eliminates any risk of contaminating the test kit objects before the objects are used for taking a biological sample.
The robotic sampling apparatus 199 comprises a robot system 100 for taking biological samples, e.g. surface samples of a test surface of a patient such as a test surface within an intra-cavity, or cavity, of a patient. The robot system comprises a motorized multi-degree of freedom motion system 101 which is connectable with a tool holder 102, wherein the tool holder is configured to hold a sample probe 103 for obtaining the biological sample.
Specifically, the robot system 100 may be configured for taking samples from the oral cavity or throat of a patient, e.g. for the purpose of identifying patients infected by a virus such as the COVID-19 virus or other viruses having a high potential of infecting a high percentage of the population.
Examples of intra-cavities of a patient comprises the rectum, the urinary tract and nostrils. Examples of test surfaces which are not specifically located within a cavity comprises skin surfaces, the eye region, the tongue and other. Examples referring to test surfaces of intra-cavities are not limited as such to the intra-cavities but may apply equally to other test surfaces of the patient.
Fig. 1B shows an example of the tool holder 102 and the sample probe 103 inserted in the tool holder. The sample probe 103 may be a swab or other probe arranged to obtain surface samples of test surface of a human. The tool holder 102 further comprises a vision system 110 arranged for obtaining images of the intra-cavity.
The vision system 110 may comprise one or more visible light or IR imaging cameras, Lidar sensors, flight-of-time sensors, laser scanners, ultrasound probes and other sensors capable of determining orientation, dimensions, direction, distance, image characteristics or other characteristics of an object such as a surface of the human intra-cavity, e.g. relative to the vision system 110 or the probe-end 103a.
Accordingly, the image obtained by the vision system may an image, or images, of any one or more of the sensors of the vision system described above.
Fig. 2 schematically illustrates the robot system 100 which comprises the vision system 110. The vision system may be arranged to move with a part of the motorized multi-degree of freedom motion system 101 such as an end-effector thereof, e.g. the tool holder 102. Alternatively or additionally, the vision system may comprise a stationary vision system which is arranged separated from the motorized multi-degree of freedom motion system 101.
The robot system 100 further comprises a force adjustment system for obtaining forces and/or torques acting on the tool holder or acting on the sample probe 102. The force adjustment system may be configured in different ways as described below.
The robot system 100 further comprises a control system 220 for controlling the motorized multi-degree of freedom motion system 101 based on image data 222 from the vision system 110 and force data 223 from the force adjustment system.
The motorized multi-degree of freedom motion system 101 may comprise a linkage system provided with electric motors for rotating elements of the linkage system relative to each other. Accordingly, the multi-degree of freedom motion system 101 may be an industrial robot.
The linkage system may as illustrated comprise rigid elements 211 and motion actuators 212 capable of rotating and/or displacing the rigid elements 211 or other elements such as the tool holder 102. The rotation provided by one actuator 212 may be about any one or more axes such as an axis perpendicular or parallel to the longitudinal extension of a rigid element 211.
The multi-degree of freedom motion system 101 would have at least two degrees of freedom, preferably at least three degrees of freedom.
The force adjustment system may be configured as a force feedback system comprising a sensor function capable of determining the forces and/or torques acting on the tool holder 102 or sample probe 103. In this example, the force data 223 comprises data corresponding to said forces and/or torques. The sensor function may be embodied by a force sensor and/or a torque sensor 203. The force and/or torque sensors may be located between the tool holder and a rigid element 211 or a motion actuator 212, alternatively, the force-torque sensor may be arranged between the tool holder 102 and the sample probe 103. Alternatively, or additionally, the sensor function may be embodied by a function configured to estimate forces and/or torques acting on the tool holder 102 or sample probe 103, e.g. based on motor torques or motor currents.
The force feedback system further comprises a feedback control system for adjusting the operation of the motors, such as adjusting motor speed or torque, based on the feedback of the determined forces and/or torques.
Alternatively or additionally, the force adjustment system is configured with a spring device arranged to displace in response to a force acting on the sample probe. The spring device may be arranged in the tool holder so that the sample probe 103 is elastically connected to the tool holder 102 to allow a certain displacement of the sample probe, e.g. along the longitudinal direction of the sample probe.
The spring device may be configured as a passive spring such as a helical spring, an air spring or other.
In the configuration of the force adjustment system with a spring device, the force adjustment system may further comprise a sensor, such as a force sensor like a strain gauge sensor or displacement sensor like an optical distance sensor, which is configured to determine sensor data being indicative of the distance between the the sample probe 103 and the tool holder 102, or the compression/elongation of the spring device. In this example, the force data 223 are based said sensor output and may be used equivalently with the force data from the other alternative of the force adjustment system.
Accordingly, the force data indicative of a force and/or torque acting on the sample probe may be provided by the sensor function of the force feedback system and/or by the sensor of the spring configured force adjustment system.
The robot system 100 further comprises a control system 220 arranged to control the motion of the tool holder 102 and the contact force applied by the probe-end 103a on the external object by determination of control input 221 to the motorized motion system 101.
The control input 221 may be in the form of control parameters for each of the actuators 212, e.g. control parameters which are used as a positional reference for controlling motion of the actuators 212. The control input 221 could have other formats, e.g. as a reference motion of the tool holder 102 which is received by a controller of the motorized multi-degree of motion system 101 and used by this controller for controlling the actuators 212. Thus, the motorized motion system 101 may be controlled by forward or inverse kinematics. Accordingly, the control input 221 may describe a desired trajectory of the probe-end 103a in terms of specific control parameters for the actuators 212 or by the desired trajectory of the probe-end 103a itself.
Fig. 2 schematically illustrates a patient reference arrangement 290 such as a head support which has a fixed position relative to the stationary part of the motorized motion system 101. For example, for the purpose of taking samples from the oral cavity or throat, the patient initially positions her head in a position supported by the head support. In this way, the vision system 203 can use the known position of the patient reference arrangement 290 or the related approximate reference position of the mouth or the, e.g. open mouth, to start performing a recognition and identification of the desired test surface of the intra-cavity, e.g. based on images recorded by the vision system and pre-stored intra-cavity image features defining the desired test surface.
In connection with obtaining images of the oral cavity or throat, guiding systems such as a tongue guiding tool may be used for pushing the tongue down to allow a free passage for imaging purposes. In an example, the robot system 100 is configured to position the guiding system automatically, e.g. based on image data from the vision system 110.
The robot system may further comprise a display 150 for providing a visual feedback to the user, e.g. the patient, during the identification period where images are obtained for determining the probe motion data 493 and/or during a period where the probe motion data is determined. Experimentation has found that users are much better able to adapt to imaging requirements necessary for obtaining images of the desired test surface 490, e.g. to lower their tongue when visual feedback is presented to them. Visual feedback may be used during the image recording, but could also be used during determination of probe motion data, particularly if the probe motion data are determined real time while images are recorded.
Additionally, the visual feedback and/or audible feedback may be given to the user during a period of moving the sample probe away from the test surface of the patient or out of the intra-cavity. The feedback could be in form on instructions to not move, or to hold the breath, alternatively or additionally in the form of images of the test surface region or intra-cavity, e.g. to maintain the tongue position.
Fig. 3 shows an example of the automatic sample taking process involving:

initial steps 301 for preparing a test kit of the sample probe 103 and a sample container such as a vial for storing the probe-end 103a after the sample has been taken from the patient,
obtaining images, in step 302, of the test surface or the intra-cavity during an identification period,
determining probe motion data based on the image, in step 303,
moving the sample probe until contact with the test surface, in step 304,
moving the sample probe while maintaining contact with the test surface, e.g. to perform a swab in step 305, and
subsequent steps comprising one or more of removing the probe- end 103a, 306, e.g. by breaking the probe-end 103a off the probe to store the probe-end 103a in the container, a step of securing the biological sample in the container e.g. by closing the container such as by attaching a lid to the container 308, possibly a step of discharging the tool holder (probe tool) 102, 307, e.g. for sanitizing the tool holder, and possibly a step of discharging the tool, e.g. lid tool, used for securing the biological sample 309.

Fig. 3 shows additional steps such the step of enabling compliant control. Such a step may be used for setting the sample probe in an initial position by initially setting the robot device in a user guided mode wherein the user can position the sample probe in the initial position. The user guided mode may be achieved by setting the control system 220 in a compliant control mode which enables the user to freely move the tool holder 102.
A further step (not shown) comprises controlling the motorized multi-degree of freedom motion system 101 for moving the sample probe away from the test surface and/or out of the intra-cavity. The motion away from the test surface may be based on image data obtained during the motion of the sample probe and/or the force data, e.g. to avoid contacting other structures or a body surface with the sample-end 103a.
One or more of the steps 301-309 and other step illustrated in Fig. 3 may be omitted and/or other steps may be included.
An initial step according to an embodiment comprises setting the sample probe 103 or the tool holder 102 in an initial position. The initial position may be a pre-determined position which may be fixed relative to the patient reference arrangement 290, or may be determined based on image recognition of structures such as the patient reference arrangement 290 or characteristics of the patient, such as the mouth, an open mouth, generic facial characteristics. In this way, the sample probe 103 can be arranged in front of a test surface or an intra-cavity opening of the patient.
After the initial position has been obtained, the vision system 110 is controlled to obtain one or more images of the test surface during an identification period.
Alternatively to the automatic setting of the motorized multi-degree of freedom motion system 101 to achieve the initial position, the user guided mode (described elsewhere) may be used to achieve the initial position.
Thus, the vision system 110 is configured to obtain images of the intra-cavity during the identification period, and/or 3-dimensional data representing positions of at least a portion of the intra-cavity biological surface.
In an example, the vision system is configured to obtain 3-dimensional position data of locations, i.e. to achieve automated localization in the 3-dimensional space of suitable sampling locations.
Since, in this example, the movement of the sample probe on the intra-cavity biological surface is not solely based on 3-dimensional position data from the vision system, but is additional based on data from the force feedback system 203, i.e. based on admittance control configured to keep a suitable pressure against patient's tissue at the intra-cavity surface, the accuracy of the vision system need not be very high. For example, an accuracy in the range from 0.5 to 5 mm, such as in the range from 1 to 3 mm, e.g. around 2 mm may be sufficient.
As mentioned above, the image sensor of the vision system 110 may be placed at a distance from the patient. An advantage of mounting the camera at a distance is less contamination, e.g. due to droplets possible coughs of the patient, and easier cleaning and disinfection.
An advantage of the tool-mounted image sensor is that the robot can be used to align the camera view to each individual patient ensuring the most optimal view.
The vision system 110 may comprise 2D image sensors and a processor for determining depth information, e.g. of the test surface, based on assisted passive stereo reconstruction and other methods. Additionally or alternatively, the vision system 110 may comprise 3D sensors such as active stereo, TOF sensors, Lidar sensors and compact laser scanners.
The location of the test surface, e.g. the position in 3D space relative to a stationary part of the motorized multi-degree of freedom motion system 101, optionally the 3D shape of test surface and optionally the extension of the test surface may be determined by segmentation methods applied on the image data obtained from image sensors of the vision system 110.
As an example, the segmentation may comprise the steps: 1) semantic segmentation of a current sensor reading comprising intensity and depth data into foreground and background. The foreground is represented by all areas which are suitable for taking the biological surface sample, e.g. by swabbing, and which can and should be in contact with the probe-end 103a. 2) Determining the probe motion data, e.g. motion data defining the trajectory that the probe-end 103a should follow to take the sample based on the segmentation.
High inter-subject variance in throat anatomy and the lack of man-made, salient image features may render classical computer vision techniques, such as pre deep learning methods, unsuitable for the segmentation task. Consequently, a modified U-Net neural network, i.e. a convolutional neural network, has shown to be suitable for determining the test surface. This convolutional neural network architecture was specifically designed for semantic segmentation on the biomedical images containing the test surface based on training examples.
In an example shown in Fig. 4A, the segmented foreground area includes the tonsillar area 401, the pharyngopalatine arch 402 and the posterior nasopharynx 403, but excludes the uvula 404. Accordingly, Fig. 4A shows an example of the test surface 490 which comprises the above-defined foreground area.
Experimentation showed that good performance was achieved with a relatively slim neural network comprising 4 network layers and 32 filters per layer, two convolution levels, and with batch normalization. The general heuristic for choosing these parameters was based on conditions for keeping the network as slim as possible to avoid overfitting, while reaching the narrowest possible gap between training and test accuracy.
For training data, approximately 300 intensity images from 7 different subjects were annotated manually. These images were divided into training and test data sets. Due to this small population size, all participants were represented in both training and test data, but from different time points, which necessarily overestimated accuracy and generalization. While early training data included a very large variation in image sensor pose, and included data from a different, but similar sensors, better performance was achieved by limiting input data to actual swabbing scenario, in which the sensor is always positioned at the same approximate distance from the patients mouth opening when segmentation results are needed.
The U-Net was trained without transferred/pretrained weights in 10 epochs with 500 steps and a batch size of 2. These parameters were determined to be suitable by trial and error. A test and training accuracy above 99% were achieved with the settings. Fig. 4A shows a predicted result from the based on a trained network.
In this way, characteristics of the test surface may be determined based on actual image data and pre-stored data obtained from test images, such as image data obtained based on images of the intra-cavity or other body region containing the test surface. The characteristics of the test surface comprises the location of the test surface, e.g. relative to a reference coordinate system or stationary part of the motorized multi-degree of freedom motion system 101. Further, the characteristics of the test surface may comprise the 3D shape of the test surface and/or the extension of the test surface.
The pre-stored data may be embodied by the parameters of a trained neural network or other trained algorithm configured to recognize a desired test surface based on training images. The probe motion data is determined based on characteristics of the test surface 490, such as the location and the extension of the test surface, e.g. the extension defined by the outer boundary of the segmented foreground area shown in Fig. 4A.
The determination of the test surface 490 based on the segmentation method or other method may be performed consecutively, e.g. immediately after an image has been obtained. In this way, it can be determined if the quality of the determined test surface is sufficient, and if not, the image recording process can be continued until the quality of the determined test surface is satisfactorily. For example, if the quality is not satisfactorily, the orientation of the vision system may be adjusted to improve the image quality and thereby the quality of the test surface 490.
The probe motion data is determined based on the segmentation result, based on the following example or by other methods. The semantic segmentations may be represented by matrices with the same dimensions as the input images and with values of each pixels ranging from 0 to 1. The pixel values represent the probability of being part of the intended test area 490. The matrices may be processed based on simple morphology based on network output binarized with a threshold of 0.5. The so-obtained binary blobs resulting from the threshold comparisons may be checked for a necessary minimum area and it is confirmed that only one dominating blob was found. Within this dominating foreground blob, the left and right boundary points are determined, and four equidistant sampling points 491 are sampled in between as illustrated in Fig. 4A. Clearly any other number of sampling points 491 equal to or above 2 can be used, as well as other methods for processing the segmented images for determining the probe motion data 493 (Fig. 4B).
According to other examples, the probe motion data 493 may be found by other methods, which do not require an initial image segmentation. For example, probe motion data 493 may be determined directly from images of the vision system, not necessarily intensity images, e.g. based on artificial intelligence methods, e.g. neural network based, which are capable of processing image data received from the vision system, into probe motion data 493 such as trajectories for the probe-end 103a to follow.
Fig. 4B illustrates an example of determining the probe motion data 493, here a probe-end trajectory, based on the image data.
As a pre-requirement to performing the swab, the vision system 110 is queried in order to determine where in the workspace of the motorized multi-degree of freedom motion system 101, test surface 490 or sampling points 491 of the test surface 490 are located, e.g. sampling points 491 at a surface of the back of the throat as shown in Fig. 4A. The location of the test surface 490 itself may be characterized by a single point of the test surface.
Once the vision system 110 has scanned the patient's mouth it determines four sampling points 491 that are then sent to the control system 220. The current pose of the motorized multi-degree of freedom motion system 101, e.g. the pose of the tool holder 102, is obtained and saved as the image pose, as the tool holder 102 is assumed to not have moved since the one or more images of the test surface 490 was obtained during the identification period.
As the sampling points 491 do not contain orientation information, and as their positions are provided in the reference coordinate system of the tool holder 102, they must be transformed and an orientation must be calculated, such that they can be inputted to the control system 220. Fig. 4B shows an example for determining the orientation of the sampling points 491. Once the full swab poses have been calculated by adding an orientational component to the sampling points, an approach pose is calculated by offsetting the swab poses along the z-axis. Both the swab and approach poses are then transformed to the reference coordinate system of the motorized multi-degree of freedom motion system 101. The probe motion data 493, i.e. a swab or probe-end trajectory, are calculated based on the sampling point 491, i.e. test surface points, e.g. by linearly interpolating between the sampling points 491 by use of the swab poses. The swab poses are defined by the surface normal (z-axis) and the surface tangents (x and y-axes).
Since the motorized multi-degree of freedom motion system 101 will be controlled in task space, the interpolated segment and the approach and retraction relative to the test surface 291 may be checked for proximity to singularities.
If any of the path segments of the is determined to come close to a singularity, the software running the implantation of the method, may exit and let the user know in which segment the singularity would have occurred (e.g. approach, swab, retraction). For example, the user may be prompted to find a different robot configuration to perform the swabbing test of the patient.
Fig. 5 shows an example of a process for determining the probe motion data 493 based determined sampling points 491. In Fig. 5, sampling points 491 are referred to as swab points.
The process for taking the biological surface samples of the test surface 490 continues with moving the sample probe 103 until contact with the test surface 490 is achieved as determined based on the force data and subsequently, moving the sample probe while maintaining contact with the test surface based on probe motion data and force data.
The determination of the probe motion data 491 may be computed offline before physically controlling the motorized multi-degree of freedom motion system 101. Once that is complete, the rest of the sample taking process may performed real-time.
As an additional safety measure in real-time processing, every computed target set-point can be checked for singularities as described above prior to being sent to the control system of the motorized multi-degree of freedom motion system 101.
In this way, moving of the sample probe 103 with high accelerations due to singularities, e.g. near a test surface 490, can be prevented.
As a first step, the motorized multi-degree of freedom motion system 101 is controlled to move the sample probe until contact with the test surface is determined. This step may involve aligning the motorized multi-degree of freedom motion system 101 with one of the swab poses (see def. above) before moving the sample probe 103 to obtain contact with the test surface 490.
Fig. 6 shows an example of a process for controlling the motorized multi-degree of freedom motion system 101 to move the sample probe until contact with the test surface 490 is determined based the force data 223.
As shown in Fig. 4B, the tool holder 102 is first aligned with the approach pose of the swab pose of sampling point 3. The tool holder 103 is moved forward in speed control along the z-axis of the tool holder 103 towards sampling point 3. While the tool holder 102 is moving, the readings of force data from the force adjustment system are continuously monitored to detect contact.
Contact detection may be performed by integrating the force data so that a possible integrated contact value which exceeds a contact threshold indicates contact. Contact detection may also be based on instantaneous values exceeding a threshold value. Thus, the force data may be used for determining a non-intended contact with biological surfaces of the patient other than the test surface. In Fig. 6, the contact detection method is referred to as the "CUSUM".
While the robot is still far from sampling point 3, i.e. when contact is not expected, a relative high contact threshold may be used, i.e. the contact threshold may be set for a lower sensitivity. In Fig. 6, the lower sensitivity is referred to as low sensitivity CUSUM parameters. Anyhow, the contact threshold should still enable determination of contact with the test surface 490, e.g. in case the depth information from the vision system 110 is imprecise.
Once the distance between the probe-end 103a and sampling point 3 is lower than a specified threshold, d (see Fig. 4B), the tool holder 102 is decelerated in anticipation of contact.
Accordingly, the control of the motion of tool holder 102 may be determined subject to a calculated distance between the sample probe and the test surface 490.
When the tool holder 102 has finished decelerating and reached a constant lower speed, the contact threshold may be set for a higher sensitivity, e.g. to a lower value (High sensitivity CUSUM parameters in Fig. 6). The so-far determined contact value, e.g. the CUSUM state, is reset. The waiting for the tool holder 102 to fully decelerate before the contact threshold is set for a higher sensitivity may advantageously avoid false contact determinations, i.e. due to the force data generated by the deceleration. When the probe-end 103a contacts the test surface 490, reading of the force data from the force adjustment system leads to a detection of contact, e.g. by comparing the contact threshold with an integrated value of the force data or by comparing the threshold with instantaneous values of the gradually increasing force values indicated by the force data.
Once the contact has been detected, further motion of the tool holder 102 may be stopped.
In addition to controlling the motorized multi-degree of freedom motion system 101 to move the sample probe until contact with the test surface is determined based on the force data and optionally, as in this example, the swab pose of one of the sampling points 491, the control may further be determined based on images obtained during the motion of the sample probe. In this way, unintended contact with other body parts of the patient may be avoided by analysing the images obtained during the motion.
Fig. 7 shows an example of a process for controlling the motorized multi-degree of freedom motion system 101 to move the sample probe 102 while maintaining contact with the test surface 490 based on the probe motion data 493 and the force data.
This motion of the sample probe 102 is performed to take the biological surface sample of the test surface 490, e.g. to perform a swab.
The process which is initiated after contact between the prove-end 103a and the test surface 490 has been established may begin with a pre-computation step before initiating the motion. First, the position of the probe-end 103a, which corresponds to the position at the contact with the test surface 490 is logged as a contact point, i.e. the actual contact point which may deviate from the desired sampling point 491. Next, a correction of the probe motion data 493 may be determined if required based on a possible offset between the location of the actual contact point and the location of the calculated sampling point 491, such as the sampling point 3 shown in Fig. 4B. This correction is beneficial for compensating any inaccuracy in the image data, such as depth information, provided by vision system 110.
The correction is performed by calculating an error vector 495 between the sampling point 3 and actual contact point 496. The error vector 495 may be applied for all sampling points 491. The execution of the motion of the probe-end 103a or swab trajectory then commences. To obtain a linear and angular speed for each trajectory segment between the sampling points 491, the linear and angular differences between each pair of consecutive sampling points 491, i.e. actual contact point 496 - sampling point 2, sampling point 2 - sampling point 1, etc. are calculated and a time between the points is assigned so that the speed will remain constant for each trajectory segment. In addition to this target swab speed, a second speed originating from the output of an admittance controller of the control system 220 is determined, which adds compliance around the target set point. The input forces to this admittance controller are first filtered with a moving average filter, and so is the output of the admittance controller itself. The target swab velocity is summed with the target compliance velocity, and the sum is fed into the speed controller of the control system 220. Once the motorized multi-degree of freedom motion system 101 has reached the final point of each trajectory segment (i.e. segments between sampling points 491), the process is repeated for the next pair of sampling points 491, until the last point has been reached.
The control of the motorized multi-degree of freedom motion system 101 to move the sample probe while maintaining contact with the test surface may be repeated after the last point was reached, wherein before the motion of the sample probe is repeated, the multi-degree of freedom motion system 101 is controlled to rotate the sample probe 103 around the longitudinal axis of the sample probe 103, e.g. by 90 degrees or 180 degrees, in order to collect additional biological sample material with the probe-end 103a.
The contact between the sample probe 103 and the test surface 490 may be maintained in between rotation of the sample probe 103. The rotation may be performed one or more times during the period when the sample probe 103 moves according to the probe motion data 493, or after said period so that probe motion according to the same or a different trajectory - as defined by probe motion data 493 - is performed after a first motion of the sample probe.
After the sample taking process has been completed with or without repetitions of the motion of the sample probe 103 on the test surface 490, the sample probe 103 is moved away from the test surface 490, possibly back to the image pose.
The control of the motorized multi-degree of freedom motion system 101 for moving of the sample probe away from the test surface 490 and/or out of the intra-cavity may be based on the image data and/or the force data obtained during the retraction of the sample probe 103. Thus, the control may be determined based on the image data 222 and/or the force data 223 equivalently to the methods described for moving the sample probe towards the test surface 490 for avoiding contact with other body parts which could result in a reduced quality of the obtained biological sample.
As described in connection with Fig. 3, in an embodiment, the motorized multi-degree of freedom motion system 101 is controlled to move the sample probe into the sample container positioned at the sanitation compartment 182 after the retraction of the sample probe 103. The sample probe 103 may be configured to allow the probe-end 103a to break off by applying a perpendicular force on the sample probe 103. Accordingly, to break or snap the end-portion 103a of the sample probe 103 into the sample container, the multi-degree of freedom motion system 101 may be controlled to move the sample probe in a direction transverse to a longitudinal axis of the sample probe after the sample probe has been positioned with the probe-end 103a in the sample container.
The tool holder 102 may be configured with a security means 190, principally illustrated in Fig.1B. The security means 190 is configured to provide a passive protection against high forces applied by the sample probe 103 on the patient. For example, in case of a incorrect force data 223, the sample probe 103 could be pressed against the test surface 490 with a too high force due to the control of the motorized multi-degree of freedom motion system 101. The security means 190 could be configured to allow free travel of the sample probe if the force acting on the probe-end exceeds a security force threshold. For example, security means 190 may comprise a membrane arranged to break and thereby allow free travel of the sample probe, or the security means 190 may comprise a magnetic arrangement connecting the sample probe 103 with the tool holder and arranged to disconnect and thereby allow free travel. An alarm sensor such as light detector or a switch comprised by the security means 190 or otherwise integrated, e.g. with the tool holder 102, may be arranged to detect the activation of the passive security means, e.g. by detecting displacement of the sample probe within a displacement range of the free travel. In case of detection, the alarm sensor generates an alarm signal 224 (Fig. 2) supplied to the control system 220.
For example, the control system 220 may be configured to perform a safety procedure, e.g. for prohibiting further motion of the tool holder 102, dependent on the alarm signal 224 if a force acting on the sample probe exceeds a threshold.
Although the present invention has been described in connection with the specified embodiments, it should not be construed as being in any way limited to the presented examples. The scope of the present invention is set out by the accompanying claim set. In the context of the claims, the terms "comprising", "comprises", "including" or "includes" do not exclude other possible elements or steps. Also, the mentioning of references such as "a" or "an" etc. should not be construed as excluding a plurality. The use of reference signs in the claims with respect to elements indicated in the figures shall also not be construed as limiting the scope of the invention. Furthermore, individual features mentioned in different claims, may possibly be advantageously combined, and the mentioning of these features in different claims does not exclude that a combination of features is not possible and advantageous.

Claims

A method for controlling an apparatus (100) for taking biological surface samples of a test surface (490) of a patient, the apparatus comprises
- a motorized multi-degree of freedom motion system (101) which is connectable with a tool holder (102), wherein the tool holder is configured to hold a sample probe (103) for obtaining the biological sample,

- a vision system (110) arranged to provide image data (222),

- a control system (220) for controlling the motorized multi-degree of freedom motion system based on the image data, the method comprises:
- setting the sample probe in an initial position,

- obtaining an image of the test surface during an identification period,

- determining probe motion data (493) based on the image data of the image,

- controlling the motorized multi-degree of freedom motion system to move the sample probe until contact with the test surface is determined, and

- controlling the motorized multi-degree of freedom motion system to move the sample probe while maintaining contact with the test surface based on the probe motion data.
A method according to any of the preceding claims, comprising determining a location of the test surface based on the image data and pre-stored data obtained based on training images, wherein the probe motion data is determined based at least on the location of test surface.
A method according to any of the preceding claims, wherein controlling the motorized multi-degree of freedom motion system to move the sample probe until contact with the test surface is determined, is based on images obtained during the motion of the sample probe.
A method according to any of the preceding claims, wherein
- the controlling of the motorized multi-degree of freedom motion system to move the sample probe until contact with the test surface is determined based on force data (223) indicative of a force acting on the sample probe, and/or

- the controlling the motorized multi-degree of freedom motion system to move the sample probe while maintaining contact with the test surface is based on the probe motion data and the force data (223).
A method according to any of the preceding claims, wherein controlling the motorized multi-degree of freedom motion system to move the sample probe until contact with the test surface is determined, is further subject to a calculated distance between the sample probe and the test surface.
A method according to any of the preceding claims, wherein the force data (223) indicative of a force acting on the sample probe is used for determining an unintended contact with biological surfaces of the patient other than the test surface.
A method according to any of the preceding claims, wherein controlling the motorized multi-degree of freedom motion system to move the sample probe while maintaining contact with the test surface is further based on the image data obtained during the motion of the sample probe.
A method according to any of the preceding claims, comprising providing a visual feedback to the user during the identification period, and optionally during a period where the probe motion data is determined.
A method according to any of the preceding claims, when the initial position is obtained, initiating an autonomous mode wherein the position of the sample probe is controlled based on the image data.
A method according to any of the preceding claims, comprising controlling the motorized multi-degree of freedom motion system to move the sample probe away from the test surface and/or out of the intra-cavity.
A method according to claim 10, wherein the controlling for moving the sample probe away from the test surface and/or out of the intra-cavity is based on the image data obtained during the motion of the sample probe and/or the force data.
A method according to any claims 10-11, wherein subsequent to moving the sample probe away from the test surface of the patient or out of the intra-cavity, controlling the motorized multi-degree of freedom motion system to move the sample probe into a sample container.
A method according to claim 12, subsequently, controlling the motorized multi-degree of freedom motion system to move the sample probe in a direction transverse to a longitudinal axis of the sample probe so as to break or snap an end-portion of the sample probe into the sample container.
A method according to any of the preceding claims, comprising performing a safety procedure, such as prohibiting motion of the tool holder, dependent on an alarm signal (224) generated by a security means (190) dependent on a force acting on the sample probe (103).
A method according to any of the preceding claims, wherein setting the sample probe in an initial position, comprises initially setting the motorized multi-degree of freedom motion system (101) in a user guided mode wherein the user can position the sample probe in the initial position.
A method according to any of the preceding claims, further comprising controlling the motorized multi-degree of freedom motion system to rotate the sample probe (103) around a longitudinal axis thereof and repeat or continue the moving the sample probe with contact to the test surface while a different portion of a sample-end (103a) contacts the test surface.
An apparatus (100) for taking biological surface samples of a patient, the apparatus comprises
- a motorized multi-degree of freedom motion system which is connectable with a tool holder, wherein the tool holder is configured to hold a sample probe for obtaining the biological sample,

- a vision system arranged to provide image data,

- a control system for controlling the motorized multi-degree of freedom motion system (101) based on the image data according to the method of any of claims 1-16.
An apparatus according to claim 17, further comprising,
- a force adjustment system arranged to provide force data (223) indicative of a force acting on the sample probe.
An apparatus according to claim 18, wherein the force adjustment system comprises a spring device arranged to displace in response to a force acting on the sample probe.
An apparatus according to any of claims 18-19, wherein the force adjustment system comprises a force feedback system arranged to provide the force data (223), wherein the control system is arranged for controlling the motorized multi-degree of freedom motion system (101) dependent on the force data (223).
An apparatus according to any of claims 18-20, further comprising,
- a security means (190) arranged to generate an alarm signal (224) dependent on the force data (223).
A computer program product comprising software code adapted to control an apparatus when executed on a data processing system, the computer program product being adapted to perform the method of any of claims 1-16.