EP2637571B1 - Robot à centre de mouvement distant pour balayage d'image médicale et ciblage guidé par image - Google Patents

Robot à centre de mouvement distant pour balayage d'image médicale et ciblage guidé par image Download PDF

Info

Publication number
EP2637571B1
EP2637571B1 EP11840646.1A EP11840646A EP2637571B1 EP 2637571 B1 EP2637571 B1 EP 2637571B1 EP 11840646 A EP11840646 A EP 11840646A EP 2637571 B1 EP2637571 B1 EP 2637571B1
Authority
EP
European Patent Office
Prior art keywords
imaging probe
ultrasound
probe
robot
rotary
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP11840646.1A
Other languages
German (de)
English (en)
Other versions
EP2637571A4 (fr
EP2637571A2 (fr
Inventor
Dan Stoianovici
Doru Petrisor
Felix Schafer
Chunwoo Kim
Misop Han
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Johns Hopkins University
Original Assignee
Johns Hopkins University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Johns Hopkins University filed Critical Johns Hopkins University
Publication of EP2637571A2 publication Critical patent/EP2637571A2/fr
Publication of EP2637571A4 publication Critical patent/EP2637571A4/fr
Application granted granted Critical
Publication of EP2637571B1 publication Critical patent/EP2637571B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/44—Constructional features of the ultrasonic, sonic or infrasonic diagnostic device
    • A61B8/4444—Constructional features of the ultrasonic, sonic or infrasonic diagnostic device related to the probe
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61N—ELECTROTHERAPY; MAGNETOTHERAPY; RADIATION THERAPY; ULTRASOUND THERAPY
    • A61N7/00—Ultrasound therapy
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/30—Surgical robots
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/08—Detecting organic movements or changes, e.g. tumours, cysts, swellings
    • A61B8/0833—Detecting organic movements or changes, e.g. tumours, cysts, swellings involving detecting or locating foreign bodies or organic structures
    • A61B8/0841—Detecting organic movements or changes, e.g. tumours, cysts, swellings involving detecting or locating foreign bodies or organic structures for locating instruments
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/08—Detecting organic movements or changes, e.g. tumours, cysts, swellings
    • A61B8/0833—Detecting organic movements or changes, e.g. tumours, cysts, swellings involving detecting or locating foreign bodies or organic structures
    • A61B8/085—Detecting organic movements or changes, e.g. tumours, cysts, swellings involving detecting or locating foreign bodies or organic structures for locating body or organic structures, e.g. tumours, calculi, blood vessels, nodules
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/12—Diagnosis using ultrasonic, sonic or infrasonic waves in body cavities or body tracts, e.g. by using catheters
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/13—Tomography
    • A61B8/14—Echo-tomography
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/42—Details of probe positioning or probe attachment to the patient
    • A61B8/4209—Details of probe positioning or probe attachment to the patient by using holders, e.g. positioning frames
    • A61B8/4218—Details of probe positioning or probe attachment to the patient by using holders, e.g. positioning frames characterised by articulated arms
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/42—Details of probe positioning or probe attachment to the patient
    • A61B8/4245—Details of probe positioning or probe attachment to the patient involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient
    • A61B8/4254—Details of probe positioning or probe attachment to the patient involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient using sensors mounted on the probe
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/48—Diagnostic techniques
    • A61B8/483—Diagnostic techniques involving the acquisition of a 3D volume of data
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B10/00—Other methods or instruments for diagnosis, e.g. instruments for taking a cell sample, for biopsy, for vaccination diagnosis; Sex determination; Ovulation-period determination; Throat striking implements
    • A61B10/02—Instruments for taking cell samples or for biopsy
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B10/00—Other methods or instruments for diagnosis, e.g. instruments for taking a cell sample, for biopsy, for vaccination diagnosis; Sex determination; Ovulation-period determination; Throat striking implements
    • A61B10/02—Instruments for taking cell samples or for biopsy
    • A61B10/0233—Pointed or sharp biopsy instruments
    • A61B10/0241—Pointed or sharp biopsy instruments for prostate
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A61B2090/378—Surgical systems with images on a monitor during operation using ultrasound
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/48—Diagnostic techniques
    • A61B8/485—Diagnostic techniques involving measuring strain or elastic properties
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/52—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/5207—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of raw data to produce diagnostic data, e.g. for generating an image
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61N—ELECTROTHERAPY; MAGNETOTHERAPY; RADIATION THERAPY; ULTRASOUND THERAPY
    • A61N7/00—Ultrasound therapy
    • A61N7/02—Localised ultrasound hyperthermia

Definitions

  • the present invention pertains to a remote center of motion robot for medical image scanning. More particularly, the present invention pertains to a remote center of motion robot for medical image scanning and image-guided targeting.
  • Prostate cancer is the most common form of cancer in American men. Like other cancers, early diagnosis and treatment is critical to the clinical management of the disease, and to preserving patients' quality of life and increasing life expectancy. In furtherance of these objectives, advanced imaging technologies have been developed to improve physicians' ability to accurately detect and stage clinically relevant cellular changes and to deliver treatment.
  • Ultrasound elastography is an emerging imaging modality with a potential to improve cancer detection.
  • the principle behind elastography is that different tissues exhibit distinct strain profiles under stress. If stress is induced (excited) with an ultrasound probe, images may be used to visualize the resulting strain patterns. During compression and retraction motions, zones of the image that remain relatively uncompressed reveal zones of higher stiffness which may correlate to cancer nodules. Elastography currently involves freehand probe motion to induce the stress. Consequently, the uniformity of repeated compression and retraction of the probe is critical to optimizing the reliability of elastography.
  • Prostate Specific Antigen PSA
  • TP transperineal
  • Brachytherapy accounts for a significant and growing proportion of prostate cancer treatments, as it is delivered with minimal invasiveness in an outpatient setting. Brachytherapy involves inserting multiple radioactive "seeds" into the prostate gland either temporarily (high dose-rate (HDR) brachytherapy), or permanently (low dose rate (LDR) brachytherapy). Brachytherapy, like transrectal and transperineal biopsy, requires multiple punctures to obtain multiple tissue cores.
  • HDR high dose-rate
  • LDR low dose rate
  • Transperineal biopsy is often performing using brachytherapy mapping templates to facilitate localization of cancerous tissue and to guide future biopsies. Both brachytherapy and biopsy are typically guided by 2-dimensional (2-D) transrectal ultrasound (TRUS). While 2-D TRUS provides adequate imaging of the soft tissue anatomy, it does not allow for localization or precise placement of biopsy needles or implanted brachytherapy seeds.
  • 2-D TRUS provides adequate imaging of the soft tissue anatomy, it does not allow for localization or precise placement of biopsy needles or implanted brachytherapy seeds.
  • LRP laparoscopic radical prostatectomy
  • TRUS probes are often operated manually; as such, image stability is comprised and the probe position data that is required for 3-D computation is lost. Also, TRUS systems have been too large to be used in tandem with advanced robotically-assisted minimally invasive surgical systems, where the space between the surgical robot and the patient is too limited to accommodate a human assistant for operating the TRUS system.
  • Robotic systems have recently emerged for TRUS probe tracking and needle interventions. Although these systems improve image stability and allow the operator to track probe position, these systems typically use extrinsic optical or electromagnetic tracking systems that lack the automated motion capabilities required to obtain uniform images or to digitally lock a trajectory. Another limitation of some robotic systems is that they commonly handle the needle rather than the probe and are built for transperineal rather than endocavity access. Therefore, the systems are not adaptable for TRUS-guided LRP and the points at which brachytherapy or biopsy needles are inserted must still be determined based on surgeon experience.
  • US Patent Application Publication No. US 2009/0234369 A1 describes a guide apparatus for orienting a medical tool relative to and through a remote fulcrum or remote center of motion.
  • the apparatus comprises at least one crank arm comprising at least a portion of a first hinged coupling for hinged coupling to a stabilizer; at least one link arm comprising at least a portion of a second hinged coupling for hinged coupling to the crank arm at a location spaced from the first hinged coupling; a tool holder for supporting a medical tool on the link arm at a location spaced from the first hinged coupling; wherein the rotational axes of the first and second hinged couplings intersect to define a remote fulcrum.
  • the present invention pertains to a remote center of motion robot for medical image scanning and image-guided targeting, hereinafter referred to as the "Euler” robot.
  • the Euler robot allows for ultrasound scanning for 3-Dimensional (3-D) image reconstruction and enables a variety of robot-assisted image-guided procedures, such as needle biopsy, percutaneous therapy delivery, image-guided navigation, and facilitates image-fusion with other imaging modalities.
  • the Euler robot can also be used with other handheld medical imaging probes, such as gamma cameras for nuclear imaging, or for targeted delivery of therapy such as high-intensity focused ultrasound (HIFU).
  • 3-D ultrasound probes may also be used with the Euler robot to provide automated image-based targeting for biopsy or therapy delivery.
  • the Euler robot enables the application of special motion-based imaging modalities, such as ultrasound elastography.
  • the Euler robot uses remote center of motion kinematics, which is a characteristic of minimally invasive access devices.
  • the Euler robot according to features of the present invention is also applicable to manipulating other medical instruments and devices, such as laparoscopic instruments, including ultrasound.
  • the Euler robot 10 includes a support arm 12 and a remote center of motion (RCM) module 14, which is operatively connected to the support arm 12.
  • the support arm 12 is shown as having a generic structure with two spherical joints 16 and one cylindrical joint 18 which can be located at a desired location to support the base of the Euler robot 10 in place as needed for the clinical operation.
  • a mount 20 may also be provided at the end of the support arm 12, which can be attached to a fixed support such as a medical table.
  • the remote center of motion module preferably has a parallelogram structure built with belts, as described in U.S. Patent No. 7,021,173 (the '173 patent).
  • the RCM module includes first, second, and third connecting link units, the first connecting link unit coupled to a base link unit at a passive revolute joint.
  • the base adjustment angle can be changed by adjusting the angle between the base link unit and first connecting link unit.
  • the base link unit further comprises a base shaft, which provides one rotational degree of freedom about a first axis.
  • a second connecting link unit is coupled to the first connecting link unit at a revolute joint, and the second connecting link unit is further coupled to a third connecting link unit by another revolute joint.
  • the third connecting link unit is configured to receive, at another joint (output shaft), a driver or holder for an end-effector, such as an ultrasound probe, biopsy needle, or other medical instrument.
  • the revolute joint between the first and second connecting link units may be actuated by motor, while the revolute joints between the second and third connecting link units and at the output shaft are coupled to the first revolute joint through a belt-drive mechanism.
  • the second rotational axis of the RMC module that provides a second degree of freedom is materialized by the first, second, and third connecting link units and the end-effector/driver holder.
  • the system has been designed so that the second connecting link may be actuated with respect to the first, while the third connecting link maintains its parallel orientation with respect to the first connecting link.
  • the end-effector holder maintains its parallel orientation with respect to the second connecting unit.
  • the configuration of the first and second rotation axes and corresponding rotating joints and connecting links form a double-parallelogram-based structure. This design allows for rotating the end-effector about an axis, y, which is remote from the mechanism and forms the classic RCM design.
  • the RCM module 14 presents two rotary degrees of freedom (DOF) (Ry, Rz) about joints 30 and 32 with axes intersecting at an RCM pivot point.
  • DOF degrees of freedom
  • An important feature of this RCM mechanism is that its structure is located from the pivot point, so that space is cleared from the medical intervention and current instrument and/or probe.
  • RCM mechanism may be used, depending upon application and design preference.
  • a driver module 26 (RT driver) is operatively connected to the RCM module 14.
  • the driver module manipulates an imaging probe 24 positioned therein in at least two degree of freedoms.
  • the driver module 26 may provide one degree of freedom around a first rotation axis Rx and one linear degree of freedom for translation along a second axis Tx.
  • an alternative driver module 36 may be operatively connected to the RCM module 14.
  • the driver module 36 manipulates the imaging probe 24 (positioned therein) in one degree of freedom.
  • the driver module 36 provides one degree of freedom around a first rotation axis Rx.
  • the driver modules can be constructed such that palpations are allowed for elastography and/or the robot is adapted for holding and operating a needle for biopsy.
  • the probe 24 is moved to sweep the region of interest for imaging. Motion is also used to orient the probe as needed for image-guided targeting.
  • a main advantage is that the robot automatically tracks the position and orientation of the image space because it handles the probe. As such, intraoperative image-to-robot registration which is common for most image-guided robots is not required. Instead, a calibration is performed once, ahead of time, and should remain constant.
  • FIGS. 3(a) and (b) the orientation of the tool frame T (end-effector instrument) can be described by a classic set of yaw, pitch, and roll Euler angles (Z-Y-X).
  • FIG. 3(a) is a symbolic representation of the RCM module implementing Z-Y-X Euler angles
  • FIG. 3(b) is a fixed frame X-Y-Z rotations of its tool frame ⁇ T ⁇ in the fixed base frame ⁇ B ⁇ .
  • the remote axis of the RCM mechanism uses a parallelogram mechanism implemented with two belts, as described in U.S. Patent No. 7,021,173 . Since these joints are mechanically coupled, they are represented symbolically with a single joint connecting links 1-2.
  • the Euler robot implements a sequence of three rotations ( ⁇ 3 , ⁇ 2 , ⁇ 1 ) performed about the axes of the moving frames, starting from the base frame ⁇ B ⁇ of link (0) to the tool frame ⁇ T ⁇ of link (3).
  • This RCM3 link configuration has a very interesting robot kinematic consequence, as follows.
  • a tool frame orientation can be expressed as a sequence of rotations about the X ⁇ B ⁇ ⁇ B ⁇ ⁇ B axes of the fixed ⁇ B ⁇ coordinate frame shown in b.
  • R T B r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33
  • the Euler robot with the driver module 26 (RT driver) will be described in more detail.
  • the RCM module 14 supporting the ultrasound probe 24 is illustrated in its folded/collapsed position, which the parallelogram is collapsed, as shown in FIGS. 4 and 5 .
  • An ultrasound probe 24 is mounted using an adapter 40 that is custom built to match the shape of the probe.
  • adapter 40 that is custom built to match the shape of the probe.
  • various other probes may be attached to the driver by other special adapters and the like.
  • the adapter 40 may be attached to the driver 26 through preload plates 42, including sensors, as is supported by a structure including a linear rail 44 and guide 46.
  • the linear rail 44 is actuated with a ball screw 48 by a geared servomotor 50.
  • the linear rail 44 and guide 46 are then supported by the rail structure of a custom built rotary rail 52 and guide 54.
  • the rotary rail 52 preferably presents a horseshoe shape and comprises a spur gear sector 56 mounted between two rail parts.
  • the rotary rail 52 and guide 54 present a similar structure to classic linear ball rails, but the rails have circular geometry so that the resulting motion is rotary rather than linear.
  • ball grooves 60 are made in the guides and on each side of the rail 52, and two series of steel balls 62 circulate in these grooves and then recirculate through special paths made within the blocks of the guides through a ball recirculation path 64.
  • actuation of the horseshoe shaped rotary rail 52 is given by its gear sector 66 which is engaged by a pinion 68 from a geared servomotor 70.
  • Both axes are equipped with redundant encoders 72, 74, one encoder 72 connected directly to the screw 48, the other encoder 74 connected to the pinion 68 of the rotary axes through another pinion.
  • a set of hardware limit switches 76 is used on each axis. Since both the motor and redundant encoders are incremental, these are also used for homing.
  • a set of five force sensors are used to measure all force-torques interactions of the probe 24 except for the torque exerted about its longitudinal axis.
  • the force sensors are miniature quartz sensor series with extremely flat design for measuring dynamic and quasistatic forces, but other types of sensors are possible.
  • Four of the sensors are intercalated between the probe adapter 40 and the RT driver 26, to measure all forces and moments exerted lateral to the probe 24.
  • the axial force is measured by a sensor intercalated between the ball nut and the rotary rail 52.
  • An arm 84 connects the base of the mechanism, which is the rotary guide assembly, to the distal RCM mount (not shown).
  • the RT driver 26 implements a rotation which in addition to the RCM module completes the set of three Euler orientation angles. Probe adapters are built so that these align with the longitudinal axis of the probe. The RT driver also provides a linear degree of freedom for motion along this axis.
  • the horseshoe geometry of the rotary rail has been chosen to clear the space above the ultrasound probe and so facilitate manual access and manipulation of needles about the probe for ultrasound-guided interventions such as biopsy. Alternatively, if this access to the probe is not required, the construction can be substantially simplified by using a disk like structure for the rotary axis which eliminates the need for the rotary ball rails and guide.
  • the Euler robot with the driver module 36 (R driver) will be described in more detail.
  • the RCM module 14 supporting the ultrasound probe 24 is illustrated in its folded/collapsed position, which the parallelogram is collapsed, as shown in FIGS. 8 and 9 .
  • An ultrasound probe 24 is mounted using an adapter 90 that is custom built to match the shape of the probe 24.
  • adapter 90 that is custom built to match the shape of the probe 24.
  • various other probes may be attached to the driver by other special adapters and the like.
  • an ultrasound probe 24 (external body probe shown) is supported with an adapter 90 so that various probes can be used.
  • the adapter 90 also provides a regular geometric shape for mounting the probe 24 because typical handheld probes have notoriously irregular geometry.
  • the adapter 90 may also include a set of mounting clips 92 for easy attachment/disconnection to the driver 36. Overall, these features ensure that the same probe 24 can be repeatedly connected to the driver 36 in the same relative position and orientation. This is.important to maintain the calibration of the image frame with respect to the robot.
  • the driver 36 presents a disk like structure.
  • the adapter 90 of the probe 24 connects to a central rotor 94 that spins the probe 24.
  • the rotor 94 is supported by a set of ball bearings 96 and o-rings 98, which help to insolate the mechanisms.
  • the races of the bearings are built within the body 100 and cap 102 of the driver 36.
  • the bearings do not use a cage, but a sequence of intercalated slightly smaller balls. These normally roll in a reversed direction than the main balls possibly reducing friction.
  • the rotor 94 is engaged by a spur gear transmission from a servomotor 104 having gears 105, which is contained within the motor cover 106.
  • a limit switch 104 mounted on the base triggers a rotary location for homing an incremental encoder.
  • the driver 36 does not use redundant encoding, but could simply be added with another pinion 107 engaged by the gear, similar to the RT Driver described above.
  • the body 100 of the driver connects to the RCM module 14 through a custom made force sensor 108 to measure the force exerted on the probe in the axial direction, which is secured within the mount caps 109.
  • a structure of four strain concentration elements may be used. Strain gauges are applied to the opposite sides of a bridge and a half-bridge connection may be used for the measurement.
  • FIG. 13 a block diagram of the system is presented in, showing its operation with an ultrasound probe.
  • a video camera is also included for system monitoring in applications where direct observation of the Euler robot may be difficult, such as the tandem robot approach described in the following sections.
  • the controller of the robot may be based on an Intel (Santa Clara, CA), Core 2 Quad Q9550 processor on Huawei (ASUSTeK Computer Inc., Taiwan), P5N7A-VM motherboard with 4GB of DDR2 memory running Windows XP (Microsoft Corp, Redmond, WA).
  • the computer system may be equipped with a built-in uninterruptable power supply or operation safety and short range portability.
  • Motor control may be implemented on an MC8000-DUAL (PMDI, Victoria, B.C., Canada) motion control card with onboard digital signal processor for real-time motion control. This card presents dual quadrature decoders and counters for each axis, used for the motor and redundant encoders of each axis.
  • Linear servomotor amplifiers may be used for all four axes.
  • a purpose built relay and watchdog board may be included.
  • a single 50 connector cable connects the PC to a connection box at the base of the passive arm.
  • the arm mounts to a custom oversized rail for increased rigidity.
  • the connection box also includes the amplifiers for force sensors to reduce the length of force sensor cables.
  • the watchdog checks the state of several components of the system once every 100ms, disabling power to the motor amplifiers if a faulty condition is detected. Two visual signs are used to signal the operation state of the robot being powered and in motion. An Emergency Stop button disables the system and also suspends motor amplifier power. The operation of the system is performed from either a 2-DOF joystick equipped with RCM or RT Driver enable buttons, or under numeric control.
  • the software of the TRUS Robot consists of axis-level motion control software, robot kinematics software, ultrasound visualization, and 3-D image processing software.
  • Motion control software is built in Visual C++ (Microsoft Corp, Redmond, WA) based on high-level libraries of functions of the motion control card (MCI-SoftLib, PMDI). Imaging components and interface to the robot are also written in Visual C++ based on the Amira Visualization platform (Visage Imaging Inc, San Diego, CA).
  • the positional data along with the images is readily available. This allows recording image-position data pairs for robot navigation and 3-D reconstruction.
  • the robot may scan the prostate in arbitrary directions.
  • a rotary scan about the probe axis is normally proffered to reduce soft tissue deflections.
  • Special imaging algorithms were developed for the rotary scan, because typically 3-D reconstruction is based on parallel image slices (for example CT or MRI).
  • the present robot facilitates navigation during surgery and medical interventions.
  • the robotic apparatus may be used to manipulate the ultrasound imaging transducer, in tandem with the at least one medical instrument suitable as a fiducial marker.
  • a region of interest is scanned and at least one parameter of at least one anatomical feature is measured therein with the ultrasound imaging transducer.
  • the position of the imaging transducer is tracked during surgery and interventions using a programmable computer linked to the robotic apparatus.
  • the information obtained from tracking is applied to the transducer to construct at least one three-dimensional model of the region of interest in which the medical instrument can be visualized during the surgery or intervention.
  • the medical instrument is manipulated about the region of interest using the information derived from the at least one three-dimensional model.
  • the particular medical instrument used preferably includes at least one arm and associated or incorporated instrument of a robotically-assisted minimally invasive surgical system, including a therapy delivery device, needle for obtaining tissue samples, fiducial marker, or laparoscopic or other surgical instrument.
  • the robotic apparatus may be used in tandem with a medical procedure performed manually or robotically. During scanning of the ultrasound transducer, the medical instrument is visible as a hyperechoic region in at least one live ultrasound image captured intraoperatively. The apparatus manipulates the ultrasound imaging transducer to scan in arbitrary directions. The robotic apparatus allows continuous tracking of the ultrasound transer position within the available reference frame. A computer system captures at least one ultrasound image and the corresponding ultrasound imaging transducer position coordinates. The robotic apparatus provides measurements of the remote center of motion and ultrasound transducer drive angles from which ultrasound images are acquired. The information gathered from the images are segmented and used for producing a three-dimensional ultrasound image volume of anatomic features of interest in the target surgical site.
  • the Euler robot 120 is used in tandem with a DA VINCI® robot 122.
  • the Euler robot 120 may be attached to the end of the surgical table, in the confined space below the DA VINCI® robot 122, and may be maneuvered by a joystick 123.
  • ultrasound images are acquired through a computer system 124 and presented to the surgeon on the console 126 along the current laparoscopic view using a side-by-side display.
  • a video camera 128 may be used to visualize the surgical site, and provide images to the display.
  • the Euler robot may be used in connection with elastography.
  • a programmable computer system may be provided for controlling repeated compression and retraction of the ultrasound transducer for palpating the anatomic region of interest.
  • the Euler robot may be used in connection with performing needle biopsies on a patient.
  • a programmable computer system may be provided for controlling insertion and retraction of the biopsy needle to obtain a tissue sample from an anatomic region of interest.
  • Example has been included to provide guidance to one of ordinary skill in the art for practicing representative embodiments of the presently disclosed subject matter.
  • those of skill can appreciate that the following Example is intended to be exemplary only and that numerous changes, modifications, and alterations can be employed without departing from the scope of the presently disclosed subject matter.
  • the following Example is offered by way of illustration and not by way of limitation.
  • FIG. 14(a) shows a mockup which simulates the pelvic bones, prostate, bladder, urethra, and two structures representing neurovascular bundles.
  • the mockup was gelatin based, made of 300 Bloom gelatin powder in solution with sorbitol, glycerin, and water, to create a realistic ultrasound view. Images acquired with a rotary scan were post processed for segmentation and represented, as shown in FIG. 14(b).
  • FIG. 14(c) shows a 3-D representation of the prostate that is generated directly from the "gathered” image space, using a "volume rendering" mode. This can display the 3-D shape of the gland without segmenting it. Volume rendering displayed each gathered image voxel translucent, and light absorption through the voxel was correlated to its brightness according to a scale factor.
  • the robotically-assisted ultrasound manipulator of the present invention allows intrinsic tracking of the probe position and the orientation of the image space, which greatly benefits performing brachytherapy, needle biopsy, LRP, and transrectal ultrasound elastography. Practitioners also benefit from the remote center of motion kinematics in a compact system that extends the application range to manipulating medical instruments and devices such as laparoscopic instruments, including laparoscopic ultrasound. Also, the compact configuration allows practitioners to use the robot in tandem with robotically-assisted minimally invasive surgical systems. Finally, a robotic system that automatically tracks ultrasound probe position and that permits the uniform application and retraction of the probe optimizes the quality and reliability of ultrasound elastography.

Claims (13)

  1. Un appareil pour soutenir et manipuler une sonde d'imagerie (24) lorsque des interventions guidées par ultrasons sont réalisées, comprenant :
    un module centre de mouvement distant (14) fournissant à une sonde d'imagerie (24) au moins un premier degré de liberté rotatoire autour d'un premier axe (Ry) et un deuxième degré de liberté rotatoire autour d'un deuxième axe (Rz) ; et
    un module d'entraînement (26) relié fonctionnellement au module centre de mouvement distant (14), le module d'entraînement (26) fournissant un degré de liberté linéaire pour une translation le long d'un axe (Tx) aligné avec un axe longitudinal de la sonde d'imagerie (24) et fournissant en sus un troisième degré de liberté rotatoire pour manipuler ladite sonde d'imagerie (24) autour d'un troisième axe (Rx), ledit module d'entraînement (26) comprenant en sus un guide pour rotation (54) et un rail pour rotation (52), ledit rail pour rotation (52) ayant une géométrie pour permettre à un instrument médical supplémentaire d'être positionné de façon adjacente à la sonde d'imagerie (24),
    dans lequel lesdits premier (Ry), deuxième (Rz), et troisième (Rx) axes de rotation représentent un jeu de trois angles d'orientation d'Euler et se coupent au niveau d'un point unique distant du module centre de mouvement distant (14) ;
    l'appareil comprenant en sus un adaptateur de sonde d'imagerie (40) fixé au module d'entraînement (26) pour assujettir la sonde d'imagerie (24) ;
    caractérisé en ce que l'adaptateur de sonde d'imagerie (40) est conçu pour assujettir la sonde d'imagerie (24) de telle sorte que le rail pour rotation (52) encercle au moins partiellement la sonde d'imagerie (24).
  2. L'appareil de la revendication 1, dans lequel au moins un ensemble de capteurs de force relie fonctionnellement le module d'entraînement (26) à l'adaptateur (40).
  3. L'appareil de la revendication 2, dans lequel l'au moins un ensemble de capteurs de force et l'adaptateur (40) sont soutenus par un rail linéaire (44) aligné longitudinalement avec l'adaptateur (40).
  4. L'appareil de la revendication 1, dans lequel le module d'entraînement (26) comprend en sus un rail linéaire (44) et un guide linéaire (46), lesdits rail linéaire (44) et guide linéaire (46) étant soutenus par ledit rail pour rotation (52) et ledit guide pour rotation (54), le rail linéaire (44) étant actionné avec une vis à billes (48) par un servomoteur à engrenages (50).
  5. L'appareil de la revendication 1, dans lequel ledit adaptateur (40) comprend en sus un guide-aiguille.
  6. L'appareil de la revendication 1, comprenant en sus un organe de commande, l'organe de commande amenant l'appareil à manipuler la sonde d'imagerie (24) afin de parcourir une région d'intérêt en utilisant un balayage par mouvement rotatif.
  7. L'appareil de la revendication 1, comprenant en sus un ordinateur programmable connecté à l'appareil, ledit ordinateur programmable permettant un suivi continu d'une position de la sonde d'imagerie (24) à l'intérieur d'un référentiel disponible.
  8. L'appareil de la revendication 1, comprenant en sus une sonde d'imagerie ultrasonore comme sonde d'imagerie (24) et un système informatique (124) configuré pour capturer au moins une image ultrasonore et des coordonnées de position de sonde d'imagerie correspondantes.
  9. L'appareil de la revendication 8, l'appareil fournissant des mesures du centre de mouvement distant et d'angles d'entraînement de sonde d'imagerie auxquels des images ultrasonores sont acquises.
  10. L'appareil de la revendication 8, ledit appareil segmentant des informations rassemblées à partir de ladite au moins une image ultrasonore et utilisant lesdits segments pour produire un volume d'image ultrasonore en trois dimensions de caractéristiques d'intérêt dans un site cible.
  11. L'appareil de la revendication 2, dans lequel ledit au moins un ensemble de capteurs de force mesure des forces et des moments exercés en ce qui concerne la sonde d'imagerie (24).
  12. L'appareil de la revendication 1, dans lequel ledit rail pour rotation (52) a une géométrie en fer à cheval.
  13. L'appareil de la revendication 8, dans lequel ledit système informatique (124) est configuré pour commander une compression et une rétraction répétées de la sonde d'imagerie ultrasonore (24) pour palper une région d'intérêt.
EP11840646.1A 2010-11-11 2011-11-11 Robot à centre de mouvement distant pour balayage d'image médicale et ciblage guidé par image Active EP2637571B1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US41258910P 2010-11-11 2010-11-11
PCT/US2011/060362 WO2012065058A2 (fr) 2010-11-11 2011-11-11 Centre distant de robot mobile pour balayage d'image médicale et ciblage guidé par image

Publications (3)

Publication Number Publication Date
EP2637571A2 EP2637571A2 (fr) 2013-09-18
EP2637571A4 EP2637571A4 (fr) 2017-07-12
EP2637571B1 true EP2637571B1 (fr) 2020-01-08

Family

ID=46051580

Family Applications (1)

Application Number Title Priority Date Filing Date
EP11840646.1A Active EP2637571B1 (fr) 2010-11-11 2011-11-11 Robot à centre de mouvement distant pour balayage d'image médicale et ciblage guidé par image

Country Status (5)

Country Link
US (1) US20140039314A1 (fr)
EP (1) EP2637571B1 (fr)
KR (1) KR102048352B1 (fr)
CN (1) CN103200877B (fr)
WO (1) WO2012065058A2 (fr)

Families Citing this family (51)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR2972132B1 (fr) * 2011-03-02 2014-05-09 Gen Electric Dispositif d'assistance a la manipulation d'un instrument ou outil
US20150238167A1 (en) * 2012-03-22 2015-08-27 Gamma Medical Technologies, Llc Dual modality endocavity biopsy imaging system and method
WO2013142778A2 (fr) * 2012-03-22 2013-09-26 Lall Terrence Applications d'une sonde de diagnostic d'endocavité dans le traitement d'une maladie
US9615815B2 (en) * 2012-09-28 2017-04-11 Clemson University Research Foundation Devices that cooperate with ultrasound probes for muscoskeletal evaluations and related systems and methods
US10383598B2 (en) * 2013-01-14 2019-08-20 B-K Medical Aps Ultrasound imaging probe
KR101474309B1 (ko) * 2013-03-21 2014-12-18 가톨릭관동대학교산학협력단 초음파 탐촉자 고정장치
RU2692206C2 (ru) 2013-05-09 2019-06-21 Конинклейке Филипс Н.В. Роботизированное управление эндоскопом на основе анатомических признаков
CN103230304B (zh) * 2013-05-17 2015-05-13 深圳先进技术研究院 手术导航系统及其方法
CN103371870B (zh) * 2013-07-16 2015-07-29 深圳先进技术研究院 一种基于多模影像的外科手术导航系统
WO2015085257A1 (fr) * 2013-12-06 2015-06-11 Sonitrack Systems, Inc. Système et procédé de balayage ultrasonore à commande mécanique
US10743909B2 (en) 2014-04-03 2020-08-18 Corbin Clinical Resources, Llc Transperineal prostate biopsy device, systems, and methods of use
US10064681B2 (en) * 2014-04-03 2018-09-04 Corbin Clinical Resources, Llc Method, system, and device for planning and performing, guided and free-handed transperineal prostate biopsies
US10368850B2 (en) * 2014-06-18 2019-08-06 Siemens Medical Solutions Usa, Inc. System and method for real-time ultrasound guided prostate needle biopsies using a compliant robotic arm
WO2016025389A1 (fr) * 2014-08-11 2016-02-18 The Regents Of The University Of California Dispositif et système d'élastographie par aiguille fine pour la mesure des propriétés d'un matériau
US10561397B2 (en) * 2014-10-31 2020-02-18 Spade Co., Ltd Ultrasonic probe injection device using RCM
CN104622573B (zh) * 2014-12-12 2017-06-09 北京航空航天大学 一种具有高刚性的四自由度远程运动中心机构
DE102014226240A1 (de) * 2014-12-17 2016-06-23 Kuka Roboter Gmbh System zur roboterunterstützten medizinischen Behandlung
KR20160094105A (ko) 2015-01-30 2016-08-09 전자부품연구원 로봇 제어 시각화 장치
CN107257670B (zh) * 2015-02-26 2021-03-16 柯惠Lp公司 用软件及导管以机器人方式控制远程运动中心
EP3285652B1 (fr) 2015-04-21 2018-12-05 Koninklijke Philips N.V. Bras réglable pour un dispositif de surveillance d'un patient
WO2017004056A1 (fr) 2015-07-01 2017-01-05 Mako Surgical Corp. Systèmes et procédés robotiques pour commander un outil qui enlève de la matière d'une pièce à travailler
JP6865739B2 (ja) * 2015-09-28 2021-04-28 コーニンクレッカ フィリップス エヌ ヴェＫｏｎｉｎｋｌｉｊｋｅ Ｐｈｉｌｉｐｓ Ｎ．Ｖ． 遠隔運動中心ロボットの光学的位置合わせ
US10292779B2 (en) * 2015-11-23 2019-05-21 Sina Robotics And Medical Innovators Co. Adapting manual laparoscopic surgical instruments for robotic telesurgery applications
US20190125297A1 (en) * 2016-04-08 2019-05-02 The Regents Of The University Of Michigan Device for imaging assisted minimally invasive implant and jawbone reconstruction surgery
US11278742B2 (en) 2016-04-28 2022-03-22 Koninklijke Philips N.V. Image guided treatment delivery
CN105962971A (zh) * 2016-05-03 2016-09-28 上海交通大学 医用超声探头多自由度驱动装置
WO2017220722A1 (fr) * 2016-06-22 2017-12-28 Koninklijke Philips N.V. Guidage par imagerie pour commande cinématique découplée d'un centre de mouvement distant
CN106361431A (zh) * 2016-08-29 2017-02-01 杭州捷诺飞生物科技有限公司 基于生物3d打印技术的切割修复一体化手术机器人
JP6717713B2 (ja) * 2016-08-31 2020-07-01 テルモ株式会社 医療用デバイス
CN106327479B (zh) * 2016-09-23 2018-11-20 西安电子科技大学 血管造影中介下先心病术中血管辨识的装置及方法
US20200046215A1 (en) * 2017-02-09 2020-02-13 Gynisus Ltd. A medical monitoring system and method
CN107028626B (zh) * 2017-05-26 2023-06-06 北京龙慧珩医疗科技发展有限公司 一种前列腺术弹性配准方法及装置
CN110868937B (zh) * 2017-07-07 2024-03-08 皇家飞利浦有限公司 与声学探头的机器人仪器引导件集成
CN108577980A (zh) * 2018-02-08 2018-09-28 南方医科大学南方医院 一种对超声刀头进行自动跟踪的方法、系统及装置
CN108553768B (zh) * 2018-05-16 2023-11-24 天津商业大学 前列腺粒子植入机器人
CN109288541B (zh) * 2018-11-15 2021-08-06 深圳市比邻星精密技术有限公司 基于超声扫查的机器人系统及其扫查方法
IT201900002469A1 (it) * 2019-02-20 2020-08-20 Univ Degli Studi Di Verona Dispositivo chirurgico per biopsia transrettale, in particolare prostatica, per un braccio mobile di un robot
IT201900002475A1 (it) * 2019-02-20 2020-08-20 Univ Degli Studi Di Verona Dispositivo chirurgico per biopsia transperineale, in particolare prostatica, per un braccio mobile di un robot
EP3934558A4 (fr) * 2019-03-07 2022-12-14 PROCEPT BioRobotics Corporation Bras robotiques et méthodes de résection et d'imagerie de tissu
CN112057110B (zh) * 2019-05-22 2023-05-23 深圳市德力凯医疗设备股份有限公司 一种三维血管超声图像的成像方法及超声术中导航设备
US11426198B2 (en) * 2019-07-04 2022-08-30 Aaron Fenster Biopsy apparatus
US11071601B2 (en) 2019-11-11 2021-07-27 Procept Biorobotics Corporation Surgical probes for tissue resection with robotic arms
RU197062U1 (ru) * 2019-12-26 2020-03-26 федеральное государственное автономное образовательное учреждение высшего образования "Санкт-Петербургский политехнический университет Петра Великого" (ФГАОУ ВО "СПбПУ") Робот для термической абляции новообразований методом HIFU
US11096753B1 (en) 2020-06-26 2021-08-24 Procept Biorobotics Corporation Systems and methods for defining and modifying range of motion of probe used in patient treatment
US11877818B2 (en) 2020-06-26 2024-01-23 Procept Biorobotics Corporation Integration of robotic arms with surgical probes
CN111839592B (zh) * 2020-08-13 2023-12-29 厦门市领汇医疗科技有限公司 一种前列腺检测装置
US20220104878A1 (en) * 2020-10-06 2022-04-07 Mazor Robotics Ltd. Method, device, and system for image generation based on calculated robotic arm positions
KR102570097B1 (ko) * 2020-11-12 2023-08-24 주식회사 제우기술 로봇 암을 이용한 검체 채취 시스템 및 방법
CN113855244B (zh) * 2021-09-08 2022-10-18 江苏集奥医工交叉科技有限公司 一种用于疼痛治疗的手术机器人
CN114019031B (zh) * 2021-09-13 2023-10-13 上海工程技术大学 力触觉系统主端操作机构及力触觉核电装备运维装置
CN113951935A (zh) * 2021-10-26 2022-01-21 北京智愈医疗科技有限公司 一种经腔道的超声自动检查系统及控制方法

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2567928Y2 (ja) * 1992-08-11 1998-04-08 日本精工株式会社 ラック一体型リニアガイド装置
WO1997000649A1 (fr) * 1995-06-20 1997-01-09 Wan Sing Ng Bras articule pour des procedures medicales
JP2005516786A (ja) * 2002-02-06 2005-06-09 ザ ジョンズ ホプキンズ ユニバーシティ 遠隔動心ロボットシステムおよび方法
US20050261591A1 (en) * 2003-07-21 2005-11-24 The Johns Hopkins University Image guided interventions with interstitial or transmission ultrasound
JP4999012B2 (ja) * 2005-06-06 2012-08-15 インチュイティブ サージカル，インコーポレイテッド 腹腔鏡超音波ロボット外科手術システム
WO2007106558A2 (fr) * 2006-03-14 2007-09-20 The Johns Hopkins University Appareil d'insertion d'un dispositif medical dans un corps au cours d'une procedure d'imagerie medicale, et dispositifs et procedes associes
US20090234369A1 (en) * 2006-06-19 2009-09-17 Robarts Research Institute Apparatus for guiding a medical tool
CN100571606C (zh) * 2006-12-21 2009-12-23 中国科学院电工研究所 一种微型机器人及其体外导向系统
US7832114B2 (en) * 2007-04-04 2010-11-16 Eigen, Llc Tracker holder assembly
US20080314181A1 (en) * 2007-06-19 2008-12-25 Bruce Schena Robotic Manipulator with Remote Center of Motion and Compact Drive
US9895813B2 (en) * 2008-03-31 2018-02-20 Intuitive Surgical Operations, Inc. Force and torque sensing in a surgical robot setup arm
TW201127573A (en) * 2010-02-05 2011-08-16 Hon Hai Prec Ind Co Ltd Robot arm

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
KR20140024837A (ko) 2014-03-03
EP2637571A4 (fr) 2017-07-12
US20140039314A1 (en) 2014-02-06
EP2637571A2 (fr) 2013-09-18
WO2012065058A2 (fr) 2012-05-18
WO2012065058A3 (fr) 2012-08-02
CN103200877A (zh) 2013-07-10
CN103200877B (zh) 2016-02-10
KR102048352B1 (ko) 2019-11-25

Similar Documents

Publication Publication Date Title
EP2637571B1 (fr) Robot à centre de mouvement distant pour balayage d'image médicale et ciblage guidé par image
US20220409306A1 (en) Surgical robot platform
US11596367B2 (en) Electromagnetic tracking system and methods of using same
Cleary et al. Interventional robotic systems: Applications and technology state‐of‐the‐art
US11395706B2 (en) Surgical robot platform
Kettenbach et al. Robot-assisted biopsy using ultrasound guidance: initial results from in vitro tests
EP2836133B1 (fr) Sonde à ultrasons/robot cohésif pour biopsie de la prostate
Stoianovici et al. Endocavity ultrasound probe manipulators
Kettenbach et al. Robot-assisted biopsy using computed tomography-guidance: initial results from in vitro tests
Kronreif et al. Robotic guidance for percutaneous interventions
Kim et al. Robot for ultrasound-guided prostate imaging and intervention
Kronreif et al. Robotic System for Image Guided Therapie-B-RobII
Jiang et al. Design and experiments of ultrasound image-guided multi-dof robot system for brachytherapy
Moreira et al. Motorized template for MRI-guided focal cryoablation of prostate cancer
Bax New Mechatronic Systems for the Diagnosis and Treatment of Cancer
Bassan DESIGN, CONSTRUCTION AND CONTROL
Garth-Davis A needle positioning system for percutaneous procedures

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20130502

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20170613

RIC1 Information provided on ipc code assigned before grant

Ipc: A61B 8/12 20060101ALI20170607BHEP

Ipc: A61N 7/00 20060101ALI20170607BHEP

Ipc: A61B 8/14 20060101AFI20170607BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20180802

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20190712

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602011064558

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1221674

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200215

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20200108

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200531

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200408

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200408

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200508

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200409

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602011064558

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1221674

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200108

26N No opposition filed

Effective date: 20201009

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201111

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20201130

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201130

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201130

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201111

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201130

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230515

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20231127

Year of fee payment: 13

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20231127

Year of fee payment: 13

Ref country code: DE

Payment date: 20231129

Year of fee payment: 13