WO2002080773A1 - Appareil de realite amplifiee et procede de tomographie assistee par ordinateur (ct) - Google Patents

Appareil de realite amplifiee et procede de tomographie assistee par ordinateur (ct) Download PDF

Info

Publication number
WO2002080773A1
WO2002080773A1 PCT/US2002/010663 US0210663W WO02080773A1 WO 2002080773 A1 WO2002080773 A1 WO 2002080773A1 US 0210663 W US0210663 W US 0210663W WO 02080773 A1 WO02080773 A1 WO 02080773A1
Authority
WO
WIPO (PCT)
Prior art keywords
recited
image
patient
display
image overlay
Prior art date
Application number
PCT/US2002/010663
Other languages
English (en)
Inventor
Gabor Fichtinger
Ken Masamune
Andrew M. Bzostek
Russell H. Taylor
Original Assignee
Johns Hopkins University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Johns Hopkins University filed Critical Johns Hopkins University
Publication of WO2002080773A1 publication Critical patent/WO2002080773A1/fr

Links

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
    • A61B6/52—Devices using data or image processing specially adapted for radiation diagnosis
    • A61B6/5211—Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data
    • A61B6/5229—Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data combining image data of a patient, e.g. combining a functional image with an anatomical image
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B2090/364—Correlation of different images or relation of image positions in respect to the body
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B2090/364—Correlation of different images or relation of image positions in respect to the body
    • A61B2090/365—Correlation of different images or relation of image positions in respect to the body augmented reality, i.e. correlating a live optical image with another image
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A61B2090/372—Details of monitor hardware
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A61B2090/373—Surgical systems with images on a monitor during operation using light, e.g. by using optical scanners
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A61B2090/376—Surgical systems with images on a monitor during operation using X-rays, e.g. fluoroscopy
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A61B2090/376—Surgical systems with images on a monitor during operation using X-rays, e.g. fluoroscopy
    • A61B2090/3762—Surgical systems with images on a monitor during operation using X-rays, e.g. fluoroscopy using computed tomography systems [CT]
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/30—Surgical robots

Definitions

  • the present invention relates to surgical instruments, and, more particularly, to an image overlay system to assist surgical interventions inside conventional cross-sectional imaging systems.
  • augmented reality displays are becoming more popular in the medical field.
  • Roberts' Group has developed a frameless, image-overlay system for a stereoscopic microscope for neurosurgery.
  • Another research group named the Fuchs' Group has developed an augmented reality system under ultrasound image guidance therapy.
  • medical images are visualized using a stereoscopic display, e.g., a polarized shutter system with two small monitors located on the user's head.
  • a stereoscopic display e.g., a polarized shutter system with two small monitors located on the user's head.
  • there are potential errors in alignment and spatial registration that can occur in using this binocular system approach.
  • DiGioia and others have developed a system that requires cumbersome calibration and both the patient and operating surgeon to be instrumented with tracking devices to ensure proper registration of image, patient, and viewer at all times.
  • Masamune and others have developed a stand-alone 2D image overlay mechanism for CT and MRI images, but with no physical attachment to the imaging device.
  • the perceived location of the overlay image is independent of the viewer's location.
  • Masumune's system is not intra-operative and it requires post-imaging registration between the patient and the image overlay device.
  • the system uses multiple image slices and modalities, but it does not contain a surgical placement mechanism.
  • the ultrasonic transducer is rigidly fixed with respect to the image overlay device to preserve registration.
  • the system provides real-time imagery, but it does not allow for multiple image directions and locations, and it does not include a surgical instrument placement mechanism.
  • Each of the currently known systems is affected by one or more of the following problems: (1) the patient and surgeon are both instrumented with tracking devices; (2) there is a lack of ability to match intra-operative images into the physical patient; (3) they require cumbersome calibration and quality control; (4) they have a lack of an ability to use arbitrary image planes; and (5)they lack a surgical device placement mechanism.
  • the present invention solves the foregoing problems by displaying patient slice images so that they appear to be physically positioned with respect to the patient at the point at which they were taken and by facilitating the placement of surgical devices into the patient via a surgical instrument placement mechanism.
  • an object of the present invention to provide an intra- operative visualization system that allows simultaneous viewing of the surgical field as seen by the surgeon and the patient's internal anatomy. It is also an object of the present invention to provide an image overlay system that provides accurate depth perception and overlay of image slices on the patient's body. It is a further object of the present invention to provide a system that makes a CT image slice "appear" in the exact same location where it was acquired, while the patient is still in the CT scanner.
  • the present invention is a simple, safe and inexpensive image overlay system to assist surgical interventions inside a conventional CT scanner or other cross-sectional imaging system, such as a magnetic resonance imaging (“MRI”) system, positron emission tomography (“PET”), or single photon emission computed tomography (“SPECT”) system.
  • the overlay system of the present invention is mounted non-invasively on the gantry of the CT scanner.
  • the image overlay system includes a flat liquid crystal display (“LCD”), a half- mirror, and a passive holding attachment that supports the display and mirror and provides seven degrees-of-freedom for proper placement of the display and mirror in the CT scanner or other cross-sectional imaging system scanner.
  • the image overlay system also includes a mechanism to facilitate placement and insertion into a patient of a surgical device.
  • a pre-operative calibration process is used to spatially register the display, half-mirror, and imaging plane of the scanner using a rigid calibration object. Following the calibration, the patient is brought into the scanner, and images acquired are sent to the overlay display. Looking at the patient through the half-mirror, the CT slice image appears to be floating inside the patient in correct size and position. This vision enables the surgeon to see both the surface and the inside of the patient at the same time, and can be used by the surgeon in guiding a surgical intervention. Surgical devices are navigated in the visual field of view. Invasive devices are visible up to the point they enter the body.
  • a new image slice needs to be acquired and displayed on the overlay, provided the given imaging modality is able to visualize the surgical device.
  • CT scanner can visualize metal needles.
  • the perceived location of the overlay image is independent of the viewer's location. Thus, the surgeon and patient are not instrumented with any type of tracking devices.
  • FIGURE 1 is a schematic drawing of the basic physical layout of the image overlay system of the present invention.
  • FIGURE 2 is a schematic drawing of the image overlay system of the present invention in which multiple CT slices are obtained by vertical translation of the display to compensate for organs that are larger than one slice.
  • FIGURE 3 is a schematic drawing of the image overlay system of the present invention in which multiple CT slices are obtained by horizontal translation of the mirror to compensate for organs that are larger than one slice.
  • FIGURE 4 is a schematic drawing of the image overlay system of the present invention in which image panning by horizontal translation of the display is achieved to compensate for a body that is too large to fit on the display.
  • FIGURE 5 is a schematic drawing of the image overlay system of the present invention in which patient interference with the system is resolved by a coordinated tilting of the mirror and the display.
  • FIGURE 6 is a schematic drawing of the image overlay system of the present invention in which tilted CT slice images are obtained using a tilting of the CT gantry.
  • FIGURE 7 is a schematic drawing of the image overlay system of the present invention in which an overlay of synthetic (re-sampled) angled CT slices are obtained by tilting the display.
  • FIGURE 8 is a schematic drawing of the image overlay system of the present invention in which an overlay of synthetic (re-sampled) angled CT slices are obtained by tilting the mirror.
  • FIGURE 9 is a schematic drawing of the image overlay system of the present invention in which a robot surgical device placement mechanism is remotely controlled by a joystick control.
  • FIGURE 10 is a schematic drawing of the image overlay system of the present invention in which a robot surgical device placement mechanism is remotely controlled by a personal computer.
  • FIGURE 11 is a schematic drawing of the image overlay system of the present invention in which the surgical device placement mechanism is a mechanical device that is manually actuated by a surgeon.
  • FIGURE 12 is a schematic drawing of the image overlay system of the present invention in which the surgical device is placed by the surgeons free hand.
  • FIGURE 13 is a schematic drawing of the image overlay system of the present invention in which a tracking device is used to assist the surgeon in placement of the surgical device by free hand.
  • FIGURE 14 is a schematic drawing of the image overlay system of the present invention in which a tracking device is used with a mechanical surgical device placement mechanism that is manually actuated by the surgeon.
  • FIGURE 15 is a schematic drawing of the image overlay system of the present invention in which a tracking device transmits positioning information to be visually superimposed on the virtual overlay image to assist the surgeon in placing a surgical device by free hand.
  • FIGURE 16 is a schematic drawing of the image overlay system of the present invention in which a tracking device transmits positioning information to be visually superimposed on the virtual overlay image to assist the surgeon in placing a surgical device using a mechanical placement device.
  • FIGURE 17 is a schematic drawing of the image overlay system of the present invention in which data from an encoded instrument placement mechanism is visually superimposed on the virtual overlay image to assist the surgeon in placing a surgical device using the placement mechanism.
  • FIGURE 18 is a schematic drawing of the image overlay system of the present invention in which data from the computed tomography scanner is registered with multi-modal pre-operative images so that a composite image is sent to the image overlay system's display.
  • FIGURE 19 is a schematic drawing of the image overlay system of the present invention showing a first step in an image overlay calibration procedure.
  • FIGURE 20 is a schematic drawing of the image overlay system of the present invention showing a second step in an image overlay calibration procedure.
  • FIGURE 21 is a schematic drawing of the image overlay system of the present invention showing a third step in an image overlay calibration procedure.
  • FIGURE 22 is a schematic drawing of the image overlay system of the present invention using passive calibration for placement of the surgical device mechanism.
  • FIGURE 23 is a schematic drawing of the image overlay system of the present invention using active calibration for placement of the surgical device mechanism.
  • FIGURE 24 is a schematic drawing of the basic physical layout of the image overlay system of the present invention shown in FIGURE 1, except that the image display has an orientation other than the horizontal orientation shown in FIGURE 1.
  • FIGURE 25 is a perspective view of the image overlay system of the present invention shown in FIGURE 24.
  • FIGURE 26 is a perspective view of a holding attachment with seven degrees-of-freedom used to mount the image display in the arrangement of the image overlay system shown in FIGURE 24.
  • FIGURE 27 is a perspective view of a triangle marker used in a system registration procedure positioned on the scanner bed of the CT scanner and adjusted to the laser of the CT scanner.
  • FIGURE 28 is a computer generated image of a slice of the triangle marker taken by the CT scanner.
  • FIGURE 29 is a perspective view of the triangle marker used in the system registration procedure with the overlay image of the slice of the triangle marker taken by the CT scanner being viewed through the half transparent mirror.
  • FIGURE 1 is a schematic drawing of the basic layout of the image overlay system 10 of the present invention.
  • the system 10 is designed to be used with a computed tomography (“CT”) scanner system 11 or other cross- sectional imaging system, such as a magnetic resonance imaging (“MRI”) system, positron emission tomography (“PET”), or single photon emission computed tomography (“SPECT”) system.
  • CT computed tomography
  • MRI magnetic resonance imaging
  • PET positron emission tomography
  • SPECT single photon emission computed tomography
  • the system can also be used with ultrasound and thermal imaging systems.
  • the present invention will be described with respect to a CT system, which typically includes a CT gantry 12 and a CT table 14.
  • the system of the present invention allows a surgeon 18 to more precisely place a surgical device 32 in a patient 16 laying on CT table 14 using a CT slice image of patient 16 for guidance.
  • Image overlay system 10 includes a display 22 for displaying the slice image of patient 16 obtained by CT scanner system 11, a half transparent mirror 24 for reflecting the slice image shown on display 22, a surgical device placement mechanism 30 to assist in the placement of surgical device 32 in patient 16, and a mounting device 26 mounted on top of CT gantry 12.
  • Mounting device 26 is used to support display 22, mirror 24 and placement mechanism 30.
  • Half- mirror 24 is preferably either a half silver mirror or a light brown acrylic plate.
  • CT scanner system 11 has a CT image plane 20 in which slice images of patient 16 are taken.
  • mirror 24 is oriented with respect to image plane 20 at an angle 25 that is nominally 45°.
  • Display 22 is then oriented with respect to mirror 24 at an angle 23 that is the same as angle 25, and thus, nominally 45°.
  • Surgical device placement mechanism 30 shown in the embodiment of FIGURE 1 is a mechanical device that constrains surgical device 32 in a plane that coincides with the laser marker 21 of CT gantry 12.
  • placement mechanism 30 is designed to allow surgical device 32 to be moved in the x and y directions of the constrained plane and rotated within the same plane so that it can be properly inserted in patient 16.
  • placement mechanism 30 includes a plurality of arms that are under tension for support and that are interconnected by a plurality of pivot joints.
  • Surgical instrument 32 can be a needle or other type of surgical instrument, such as a drill, trocar, biopsy needle, injection device or tissue ablator, by way of example.
  • FIGURE 24 is another drawing of the basic layout of image overlay system 10 of the present invention, except that in FIGURE 24 the slice image display 22 has a tilted orientation that is different from the generally horizontal orientation of display 22 shown in FIGURE 1.
  • the half transparent mirror 24 also shown in FIGURE 24 thus has a different orientation and position from the position and orientation of mirror 24 shown in FIGURE 1 to accommodate the change in display 22's orientation.
  • FIGURE 25 is a perspective view of the embodiment of image overlay system 10 shown in FIGURE 24.
  • the relative angle between display 22 and mirror 24 is not the nominal 45 degrees.
  • the display 22 and mirror 24 were tilted, in order to give sufficient room for the patient and sufficient workspace for the surgeon. This solution is further explained later in Figure 5.
  • mounting device 26 is detachably placed over CT gantry 12 as in the embodiment shown in FIGURE 1.
  • a holding attachment 78 attached to the mounting device 26 shown in FIGURES 24 and 25 is a holding attachment 78 on which are mounted display 22 and half-mirror 24.
  • holding attachment 78 includes a u-shaped bracket 80 that is positioned over a cross brace 81 of mounting device 26.
  • Bracket 80 Canterlevered from bracket 80 is a shaft 82 on which a parallel link 84 is rotatably mounted at a first pivot joint 86.
  • a second pivot joint 88 At the opposite end of parallel link 84 is a second pivot joint 88 to which is connected a third pivot joint 92 supporting a mounting plate 94 on which display 22 is mounted.
  • mounting plate 94 includes a circular groove 90 in which is rotatably mounted a cross arm 96 forming part of a frame 76 that supports display 22 and half-mirror 24.
  • cross arm 96 Mounted at the ends of cross arm 96 are two vertically oriented arms 97 on which mirror 24 is rotatably mounted. As can be seen in FIGURE 26, holding attachment 78 provides seven degrees of freedom (indicated by arrows) for the placement and orientation of display 22 and half-mirror 24 with respect to a patient 16 lying on CT table 14 of CT scanner system 11.
  • Holding attachment 78 allows mirror 24 of the overlay system 10 to be accurately positioned on the center of CT gantry 12, without making alterations to CT scanner 11 in violation of safety regulations. Holding attachment 78 also allows the overlay system 10 to be easily installed on a variety of CT gantries of different shapes.
  • FIGURES 27 to 29 describe the preferred registration method used with the present invention.
  • a triangle-shaped fiducial marker 100 is placed on CT table 14 and adjusted to the laser marker 21 of CT scanner system 11 which shows the plane of acquisition of system 11.
  • triangle marker 100 is made from a clear material and has a longitudinal dimension that allows a slice image of marker 100 to be taken by scanner 11. After a slice image of triangle marker 100 is taken, the image is transferred, preferably via DICOM, to a personal computer 50 (FIGURE 24) used for image processing.
  • the resolution of display 22 is then pre-calibrated to obtain a full sized image, after which the slice image is transferred to computer 50 and adjusted to its full size.
  • the image is then sent to display 22 and adjusted while looking through mirror 24 to the real image of triangle marker 100 using the edges of triangle marker 100 and surgeon 18's perception.
  • the calibration object triangle marker 100 is then replaced with patient 16, while making sure that mirror 24 and display 22 are not moved.
  • the image overlay system 10 is designed to make a CT slice image 75 appear in the exact same location where it was acquired from patient 16, while patient 16 is still in CT scanner 11.
  • the physician 18 can see the surface of patient 16 and the inside of the body of patient 16 at the same time. This helps the physician 18 to find the optimal access to a target within patient 16, while minimizing collateral damage to normal, healthy structures around the target.
  • Slice image 75 is not projected into the surgical field. It is merely made to appear as though it is there.
  • the central piece of system 10 is half-mirror 24.
  • Half-mirror 24 allows the scene in the surgical field behind it to be seen at the same time as reflections in half-mirror 24.
  • CT scanner 11 acquires a CT slice image of patient 16.
  • Slice image 75 is rendered on display 22 which preferably is a distortion-free flat panel display.
  • Display 22 is positioned above half-mirror 24, so that the reflection of slice image 75 coincides with patient 16's body seen behind mirror 24. Given the previous registration of the real view and the slice image 75 in mirror 24 using triangle 100, the result is an optically stable alignment of the real view and the slice image 75 in mirror 24, provided that mirror 24, display 22, and scanner 11 have been properly aligned.
  • FIGURES 2 and 3 depicted therein are two schemes for using multiple CT slice images to view organs that are larger than a single CT slice image without physically moving patient 16.
  • display 22 is moved some vertical distance d from its original position, either toward or away from patient 16, to a new position.
  • another overlay image 28A is displayed a horizontal distance d away (commensurate with the change in position of display 22) from the position at which the original overlay image 28 was displayed.
  • This allows a second CT slice image of patient 16 taken at a position different from where the original image slice was taken to be displayed at the proper location with respect to patient 16.
  • half-mirror 24 is moved some horizontal distance d from its original position, either toward or away from patient 16, to a new position.
  • second overlay image 28A is displayed at a new position a horizontal distance d away (commensurate with the change in position of half- mirror 24) from the position at which original overlay image 28 was displayed.
  • this allows the second CT slice image of patient 16 to be displayed at the proper location with respect to patient 16.
  • FIGURE 4 discloses a scheme for image panning by parallel translation of display 22 or by moving the slice image on display 22 via software control to accommodate a body that is too large to fit on display 22.
  • display 22 is translated in a horizontal plane a distance d.
  • overlay image 28 is translated in a vertical plane by the same distance d as display 22 is moved.
  • FIGURE 5 shows an arrangement for resolving interference of the image overlay system 10 with a patient 16.
  • half-mirror 24 is tilted so as to form with respect to image plane 20 an angle 25A that is equal to some angle °.
  • the angular relationship between half-mirror 24 and display 22 must be maintained.
  • the angular relationship between display 22 and half- mirror 24 must also be an angle 23A that is equal to ⁇ °.
  • FIGURE 6 discloses an arrangement for using tilted CT slice images that are obtained from patient 16 by tilting CT gantry 12. Tilting the CT gantry is achieved by a standard mechanism commonly available on contemporary CT scanners. This feature is not typically available in other cross sectional imaging modalities. Such an arrangement could be used, for example, to obtain a different perspective on a patient's spine which is typically curved, and which may be further curved due to a disease such as kyphosis or scoliosis. Using the tilted CT slice images allows physician 18 to follow the curvature of a patient's spine while obtaining a single CT slice images of the spine. In the embodiment shown in FIGURE 6, the angular relationship between display 22 and half- mirror 24 is maintained.
  • FIGURES 7 and 8 show embodiments of image overlay system 10 in which tilted CT slice images are obtained from synthetic (or re-sampled) CT slice images that are oblique cuts of a computer-generated 3D image or digital representation of some volume within patient 16.
  • display 22 is tilted at some angle 44 equal to ⁇ ° to achieve a rotation of overlay image 28 with respect to image plane 20 at an angle 45 that is also equal to ⁇ °, but in the opposite direction.
  • half-mirror 24 is tilted so as to produce a different angle 23B with respect to display 22 that is equal to some angle ⁇ °. The result is an angle 25B between half-mirror 24 and overlay image 28.
  • Both of the embodiments shown in FIGURES 7 and 8 use a computer-generated 3D image to form a new cross-section that is the overlay image 28.
  • FIGURES 9 and 10 show embodiments of image overlay system 10 in which the surgical device placement mechanism is a robot device 30A that is remotely controlled.
  • the surgical device placement mechanism is a robot device 30A that is remotely controlled.
  • the surgeon 18 uses a joystick control unit 48 that is connected to robot placement mechanism 30A through a cable 56.
  • surgeon 18 uses a desktop computer 50 that is connected to robot placement mechanism 30A through a cable 57.
  • robot placement mechanism 30A can be a device that is, for example, electromechanical, pneumatic, ultrasonic, or some other control arrangement.
  • robot placement mechanism 30A is designed to allow surgical instrument 32 to be moved in the X,Y directions of a constrained plane and rotated within that same plane so that it can be properly inserted in patient 16.
  • FIGURE 11 shows an embodiment of image overlay system 10 in which a passive mechanism 30 for the placement of a surgical device 32 is used.
  • placement mechanism 30 is a mechanical device that is designed to allow surgical instrument 32 to be moved in the X and Y directions of a constrained plane and rotated within that same plane so that it can be properly inserted in patient 16.
  • Placement mechanism 30 is positioned by surgeon 18's hand 52 into the proper location and orientation for insertion of surgical instrument 32 into patient 16.
  • FIGURE 12 shows image overlay system 10 without placement mechanism 30.
  • physician 18 holds surgical instrument 32 in his hand 52 and positions instrument 32 for insertion into patient 16 using the laser marker 21 of the CT scanner 11 as a visual guide for proper placement and orientation of instrument 32 prior to insertion.
  • FIGURE 13 also shows image overlay system 10 without placement mechanism 30.
  • physician 18 again holds surgical instrument 32 in his hand 52 and positions instrument 32 for insertion into patient 16.
  • the physician uses a tracking system that includes a base unit 54 attached to mounting device 26 and a transmission unit 56 attached to surgical instrument 32.
  • a tracking system that includes a base unit 54 attached to mounting device 26 and a transmission unit 56 attached to surgical instrument 32.
  • Several commercially available tracking devices can be used for this purpose, such as Polaris (Northern Digital, Canada) , Flashpoint (Image Guided Technologies, Boulder, CO).
  • Tracking information generated by units 54 and 56 is available to surgeon 18 on personal computer 50 that is connected to base unit 54 through a cable 57 running between base unit 54 and computer 50.
  • surgeon 18 uses the tracking data available on computer 50 as a guide for proper positioning and orientation of instrument 32 prior to insertion in patient 16.
  • the position data is numerical data reflecting the coordinate location of the surgical instrument 32 with respect to the patient 16.
  • FIGURE 14 again shows the image overlay system 10 of the present invention with a tracking system that includes a base unit 54 attached to mounting device 26 and a transmission unit 56 attached to surgical instrument 32.
  • system 10 also includes a surgical device placement mechanism 30 for holding and positioning surgical instrument 32.
  • Tracking information generated by units 54 and 56 is again available to surgeon 18 on computer 50, which is connected to base unit 54 through cable 57.
  • surgeon 18 uses the tracking data available on computer 50 as a guide for positioning and orientation of placement mechanism 30 and instrument 32 held by mechanism 30 prior to insertion in patient 16.
  • the position data generated by tracking device units 54 and 56 is numerical data reflecting the coordinate location of the surgical instrument 32 with respect to the patient 16.
  • FIGURE 15 shows image overlay system 10 without the placement mechanism 30, but with tracking units 54 and 56.
  • physician 18 again holds surgical instrument 32 in his hand 52 and positions instrument 32 for insertion into patient 16 using tracking base unit 54 attached to mounting device 26 and transmission unit 56 attached to surgical instrument 32.
  • Tracking information generated by units 54 and 56 is again displayed on personal computer 50, but also overlaid on overlay image 28. This is accomplished by a transfer of such information from computer 50 to display 22 through cable 58 connecting computer 50 to display 22.
  • the placement of the tracking data on overlay image 28 readily allows surgeon 18 to use the tracking data as a guide for proper positioning and orientation of instrument 32 prior to insertion in patient 16.
  • the position data is numerical data reflecting the coordinate location of the surgical instrument 32 with respect to the patient 16.
  • the embodiment of image overlay system 10 shown in FIGURE 16 again uses a tracking system that includes base unit 54 attached to mounting device 26 and transmission unit 56 attached to surgical instrument 32.
  • Tracking information generated by units 54 and 56 is again available to surgeon 18 on computer 50 and overlaid on overlay image 28. This is accomplished by a transfer of such information from computer 50 to display 22 through cable 58 connecting computer 50 to display 22.
  • the placement of the tracking data on overlay image 28 readily allows surgeon 18 to use the tracking data as a guide for proper positioning and orientation of placement mechanism 30 and instrument 32, held by mechanism 30, prior to insertion in patient 16.
  • the position data generated by tracking device units 54 and 56 is numerical data reflecting the coordinate location of the surgical instrument 32 with respect to the patient 16.
  • FIGURE 17 shows an embodiment of the image overlay system of the present invention in which coordinate data from an encoded instrument placement mechanism 30B is visually superimposed on virtual overlay image 28 to assist surgeon 18 in the placement of surgical device 32 in patient 16.
  • encoded placement mechanism 30B constrains surgical device 32 in a plane that coincides with the laser marker 21 of CT scanner 11.
  • Placement mechanism 30B is also designed to allow surgical device 32 to be moved in the X, Y directions of the constrained plane and rotate within the same plane so that device 32 can be properly inserted in patient 16.
  • this embodiment of placement mechanism 3 OB there is also included an encoding mechanism for providing the X, Y coordinates of device 32 and coordinates corresponding to its rotational orientation.
  • Encoded instrument placement mechanism 30B can be a passive placement device or robot of some sort, not specified further here.
  • the coordinate data from encoded instrument placement mechanism 3 OB is sent to personal computer 50, and then transmitted to display 22 so that it can be visually superimposed on virtual overlay image 28.
  • FIGURE 18 shows an embodiment of image overlay system 10 in which slice images 60 from CT scanner 11 are registered with multi-modal pre- operative slice images 62 and 64 from, for example, MRI and PET scanners, respectively, so that a composite, more comprehensive slice image is sent to display 22 for overlay image 28C .
  • Composite overlay images are desirable because different types of scans have different advantages. For example, CT scans are good for hard tissue, such as bones and ligaments. In contrast, soft tissue differentiation in CT scans is relatively poor, whereas it is good in MRI scans.
  • the auxiliary scans are done pre-operatively, then spatially registered to the coordinate space of the intra-operative CT images, by some standard means medical image registration not specified further in this document.
  • FIGURES 19, 20 and 21 show a robust calibration procedure without the use of a calibration object.
  • Step 1 of the calibration procedure is shown in FIGURE 19.
  • Step 1 is to confirm that display 22 is horizontal, CT gantry 12 is vertical, and half-mirror 24 is oriented at 45° with respect to image plane 20.
  • a bubble level 66 as shown in FIGURE 19, can be used for this purpose.
  • Step 2 of the calibration procedure which is shown in FIGURE 20, is to translate monitor 22 or half-mirror 24 off-plane to align the current image plane with CT scanner 1 l's laser marker 21.
  • a CT slice image of patient 16 is taken and the height of display 22 is adjusted some distance d until the plane of overlay image 28 coincides with the plane of the axial laser marker 21 of scanner 11.
  • Step 3 of the image overlay calibration procedure can be explained in FIGURE 21.
  • display 22 is translated in plane, or the image on display 22 is translated, until the overlay image 28 is aligned with patient 16's body.
  • the contour of patient 16's body is highlighted in the image on display 22 using image processing.
  • fiducial marks (not shown) can be applied to the body of patient 16 prior to taking CT images. .
  • the fiducials are then highlighted using the imaging processor of display 22.
  • Device placement mechanism 30 may also be instrumented with similar fiducials.
  • Display 22 is then adjusted within plane motion or the image on display 22 is adjusted with panning via software control without moving display 22 until overlay image 28 matches the contour of patient 16's body and all selected fiducials and anatomical landmarks.
  • FIGURE 22 shows passive surgical device placement mechanism calibration.
  • CT gantry 12 is brought to a vertical position.
  • Bubble level 66 (as shown in Figure 19) is used to confirm and insure the vertical position of CT gantry 12.
  • the laser light of scanner 1 l's laser marker 21 in the axial image plane must be cast in the central line of surgical device placement mechanism 30.
  • FIGURE 23 shows a calibration method applicable to encoded surgical device placement mechanism, either passive or active robotic mechanism.
  • the method employs rigid body fiducial based stereotactic registration system attached to the surgical device placement mechanism.
  • the rigid fiducial frame is assembled from several rods, in a known geometrical pattern. Upon imaging, the rods are cut across the CT slice and the mark of each rod is detected in the CT image. Based on the relative position of these marks and the a priori known relative spatial position of the rods, one can calculate the spatial location of the fiducial frame with respect to the CT image. This completes the registration process.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Surgery (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Veterinary Medicine (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Public Health (AREA)
  • Animal Behavior & Ethology (AREA)
  • Pathology (AREA)
  • Radiology & Medical Imaging (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Physics & Mathematics (AREA)
  • Biophysics (AREA)
  • High Energy & Nuclear Physics (AREA)
  • Optics & Photonics (AREA)
  • Gynecology & Obstetrics (AREA)
  • Robotics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

Ce système (10) est conçu pour être utilisé avec un système de tomodensitomètre (11), et permet à un chirurgien de placer avec précision un dispositif chirurgical (32) dans un patient, au moyen d'un miroir semi-transparent (24) permettant de réfléchir une image en coupe affichée sur un écran (22) d'aide au placement du dispositif chirurgical (32).
PCT/US2002/010663 2001-04-05 2002-04-05 Appareil de realite amplifiee et procede de tomographie assistee par ordinateur (ct) WO2002080773A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US28162201P 2001-04-05 2001-04-05
US60/281,622 2001-04-05

Publications (1)

Publication Number Publication Date
WO2002080773A1 true WO2002080773A1 (fr) 2002-10-17

Family

ID=23078089

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2002/010663 WO2002080773A1 (fr) 2001-04-05 2002-04-05 Appareil de realite amplifiee et procede de tomographie assistee par ordinateur (ct)

Country Status (1)

Country Link
WO (1) WO2002080773A1 (fr)

Cited By (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE10238011A1 (de) * 2002-08-20 2004-03-11 GfM Gesellschaft für Medizintechnik mbH Semitransparenter Bildschirm für AR-Anwendungen
WO2005030073A1 (fr) * 2003-09-29 2005-04-07 Sepitec Foundation Dispositif pour placer des instruments ou des implants dans un organe corporel
US7133003B2 (en) 2004-08-05 2006-11-07 Searete Llc Cosmetic enhancement mirror
EP1743578A1 (fr) * 2005-07-14 2007-01-17 General Electric Company Méthode et appareil pour la présentation d'informations en radiologie
US7259731B2 (en) 2004-09-27 2007-08-21 Searete Llc Medical overlay mirror
US7283106B2 (en) 2004-08-02 2007-10-16 Searete, Llc Time-lapsing mirror
US7429966B2 (en) 2004-08-05 2008-09-30 Searete, Llc Cosmetic enhancement mirror
US7636072B2 (en) 2004-08-02 2009-12-22 Searete Llc Cosmetic enhancement mirror
US7657125B2 (en) 2004-08-02 2010-02-02 Searete Llc Time-lapsing data methods and systems
US7663571B2 (en) 2004-08-02 2010-02-16 Searete Llc Time-lapsing mirror
US7671823B2 (en) 2004-08-02 2010-03-02 Searete Llc Multi-angle mirror
US7679581B2 (en) 2004-08-02 2010-03-16 Searete Llc Medical overlay mirror
US7679580B2 (en) 2004-08-02 2010-03-16 Searete Llc Time-lapsing mirror
US7683858B2 (en) 2004-08-02 2010-03-23 Searete Llc Cosmetic enhancement mirror
US7688283B2 (en) 2004-08-02 2010-03-30 Searete Llc Multi-angle mirror
US7705800B2 (en) 2004-09-15 2010-04-27 Searete Llc Multi-angle mirror
US7714804B2 (en) 2004-09-15 2010-05-11 Searete Llc Multi-angle mirror
US7876289B2 (en) 2004-08-02 2011-01-25 The Invention Science Fund I, Llc Medical overlay mirror
US9155373B2 (en) 2004-08-02 2015-10-13 Invention Science Fund I, Llc Medical overlay mirror
US9993273B2 (en) 2013-01-16 2018-06-12 Mako Surgical Corp. Bone plate and tracking device using a bone plate for attaching to a patient's anatomy
CN109475387A (zh) * 2016-06-03 2019-03-15 柯惠Lp公司 用于控制机器人外科手术装置和观看者自适应立体显示器的方面的系统、方法和计算机可读存储媒体
US10531925B2 (en) 2013-01-16 2020-01-14 Stryker Corporation Navigation systems and methods for indicating and reducing line-of-sight errors
US10537395B2 (en) 2016-05-26 2020-01-21 MAKO Surgical Group Navigation tracker with kinematic connector assembly
US10832486B1 (en) 2019-07-17 2020-11-10 Gustav Lo Systems and methods for displaying augmented anatomical features
US11020144B2 (en) 2015-07-21 2021-06-01 3Dintegrated Aps Minimally invasive surgery system
US11033182B2 (en) 2014-02-21 2021-06-15 3Dintegrated Aps Set comprising a surgical instrument
US11039734B2 (en) 2015-10-09 2021-06-22 3Dintegrated Aps Real time correlated depiction system of surgical tool
US20220015726A1 (en) * 2020-02-26 2022-01-20 Siemens Medical Solutions Usa, Inc. Mobile tomography imaging
CN113951913A (zh) * 2021-10-18 2022-01-21 中南大学湘雅医院 一种基于四维ct融合的pet-ct图像扫描用定位机架
US11311175B2 (en) 2017-05-22 2022-04-26 Gustav Lo Imaging system and method
US11331120B2 (en) 2015-07-21 2022-05-17 3Dintegrated Aps Cannula assembly kit
US11663735B2 (en) 2020-09-09 2023-05-30 Canon Medical Systems Corporation Optical markers for calibration/alignment of medical diagnostic devices
US12059804B2 (en) 2019-05-22 2024-08-13 Mako Surgical Corp. Bidirectional kinematic mount

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5772593A (en) * 1995-07-12 1998-06-30 Fuji Photo Film Co., Ltd. Surgical operation aiding system
US6314311B1 (en) * 1999-07-28 2001-11-06 Picker International, Inc. Movable mirror laser registration system

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5772593A (en) * 1995-07-12 1998-06-30 Fuji Photo Film Co., Ltd. Surgical operation aiding system
US6314311B1 (en) * 1999-07-28 2001-11-06 Picker International, Inc. Movable mirror laser registration system

Cited By (48)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE10238011A1 (de) * 2002-08-20 2004-03-11 GfM Gesellschaft für Medizintechnik mbH Semitransparenter Bildschirm für AR-Anwendungen
WO2005030073A1 (fr) * 2003-09-29 2005-04-07 Sepitec Foundation Dispositif pour placer des instruments ou des implants dans un organe corporel
US7688283B2 (en) 2004-08-02 2010-03-30 Searete Llc Multi-angle mirror
US7679580B2 (en) 2004-08-02 2010-03-16 Searete Llc Time-lapsing mirror
US7259732B2 (en) 2004-08-02 2007-08-21 Searete Llc Cosmetic enhancement mirror
US9155373B2 (en) 2004-08-02 2015-10-13 Invention Science Fund I, Llc Medical overlay mirror
US7283106B2 (en) 2004-08-02 2007-10-16 Searete, Llc Time-lapsing mirror
US8831300B2 (en) 2004-08-02 2014-09-09 The Invention Science Fund I, Llc Time-lapsing data methods and systems
US7636072B2 (en) 2004-08-02 2009-12-22 Searete Llc Cosmetic enhancement mirror
US7657125B2 (en) 2004-08-02 2010-02-02 Searete Llc Time-lapsing data methods and systems
US7663571B2 (en) 2004-08-02 2010-02-16 Searete Llc Time-lapsing mirror
US7671823B2 (en) 2004-08-02 2010-03-02 Searete Llc Multi-angle mirror
US7679581B2 (en) 2004-08-02 2010-03-16 Searete Llc Medical overlay mirror
US7952537B2 (en) 2004-08-02 2011-05-31 The Invention Science Fund I, Llc Medical overlay mirror
US7683858B2 (en) 2004-08-02 2010-03-23 Searete Llc Cosmetic enhancement mirror
US7876289B2 (en) 2004-08-02 2011-01-25 The Invention Science Fund I, Llc Medical overlay mirror
US7692606B2 (en) 2004-08-02 2010-04-06 Searete Llc Medical overlay mirror
US7133003B2 (en) 2004-08-05 2006-11-07 Searete Llc Cosmetic enhancement mirror
US7429966B2 (en) 2004-08-05 2008-09-30 Searete, Llc Cosmetic enhancement mirror
US7705800B2 (en) 2004-09-15 2010-04-27 Searete Llc Multi-angle mirror
US7714804B2 (en) 2004-09-15 2010-05-11 Searete Llc Multi-angle mirror
US7259731B2 (en) 2004-09-27 2007-08-21 Searete Llc Medical overlay mirror
EP1743578A1 (fr) * 2005-07-14 2007-01-17 General Electric Company Méthode et appareil pour la présentation d'informations en radiologie
US11369438B2 (en) 2013-01-16 2022-06-28 Stryker Corporation Navigation systems and methods for indicating and reducing line-of-sight errors
US9993273B2 (en) 2013-01-16 2018-06-12 Mako Surgical Corp. Bone plate and tracking device using a bone plate for attaching to a patient's anatomy
US10531925B2 (en) 2013-01-16 2020-01-14 Stryker Corporation Navigation systems and methods for indicating and reducing line-of-sight errors
US12102365B2 (en) 2013-01-16 2024-10-01 Mako Surgical Corp. Bone plate for attaching to an anatomic structure
US11622800B2 (en) 2013-01-16 2023-04-11 Mako Surgical Corp. Bone plate for attaching to an anatomic structure
US10932837B2 (en) 2013-01-16 2021-03-02 Mako Surgical Corp. Tracking device using a bone plate for attaching to a patient's anatomy
US11033182B2 (en) 2014-02-21 2021-06-15 3Dintegrated Aps Set comprising a surgical instrument
US12075981B2 (en) 2014-02-21 2024-09-03 Cilag Gmbh International Set comprising a surgical instrument
US11331120B2 (en) 2015-07-21 2022-05-17 3Dintegrated Aps Cannula assembly kit
US11020144B2 (en) 2015-07-21 2021-06-01 3Dintegrated Aps Minimally invasive surgery system
US11039734B2 (en) 2015-10-09 2021-06-22 3Dintegrated Aps Real time correlated depiction system of surgical tool
US11559358B2 (en) 2016-05-26 2023-01-24 Mako Surgical Corp. Surgical assembly with kinematic connector
US10537395B2 (en) 2016-05-26 2020-01-21 MAKO Surgical Group Navigation tracker with kinematic connector assembly
CN109475387A (zh) * 2016-06-03 2019-03-15 柯惠Lp公司 用于控制机器人外科手术装置和观看者自适应立体显示器的方面的系统、方法和计算机可读存储媒体
CN109475387B (zh) * 2016-06-03 2022-07-19 柯惠Lp公司 用于控制机器人外科手术装置和观看者自适应立体显示器的方面的系统、方法和存储媒体
US11311175B2 (en) 2017-05-22 2022-04-26 Gustav Lo Imaging system and method
US11678789B2 (en) 2017-05-22 2023-06-20 Gustav Lo Imaging system and method
US11998173B1 (en) 2017-05-22 2024-06-04 Gustav Lo Imaging system and method
US12059804B2 (en) 2019-05-22 2024-08-13 Mako Surgical Corp. Bidirectional kinematic mount
US10832486B1 (en) 2019-07-17 2020-11-10 Gustav Lo Systems and methods for displaying augmented anatomical features
US11744534B2 (en) * 2020-02-26 2023-09-05 Siemens Medical Solutions Usa, Inc. Mobile tomography imaging
US20220015726A1 (en) * 2020-02-26 2022-01-20 Siemens Medical Solutions Usa, Inc. Mobile tomography imaging
US11663735B2 (en) 2020-09-09 2023-05-30 Canon Medical Systems Corporation Optical markers for calibration/alignment of medical diagnostic devices
CN113951913B (zh) * 2021-10-18 2023-06-16 中南大学湘雅医院 一种基于四维ct融合的pet-ct图像扫描用定位机架
CN113951913A (zh) * 2021-10-18 2022-01-21 中南大学湘雅医院 一种基于四维ct融合的pet-ct图像扫描用定位机架

Similar Documents

Publication Publication Date Title
WO2002080773A1 (fr) Appareil de realite amplifiee et procede de tomographie assistee par ordinateur (ct)
US20230263586A1 (en) Systems and methods for surgical navigation, including image-guided navigation of a patient's head
ES2292593T3 (es) Sistema de guiado.
CA2003497C (fr) Visionnement d'une image anatomique correle par sonde
US7570987B2 (en) Perspective registration and visualization of internal areas of the body
Blackwell et al. Augmented reality and its future in orthopaedics.
US5526812A (en) Display system for enhancing visualization of body structures during medical procedures
US7559895B2 (en) Combining tomographic images in situ with direct vision using a holographic optical element
US8509503B2 (en) Multi-application robotized platform for neurosurgery and resetting method
EP1103229B1 (fr) Système et méthode utilisant des dispositifs d'imagerie pour simplifier la planification des procédures chirurgicales
CA2420607C (fr) Systeme et procede de fusion d'images tomographiques avec la vision humaine
BR112020022649A2 (pt) sistema de orientação de navegação holográfica 3d ao vivo para realizar procedimento de intervenção
US20020082498A1 (en) Intra-operative image-guided neurosurgery with augmented reality visualization
JP2019532693A5 (fr)
Navab et al. Laparoscopic virtual mirror new interaction paradigm for monitor based augmented reality
EP2438880A1 (fr) Système de projection d'images pour la projection d'images sur la surface d'un objet
JPH11313837A (ja) 像誘導式の介在的手順のプランニング方法及び装置
JP2004530485A (ja) ガイドシステムおよびそのためのプローブ
CN114727848A (zh) 用于ent规程的可视化系统和方法
Zhang et al. 3D augmented reality based orthopaedic interventions
ES2242118T3 (es) Registro en perspectiva y visualizacion de las zonas corporales internas.
Adams et al. An optical navigator for brain surgery
AU2021354376A1 (en) Auto-navigating digital surgical microscope
Freysinger et al. A full 3D-navigation system in a suitcase
Navab et al. Laparoscopic Virtual Mirror

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ OM PH PL PT RO RU SD SE SG SI SK SL TJ TM TN TR TT TZ UA UG US UZ VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase

Ref country code: JP

WWW Wipo information: withdrawn in national office

Country of ref document: JP

DPE2 Request for preliminary examination filed before expiration of 19th month from priority date (pct application filed from 20040101)