US20210298835A1 - Surgical navigation system, and imaging method of the same - Google Patents
Surgical navigation system, and imaging method of the same Download PDFInfo
- Publication number
- US20210298835A1 US20210298835A1 US17/212,007 US202117212007A US2021298835A1 US 20210298835 A1 US20210298835 A1 US 20210298835A1 US 202117212007 A US202117212007 A US 202117212007A US 2021298835 A1 US2021298835 A1 US 2021298835A1
- Authority
- US
- United States
- Prior art keywords
- image
- target
- instrument
- camera device
- display
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000003384 imaging method Methods 0.000 title claims description 31
- 230000001575 pathological effect Effects 0.000 claims abstract description 44
- 230000003190 augmentative effect Effects 0.000 claims abstract description 39
- 238000001356 surgical procedure Methods 0.000 claims description 14
- 238000007920 subcutaneous administration Methods 0.000 claims description 6
- 238000000034 method Methods 0.000 claims description 3
- 230000009466 transformation Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000002591 computed tomography Methods 0.000 description 2
- 210000000887 face Anatomy 0.000 description 2
- 210000003127 knee Anatomy 0.000 description 2
- 210000004556 brain Anatomy 0.000 description 1
- 238000000576 coating method Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002595 magnetic resonance imaging Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000000399 orthopedic effect Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0176—Head mounted characterised by mechanical features
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S5/00—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations
- G01S5/16—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations using electromagnetic waves other than radio waves
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0172—Head mounted characterised by optical features
-
- G06K9/00624—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/143—Sensing or illuminating at different wavelengths
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/147—Details of sensors, e.g. sensor lenses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H04N5/2251—
-
- H04N5/247—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/30—Transforming light or analogous information into electric information
- H04N5/33—Transforming infrared radiation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B17/00—Surgical instruments, devices or methods, e.g. tourniquets
- A61B2017/00477—Coupling
- A61B2017/00486—Adaptors for coupling parts with incompatible geometries
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
- A61B2034/2055—Optical tracking systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
- A61B2034/2055—Optical tracking systems
- A61B2034/2057—Details of tracking cameras
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
- A61B2034/2065—Tracking using image or pattern recognition
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2072—Reference field transducer attached to an instrument or patient
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B2090/364—Correlation of different images or relation of image positions in respect to the body
- A61B2090/365—Correlation of different images or relation of image positions in respect to the body augmented reality, i.e. correlating a live optical image with another image
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/371—Surgical systems with images on a monitor during operation with simultaneous use of two cameras
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/372—Details of monitor hardware
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/373—Surgical systems with images on a monitor during operation using light, e.g. by using optical scanners
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/39—Markers, e.g. radio-opaque or breast lesions markers
- A61B2090/3983—Reference marker arrangements for use with image guided surgery
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/50—Supports for surgical instruments, e.g. articulated arms
- A61B2090/502—Headgear, e.g. helmet, spectacles
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/50—Supports for surgical instruments, e.g. articulated arms
- A61B2090/5025—Supports for surgical instruments, e.g. articulated arms with a counter-balancing mechanism
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/50—Supports for surgical instruments, e.g. articulated arms
- A61B90/57—Accessory clamps
- A61B2090/571—Accessory clamps for clamping a support arm to a bed or other supports
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0141—Head-up displays characterised by optical features characterised by the informative content of the display
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B2027/0178—Eyeglass type
-
- G06K2209/057—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2210/00—Indexing scheme for image generation or computer graphics
- G06T2210/41—Medical
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
- G06V2201/034—Recognition of patterns in medical or anatomical images of medical instruments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
Definitions
- the disclosure relates to a surgical navigation system, and more particularly to a surgical navigation system that has two tracking units, and an imaging method thereof.
- a conventional surgical navigation system 1 is adapted to provide a pathological image (not shown) to assist a medical staff 10 in operating a surgical instrument 101 to perform a surgery on a diseased portion 103 of a patient 102 , and includes a reference target 11 disposed near the diseased portion 103 of the patient 102 , an instrument target 12 disposed on the surgical instrument 101 , an optical tracking unit 13 , and a screen 14 .
- the reference target 11 is set to a fixed position in the pathological image
- the optical tracking unit 13 is configured to photograph the diseased portion 103 , to calculate a position of the instrument target 12 in an imaging coordinate system of the optical tracking unit 13 according to relative positions of the instrument target 12 and the reference target 11 , and to output an augmented reality (AR) image that has the surgical instrument 101 and the pathological image of the patient 103 to the screen 14 .
- AR augmented reality
- the optical tracking system 13 will be unable to calculate the position of the surgical instrument 101 , and the augmented reality image cannot be provided.
- an object of the disclosure is to provide a surgical navigation system that can improve the stability or reliability of surgical navigation.
- the surgical navigation system is adapted for use with a pathological image and a position identification unit to assist a medical staff in operating a surgical instrument to perform surgery on a subcutaneous diseased portion of a patient.
- the position identification unit includes a reference target detachably fixed on the patient, and an instrument target disposed on the surgical instrument.
- the surgical navigation system includes a first tracking unit, a second tracking unit and a processing unit.
- the first tracking unit includes a first support, a display device, and a first infrared camera device disposed on the first support.
- the first infrared camera device is configured to capture a first image of the position identification unit.
- the second tracking unit includes a second support, and a second infrared camera device disposed on the second support.
- the second infrared camera device is configured to capture a second image of the position identification unit.
- the processing unit is in signal connection with the display device, the first infrared camera device and the second infrared camera device, receives the first image and the second image respectively from the first infrared camera device and the second infrared camera device, and is configured to perform image recognition on the first image and the second image with respect to the reference target and the instrument target, to use, based on a result of the image recognition, one of the first image and the second image to generate an augmented reality image by adding a first target-related pattern that corresponds to the position identification unit into the pathological image, and to output the augmented reality image to the display device for display thereby.
- Another object of the disclosure is to provide an imaging method for surgical navigation.
- the imaging method includes: (A) preparing the surgical navigation system of this disclosure; (B) capturing the first image of the position identification unit by the first infrared camera device, and capturing the second image of the position identification unit by the second infrared camera device; (C) by the processing unit, determining whether the first image has the reference target, whether the first image has the instrument target, whether the second image has the reference target, and whether the second image has the instrument target; (D) by the processing unit, upon determining that one of the first and second images has the reference target and the instrument target in step (C), generating the augmented reality image by adding, based on the one of the first and second images, the first target-related pattern into the pathological image; and (E) by the processing unit, outputting the augmented reality image to the display device for display thereby.
- FIG. 1 is a schematic perspective view illustrating use of a conventional surgical navigation system during a surgical operation
- FIG. 2 is a schematic perspective view illustrating use of an embodiment of a surgical navigation system according to this disclosure during a surgical operation
- FIG. 3 is a perspective view illustrating a reference target of the embodiment
- FIG. 4 is a perspective view illustrating an instrument target of the embodiment
- FIG. 5 is a perspective view illustrating a second tracking unit of the embodiment
- FIG. 6 is a schematic diagram illustrating coordinate systems of different components of the embodiment.
- FIG. 7 is a block diagram illustrating the embodiment
- FIG. 8 is a flow chart illustrating an embodiment of an imaging method for surgical navigation according to this disclosure.
- FIG. 9 is a schematic diagram illustrating that the embodiments according to this disclosure are also applicable to a knee surgery.
- an embodiment of a surgical navigation system is adapted for use with a known pathological image to assist a medical staff 6 in operating a surgical instrument 7 to perform a surgery on a subcutaneous diseased portion 81 of a patient 8 .
- the pathological image may be, but is not limited to, an image conforming with a format of digital imaging and communications in medicine (DICOM), such as an image of computed tomography (CT), an image magnetic resonance imaging (MRI), etc.
- DICOM digital imaging and communications in medicine
- CT computed tomography
- MRI image magnetic resonance imaging
- diseased portion refers to the tissue that is to be operated on during the surgery, or simply the surgical site, regardless of whether the surgery is intended to remove the tissue or a part of the tissue, to reconnect the tissue with other tissue, to insert prosthetics or a transplant, a combination of the above, or for other purposes.
- the surgical navigation system includes a position identification unit 2 , a first tracking unit 3 , a second tracking unit 4 , and a processing unit 5 .
- the position identification unit 2 includes a reference target 21 to be detachably fixed on the patient 8 , and an instrument target 22 disposed on the surgical instrument 7 .
- the reference target 21 has a clamp 211 extending along a first axis L 1 and adapted to be detachably fixed to the patient 8 through the act of clamping, a reference adapter 212 connected to the clamp 211 and having an X-shape, and four reference locator balls 213 connected to the reference adapter 212 .
- the reference locator balls 213 are coplanar to each other and lie on a plane P 1 that is nonparallel with or transverse to the first axis L 1 .
- the arrangement of the reference locator balls 213 of the reference target 21 and the shape of the reference adapter 212 are not limited to the above.
- the reference locator balls 213 may be non-coplanar with each other, and/or the plane P 1 may be parallel to the first axis L 1 ; so long as the relative positions among the reference locator balls 213 are known.
- the instrument target 22 has a fastener 221 adapted to be mounted to the surgical instrument 7 that extends along a second axis L 2 so as to fasten the instrument target to the surgical instrument 7 , an instrument adapter 222 having a Y-shape and connected to the fastener 221 , and four instrument locator balls 223 connected to the instrument adapter 222 .
- the instrument locator balls 223 are coplanar to each other and lie on a plane P 2 that is nonparallel with or transverse to the second axis L 2 .
- a geometrical arrangement of the instrument locator balls 213 of the reference target 21 is different from a geometrical arrangement of the instrument locator balls 223 of the instrument target 22 .
- the arrangement of the instrument locator balls 223 of the instrument target 22 and the shape of the instrument adapter 222 are not limited to the above.
- the instrument locator balls 223 may be noncoplanar with each other, and the plane P 2 may be parallel to the second axis L 2 , so long as the relative positions among the instrument locator balls 213 are known.
- the instrument target 22 is not limited to being one in number, and there may be two or more instrument targets 22 in other embodiments.
- the first tracking unit 3 includes a first support 31 , a display device 32 disposed on the first support 31 , and a first infrared camera device 33 disposed on the first support 31 .
- the first infrared camera device 33 is disposed to face the position identification unit 2 for capturing a first image (not shown) of the position identification unit 2 .
- the first support 31 includes a base 311 operable in one of a movable state and a locked state, a placement frame 312 disposed on the base 311 , and a balance arm 313 connected to the placement frame 312 and having the first infrared camera device 33 disposed thereon.
- the base 311 may include several locking caster wheels, so the base 311 is movable in the movable state where the locking caster wheels are unlocked and is fixed in position in the locked state where the locking caster wheels are locked.
- the display device 32 has a pathological image area 321 for displaying the pathological image, an augmented reality (AR) area 322 for displaying an augmented reality image, and a mixed reality (MR) area 323 for displaying a mixed reality image.
- AR augmented reality
- MR mixed reality
- the display device 32 may be disposed to be separate from the first support 31 .
- the pathological image area 321 , the augmented reality area 322 and the mixed reality area 323 are three areas in the display device 32 , and a user can arbitrarily drag, turn on or off, zoom in or out any one of these areas according to his/her own needs, so the display device 32 can present different display combinations, such as displaying images in parallel or displaying one particular image in full screen.
- the first infrared camera device 33 includes two lenses 331 spaced apart from each other.
- the first infrared camera device 33 may be an infrared-emitting camera as an example.
- the second tracking unit 4 includes a second support 41 , a second infrared camera device 42 disposed on the second support 41 , a visible light camera device 43 , and a display goggle 44 that includes a transparent lens so the medical staff 6 can view a scene therethrough, and is capable of displaying images on the transparent lens.
- the second infrared camera device 42 is disposed to face the position identification unit 2 for capturing a second image (not shown) of the position identification unit 2 from a different angle than the first infrared camera device 41 , and includes two lenses 421 spaced apart from each other.
- the second infrared camera device 42 may be an infrared-emitting camera as an example.
- the visible light camera device 43 is disposed to face the position identification unit 2 for capturing a visible light image of the position identification unit 2 , and includes two lenses 431 spaced apart from each other and disposed between the lenses 421 .
- the second support 41 is a head-mounted device
- the second tracking unit 4 is configured such that, when the medical staff 6 wears the second support 41 , the second infrared camera device 42 and the visible light camera device 43 are disposed above the eyes of the medical staff 6 , and the display goggle 44 is located below the visible light camera device 43 and is in front of the eyes of the medical staff 6 .
- the processing unit 5 is in signal connection with the display device 32 , the first infrared camera device 33 , the second infrared camera device 42 , the visible light camera device 43 and the display goggle 44 , and is configured to receive the first image, the second image and the visible light image respectively from the first infrared camera device 33 , the second infrared camera device 42 and the visible light camera device 43 , and to perform image recognition on the first image and the second image with respect to the reference target 21 and the instrument target 22 (i.e., to identify the reference target 21 and the instrument target 22 from the first image and the second image).
- the signal connection as referred in this embodiment can be realized by a wired connection (e.g., signal lines corresponding to different signal sources) or a wireless connection (e.g., wireless communication technologies such as Bluetooth and wireless network).
- a wired connection e.g., signal lines corresponding to different signal sources
- a wireless connection e.g., wireless communication technologies such as Bluetooth and wireless network.
- the processing unit 5 is configured to use, based on a result of the image recognition on the first image and the second image, one of the first image and the second image to generate an augmented reality image by adding a first target-related pattern that corresponds in position to the position identification unit 2 into the pathological image.
- the processing unit 5 may add a pattern that represents a predetermined surgical instrument (the first target-related pattern) to the pathological image at a position that corresponds to the instrument target 22 (the position may be determined based on the reference target 21 and the instrument target 22 in said one of the first image and the second image), so as to generate the augmented reality image.
- the processing unit 5 outputs the augmented reality image to the display device 32 for display thereby in the AR area 322 thereof.
- the first target-related pattern may be the pattern that represents the predetermined instrument, a pattern that represents the reference target, a pattern that represents the instrument target, or a combination thereof, and this disclosure is not limited in this respect.
- the processing unit 5 may further output, to the display goggle 44 based on a result of the image recognition on the second image, image data for generating a second target-related pattern that constitutes part of a mixed reality view.
- the image data makes the display goggle 44 display the second target-related pattern that corresponds in position to the position identification unit 2 .
- a combination of the second target-related pattern, the pathological image and an actual view, which is the view the medical staff 6 sees through the display goggle 44 forms the mixed reality view perceived by the medical staff 6 .
- the display goggle 44 may display, based on the image data, the pathological image and a pattern that represents a predetermined surgical instrument (the second target-related pattern) at specific positions of the transparent lens thereof (the positions may be determined based on the reference target 21 and/or the instrument target 22 in the second image, and the visible light image which is similar to the actual view seen through the transparent lens), so that the medical staff 6 who wears the second support 41 may see the pathological image and the pattern that represents the predetermined surgical instrument superimposed on the actual view, thereby perceiving the mixed reality view.
- the second target-related pattern may be the pattern that represents the predetermined instrument, the pattern that represents the reference target, the pattern that represents the instrument target, or a combination thereof, and this disclosure is not limited in this respect.
- the display goggle 44 may display only the pathological image or only the second target-related pattern, and this disclosure is not limited in this respect.
- the processing unit 5 may further generate a mixed reality image by adding, based on the result of the image recognition on the second image, the pathological image and a third target-related pattern that corresponds in position to the position identification unit 2 into the visible light image. For example, the processing unit 5 may add the pathological image and a pattern that represents a predetermined surgical instrument (the third target-related pattern) into the visible light image at specific positions (the positions may be determined based on the reference target 21 and/or the instrument target 22 in the second image), so as to generate the mixed reality image. Then, the processing unit 5 outputs the mixed reality image to the display device 32 for display thereby in the MR area 323 thereof.
- the third target-related pattern may be the pattern that represents the predetermined instrument, the pattern that represents the reference target, the pattern that represents the instrument target, or a combination thereof, and this disclosure is not limited in this respect.
- the processing unit 5 may add only the pathological image or only the third target-related pattern into the visible light image to form the mixed reality image, and this disclosure is not limited in this respect.
- the processing unit 5 includes a first processor 51 and a second processor 52 .
- the first processor 51 is disposed on the placement frame 312 and is in signal connection with the display device 32 and the first infrared camera device 33 .
- the second processor 52 is in signal connection with the first processor 51 , the second infrared camera device 42 , the visible light camera device 43 and the display goggle 44 , and is adapted to be carried by the medical staff 6 .
- the second processor 52 is configured to receive the second image and the visible light image respectively from the second infrared camera device 42 and the visible light camera device 43 , to transmit the second image and the visible light image to the first processor 51 for image recognition, and to perform image processing on the image data for assisting in the creation of the mixed reality view perceived by the medical staff 6 .
- the processing unit 5 may only have the first processor 51 , in which case the second infrared camera device 42 , the visible light camera device and the display goggle 44 are in signal connection directly with the first processor 51 .
- an embodiment of an imaging method for surgical navigation according to this disclosure is applied to the surgical navigation system as described above.
- the medical staff 6 may use a preoperative planning system to position the reference target 21 in the pathological image, thereby establishing a world coordinate system S 0 (see FIG. 6 ). Since this is not a feature of this disclosure, and relevant details can be derived by a person skilled in the art according to the above description, further description is omitted herein for the sake of brevity.
- step 901 the first infrared camera device captures the first image of the position identification unit 2
- the second infrared camera device 42 captures the second image of the position identification unit 2
- step 902 the processing unit 5 performs image recognition on the first image and the second image with respect to the reference target 21 and the instrument target 22 .
- the processing unit 5 determines whether the first image has the reference target 21 , whether the first image has the instrument target 22 , whether the second image has the reference target 21 , and whether the second image has the instrument target 22 through image recognition techniques.
- the next step depends on a result of the image recognition, and is exemplified as shown in FIG. 8 .
- the processing unit 5 determines that one of the first and second images has both of the reference target 21 and the instrument target 22 (including a situation that both of the first and second images are determined as having both of the reference target 21 and the instrument target 22 )
- the flow goes to step 903 .
- the processing unit 5 determines that the second image has one of the reference target 21 and the instrument target 22 (including a situation that the second image is determined as having both of the reference target 21 and the instrument target 22 )
- the flow goes to step 905 . It is noted that, when the second image is determined as having both of the reference target 21 and the instrument target 22 , the flow goes to steps 903 and 905 at the same time.
- the flow returns to step 901 .
- this disclosure is not limited to such.
- some embodiments may only make determination on one of the abovementioned conditions (i.e., to only determine whether one of the first and second images has both of the reference target 21 and the instrument target 22 , or to only determine whether the second image has one of the reference target 21 and the instrument target 22 ) in step 902 , and the flow goes to the corresponding next step 903 or 905 when the condition is met.
- step 903 the processing unit 5 uses one of the first image and the second image to generate an augmented reality image by adding a first target-related pattern that corresponds to the position identification unit 2 into the pathological image.
- the processing unit 5 uses the first image to determine a position of the first target-related pattern in the pathological image, and generates the augmented reality image by adding the first target-related pattern into the pathological image at the position thus determined.
- the processing unit 5 uses the second image to determine a position of the first target-related pattern in the pathological image, and generates the augmented reality image by adding the first target-related pattern into the pathological image at the position thus determined.
- the processing unit 5 may select one of the first and second images which has a higher image resolution to be used in generating the augmented reality image, but this disclosure is not limited to such.
- the processing unit 5 calculates spatial coordinates of the reference target 21 and the instrument target 22 based on the reference target 21 and the instrument target 22 in the first or second image, and adds the first target-related pattern into the pathological image based on the spatial coordinates thus calculated.
- the reference target 21 defines the world coordinate system SO
- the first image captured by the first infrared camera device 33 is imaged on a first imaging coordinate system S 1
- the second image captured by the second infrared camera device 42 is imaged on a second imaging coordinate system S 2
- the processing unit 5 may perform a calibration step to calculate inner and outer matrices for the first infrared camera device 33 and inner and outer matrices for the second infrared camera device 42 , so that the processing unit 5 can perform coordinate transformation between the first imaging coordinate system S 1 and the world coordinate system S 0 and between the second imaging coordinate system S 2 and the world coordinate system S 0 through the above matrices.
- relative positions of the instrument target 22 and the reference target 21 in the first image or the second image can be used to obtain the position of the instrument target 22 in the world coordinate system S 0 , and the first target-related pattern can be added into the pathological image at a proper position to form the augmented reality image.
- the details of the coordinate transformation and the inner and outer matrices of the camera devices are omitted herein since one skilled in the art can deduce the details according to the above description based on known techniques.
- step 904 the processing unit 5 outputs the augmented reality image to the display device 5 for display thereby.
- step 905 the processing unit 5 calculates spatial coordinates (coordinates in the world coordinate system S 0 ) of the one of the reference target 21 and the instrument target 22 (i.e., the spatial coordinates of the reference target 21 , of the instrument target 22 , or of both of the reference target 21 and the instrument target 22 ) based on the second image, and generates, based on the spatial coordinates thus calculated, the mixed reality image and the image data which is used in the creation of the mixed reality view perceived by the medical staff 6 .
- a pattern that corresponds to said reference or instrument target 21 , 22 will be displayed by the display goggle 44 to help create the mixed reality view perceived by the medical staff 6 , and will be combined with the visible light image to form the mixed reality image to be displayed by the display device 32 .
- the processing unit 5 determines that the second image has both the reference target 21 and the instrument target 22
- the patterns that correspond to the reference target 21 and the instrument target 22 will both be displayed by the display goggle 44 to help create the mixed reality view perceived by the medical staff 6 , and will both be combined with the visible light image to form the mixed reality image to be displayed by the display device 32 .
- the visible light image captured by the visible light camera device 43 is imaged in a third imaging coordinate system S 3
- the display goggle 44 displays the second target-related pattern and/or the pathological image in a fourth imaging coordinate system S 4 . Since relative positions among the second infrared camera device 42 , the visible light camera device 43 , and the display goggle 44 are fixed, a calibration step could be performed before the steps of the embodiment to calculate inner and outer matrices of the visible camera device 43 and a coordinate transformation relationship among the second imaging coordinate system S 2 , the third imaging coordinate system S 3 , and the fourth imaging coordinate system S 4 , so that the processing unit 5 can calculate the position of the instrument target 22 in the world coordinate system S 0 and the position of the instrument target 22 as displayed by the display goggle 44 based on a relative position ⁇ X (e.g., a vector) that indicates relative positions of the instrument target 22 and the reference target 21 in the second image when the second image has both of the instrument target 22 and the reference target 22 , such that the pattern that
- the processing unit 5 can transform the coordinates of the reference target or the instrument target 22 in the second imaging coordinate system S 2 to coordinates in the fourth imaging coordinate system S 4 based on the coordinate transformation relationship as long as the second infrared camera device 42 captures either the reference target 21 or the instrument target 22 . That is, the processing unit 5 can form the mixed reality image without knowing the relative position ⁇ X that indicates relative positions of the instrument target 22 and the reference target 21 in the first image and/or second image, although the mixed reality view perceived by the medical staff 6 may not have both of the reference target 21 and the instrument target 22 in such a case.
- step 906 the processing unit 5 outputs the image data to the display goggle 44 to facilitate the creation of the mixed reality view as perceived by the medical staff 6 , and outputs the mixed reality image to the display device 32 for display thereby.
- the processing unit 5 may output only the image data to the display goggle 44 , without outputting the mixed reality image to the display device 32 .
- Table 1 shows a relationship between the result of the image recognition by the processing unit 5 and the resultant image types.
- the processing unit 5 may perform a determination as to which one of the first and second images is to be selected for subsequent use based on the resolution of the first and second images, and use the image with a higher resolution to generate the augmented reality image, the mixed reality view, and/or the mixed reality image.
- the processing unit 5 can calculate the position of the instrument target 22 in the world coordinate system S 0 based on the relative position ⁇ X that indicates relative positions of the instrument target 22 and the reference target 21 in the first image. It should be noted that although no instrument target is recognized in the second image, the processing unit 5 can know a direction that the second infrared camera device 42 faces based on the reference target 21 in the second image, so the pattern that represents the instrument target can still be shown for generating the mixed reality view and shown in the mixed reality image based on the relative position ⁇ X and the reference target 21 in the second image.
- the processing unit 5 can add the pattern that represents the instrument target 22 for creating the mixed reality view and into the mixed reality image based on the coordinate transformation relationship among the second imaging coordinate system S 2 , the third imaging coordinate system S 3 and the fourth imaging coordinate system recognized in the second image, the processing unit cannot know which direction the second infrared camera device 42 faces. Therefore, the processing unit 5 cannot add the pattern that represents the reference target 21 for creating the mixed reality view and into the mixed reality image even when a relative position ⁇ X that indicates relative positions of the reference target 21 and the instrument target 22 in the first image is known, and the mixed reality view and the mixed reality image can only have the pattern that represents the instrument target 22 .
- the processing unit 5 cannot calculate the position of the instrument target 22 in the world coordinate system S 0 because the relative position ⁇ X that indicates the relative positions of the instrument target 22 and the reference target 21 is not available. As a result, the pattern that represents the instrument target 22 cannot be combined with the pathological image to form the augmented reality image.
- the augmented reality image is displayed on the augmented reality area 322 of the display device 32
- the mixed reality image is displayed on the mixed reality area 323 of the display device 32
- the display goggle 44 is used to provide a person wearing the second support 41 with the perception of the mixed reality view.
- the embodiment of the surgical navigation system according to this disclosure is applicable to various types of surgery operations, such as spine surgery (see FIG. 2 ), knee surgery (see FIG. 9 ), etc.
- the processing unit 5 can still use the second image and the pathological image to form the augmented reality image, and output the augmented reality image to the display device 32 , so as to reduce the chance that a navigation image (the augmented reality image) cannot be shown because a line of vision between the first infrared camera device 33 and the subcutaneous diseased portion 81 is blocked, thereby achieving the effect of improving the stability or reliability of surgical navigation.
- the processing unit 5 can use the second image to help create the mixed reality view perceived by the medical staff 6 where the medical staff can see the pathological image and the third target-related pattern via the display goggle 44 , so the medical staff 6 can focus on the subcutaneous diseased portion 81 of the patient 8 without continuously having to look up at the display device 32 , thereby achieving the effect of improving the convenience of the operation.
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Surgery (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- General Health & Medical Sciences (AREA)
- Heart & Thoracic Surgery (AREA)
- Veterinary Medicine (AREA)
- Public Health (AREA)
- Biomedical Technology (AREA)
- Animal Behavior & Ethology (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Signal Processing (AREA)
- Robotics (AREA)
- Optics & Photonics (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Gynecology & Obstetrics (AREA)
- Radiology & Medical Imaging (AREA)
- Pathology (AREA)
- Vascular Medicine (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Electromagnetism (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Image Processing (AREA)
Abstract
Description
- This application claims priority of Taiwanese Invention Patent Application No. 109110389, filed on Mar. 27, 2020.
- The disclosure relates to a surgical navigation system, and more particularly to a surgical navigation system that has two tracking units, and an imaging method thereof.
- During a surgical operation such as orthopedic surgery or brain surgery, it is often not known whether the surgery is being performed at a precise position when the incision is small and/or the diseased portion (or surgical site) is deep.
- As shown in
FIG. 1 , a conventionalsurgical navigation system 1 is adapted to provide a pathological image (not shown) to assist amedical staff 10 in operating asurgical instrument 101 to perform a surgery on adiseased portion 103 of apatient 102, and includes areference target 11 disposed near thediseased portion 103 of thepatient 102, aninstrument target 12 disposed on thesurgical instrument 101, anoptical tracking unit 13, and ascreen 14. Thereference target 11 is set to a fixed position in the pathological image, and theoptical tracking unit 13 is configured to photograph thediseased portion 103, to calculate a position of theinstrument target 12 in an imaging coordinate system of theoptical tracking unit 13 according to relative positions of theinstrument target 12 and thereference target 11, and to output an augmented reality (AR) image that has thesurgical instrument 101 and the pathological image of thepatient 103 to thescreen 14. By this, themedical staff 10 can become aware of the position of thesurgical instrument 101 inside the patient's 102 body, and the distance and direction of thesurgical instrument 101 relative to thediseased portion 103, thereby achieving the effect of assisting the surgery. - However, when the
medical staff 10 or an assistant moves and blocks the camera of theoptical tracking system 13 so any one of thereference target 11 and theinstrument target 12 cannot be captured, theoptical tracking system 13 will be unable to calculate the position of thesurgical instrument 101, and the augmented reality image cannot be provided. - Therefore, an object of the disclosure is to provide a surgical navigation system that can improve the stability or reliability of surgical navigation.
- According to the disclosure, the surgical navigation system is adapted for use with a pathological image and a position identification unit to assist a medical staff in operating a surgical instrument to perform surgery on a subcutaneous diseased portion of a patient. The position identification unit includes a reference target detachably fixed on the patient, and an instrument target disposed on the surgical instrument. The surgical navigation system includes a first tracking unit, a second tracking unit and a processing unit. The first tracking unit includes a first support, a display device, and a first infrared camera device disposed on the first support. The first infrared camera device is configured to capture a first image of the position identification unit. The second tracking unit includes a second support, and a second infrared camera device disposed on the second support. The second infrared camera device is configured to capture a second image of the position identification unit. The processing unit is in signal connection with the display device, the first infrared camera device and the second infrared camera device, receives the first image and the second image respectively from the first infrared camera device and the second infrared camera device, and is configured to perform image recognition on the first image and the second image with respect to the reference target and the instrument target, to use, based on a result of the image recognition, one of the first image and the second image to generate an augmented reality image by adding a first target-related pattern that corresponds to the position identification unit into the pathological image, and to output the augmented reality image to the display device for display thereby.
- Another object of the disclosure is to provide an imaging method for surgical navigation.
- According to the disclosure, the imaging method includes: (A) preparing the surgical navigation system of this disclosure; (B) capturing the first image of the position identification unit by the first infrared camera device, and capturing the second image of the position identification unit by the second infrared camera device; (C) by the processing unit, determining whether the first image has the reference target, whether the first image has the instrument target, whether the second image has the reference target, and whether the second image has the instrument target; (D) by the processing unit, upon determining that one of the first and second images has the reference target and the instrument target in step (C), generating the augmented reality image by adding, based on the one of the first and second images, the first target-related pattern into the pathological image; and (E) by the processing unit, outputting the augmented reality image to the display device for display thereby.
- Other features and advantages of the disclosure will become apparent in the following detailed description of the embodiment(s) with reference to the accompanying drawings, of which:
-
FIG. 1 is a schematic perspective view illustrating use of a conventional surgical navigation system during a surgical operation; -
FIG. 2 is a schematic perspective view illustrating use of an embodiment of a surgical navigation system according to this disclosure during a surgical operation; -
FIG. 3 is a perspective view illustrating a reference target of the embodiment; -
FIG. 4 is a perspective view illustrating an instrument target of the embodiment; -
FIG. 5 is a perspective view illustrating a second tracking unit of the embodiment; -
FIG. 6 is a schematic diagram illustrating coordinate systems of different components of the embodiment; -
FIG. 7 is a block diagram illustrating the embodiment; -
FIG. 8 is a flow chart illustrating an embodiment of an imaging method for surgical navigation according to this disclosure; and -
FIG. 9 is a schematic diagram illustrating that the embodiments according to this disclosure are also applicable to a knee surgery. - Before the disclosure is described in greater detail, it should be noted that where considered appropriate, reference numerals or terminal portions of reference numerals have been repeated among the figures to indicate corresponding or analogous elements, which may optionally have similar characteristics.
- Referring to
FIG. 2 , an embodiment of a surgical navigation system according to this disclosure is adapted for use with a known pathological image to assist amedical staff 6 in operating asurgical instrument 7 to perform a surgery on a subcutaneousdiseased portion 81 of apatient 8. The pathological image may be, but is not limited to, an image conforming with a format of digital imaging and communications in medicine (DICOM), such as an image of computed tomography (CT), an image magnetic resonance imaging (MRI), etc. The term “diseased portion” as used throughout this disclosure refers to the tissue that is to be operated on during the surgery, or simply the surgical site, regardless of whether the surgery is intended to remove the tissue or a part of the tissue, to reconnect the tissue with other tissue, to insert prosthetics or a transplant, a combination of the above, or for other purposes. - The surgical navigation system includes a
position identification unit 2, afirst tracking unit 3, asecond tracking unit 4, and aprocessing unit 5. - The
position identification unit 2 includes areference target 21 to be detachably fixed on thepatient 8, and aninstrument target 22 disposed on thesurgical instrument 7. - Referring to
FIGS. 2, 3 and 6 , thereference target 21 has aclamp 211 extending along a first axis L1 and adapted to be detachably fixed to thepatient 8 through the act of clamping, areference adapter 212 connected to theclamp 211 and having an X-shape, and fourreference locator balls 213 connected to thereference adapter 212. Thereference locator balls 213 are coplanar to each other and lie on a plane P1 that is nonparallel with or transverse to the first axis L1. - It is noted that the arrangement of the
reference locator balls 213 of thereference target 21 and the shape of thereference adapter 212 are not limited to the above. In other modifications of the present embodiment, thereference locator balls 213 may be non-coplanar with each other, and/or the plane P1 may be parallel to the first axis L1; so long as the relative positions among thereference locator balls 213 are known. - Referring to
FIGS. 2, 4 and 6 , theinstrument target 22 has afastener 221 adapted to be mounted to thesurgical instrument 7 that extends along a second axis L2 so as to fasten the instrument target to thesurgical instrument 7, aninstrument adapter 222 having a Y-shape and connected to thefastener 221, and fourinstrument locator balls 223 connected to theinstrument adapter 222. Theinstrument locator balls 223 are coplanar to each other and lie on a plane P2 that is nonparallel with or transverse to the second axis L2. A geometrical arrangement of theinstrument locator balls 213 of thereference target 21 is different from a geometrical arrangement of theinstrument locator balls 223 of theinstrument target 22. It is noted that the arrangement of theinstrument locator balls 223 of theinstrument target 22 and the shape of theinstrument adapter 222 are not limited to the above. In other modifications of the present embodiment, theinstrument locator balls 223 may be noncoplanar with each other, and the plane P2 may be parallel to the second axis L2, so long as the relative positions among theinstrument locator balls 213 are known. Theinstrument target 22 is not limited to being one in number, and there may be two or more instrument targets 22 in other embodiments. In the present embodiment, outer surfaces of thereference locator balls 213 and theinstrument locator balls 223 are made of infrared reflective material (e.g., thereference locator balls 213 and theinstrument locator balls 223 have infrared reflective coatings), so as to be detected by infrared camera devices as described hereinafter. Referring toFIG. 2 , thefirst tracking unit 3 includes afirst support 31, adisplay device 32 disposed on thefirst support 31, and a firstinfrared camera device 33 disposed on thefirst support 31. The firstinfrared camera device 33 is disposed to face theposition identification unit 2 for capturing a first image (not shown) of theposition identification unit 2. - The
first support 31 includes abase 311 operable in one of a movable state and a locked state, aplacement frame 312 disposed on thebase 311, and abalance arm 313 connected to theplacement frame 312 and having the firstinfrared camera device 33 disposed thereon. Thebase 311 may include several locking caster wheels, so thebase 311 is movable in the movable state where the locking caster wheels are unlocked and is fixed in position in the locked state where the locking caster wheels are locked. - The
display device 32 has apathological image area 321 for displaying the pathological image, an augmented reality (AR)area 322 for displaying an augmented reality image, and a mixed reality (MR)area 323 for displaying a mixed reality image. - It is noted that in other variations of the present embodiment, the
display device 32 may be disposed to be separate from thefirst support 31. - In this embodiment, the
pathological image area 321, the augmentedreality area 322 and themixed reality area 323 are three areas in thedisplay device 32, and a user can arbitrarily drag, turn on or off, zoom in or out any one of these areas according to his/her own needs, so thedisplay device 32 can present different display combinations, such as displaying images in parallel or displaying one particular image in full screen. - In the present embodiment, the first
infrared camera device 33 includes twolenses 331 spaced apart from each other. The firstinfrared camera device 33 may be an infrared-emitting camera as an example. - Referring to
FIGS. 2 and 5 , thesecond tracking unit 4 includes asecond support 41, a secondinfrared camera device 42 disposed on thesecond support 41, a visiblelight camera device 43, and adisplay goggle 44 that includes a transparent lens so themedical staff 6 can view a scene therethrough, and is capable of displaying images on the transparent lens. - The second
infrared camera device 42 is disposed to face theposition identification unit 2 for capturing a second image (not shown) of theposition identification unit 2 from a different angle than the firstinfrared camera device 41, and includes twolenses 421 spaced apart from each other. The secondinfrared camera device 42 may be an infrared-emitting camera as an example. - The visible
light camera device 43 is disposed to face theposition identification unit 2 for capturing a visible light image of theposition identification unit 2, and includes twolenses 431 spaced apart from each other and disposed between thelenses 421. - In this embodiment, the
second support 41 is a head-mounted device, and thesecond tracking unit 4 is configured such that, when themedical staff 6 wears thesecond support 41, the secondinfrared camera device 42 and the visiblelight camera device 43 are disposed above the eyes of themedical staff 6, and thedisplay goggle 44 is located below the visiblelight camera device 43 and is in front of the eyes of themedical staff 6. - Referring to
FIGS. 2 and 7 , theprocessing unit 5 is in signal connection with thedisplay device 32, the firstinfrared camera device 33, the secondinfrared camera device 42, the visiblelight camera device 43 and thedisplay goggle 44, and is configured to receive the first image, the second image and the visible light image respectively from the firstinfrared camera device 33, the secondinfrared camera device 42 and the visiblelight camera device 43, and to perform image recognition on the first image and the second image with respect to thereference target 21 and the instrument target 22 (i.e., to identify thereference target 21 and theinstrument target 22 from the first image and the second image). - It should be noted that the signal connection as referred in this embodiment can be realized by a wired connection (e.g., signal lines corresponding to different signal sources) or a wireless connection (e.g., wireless communication technologies such as Bluetooth and wireless network).
- The
processing unit 5 is configured to use, based on a result of the image recognition on the first image and the second image, one of the first image and the second image to generate an augmented reality image by adding a first target-related pattern that corresponds in position to theposition identification unit 2 into the pathological image. For example, theprocessing unit 5 may add a pattern that represents a predetermined surgical instrument (the first target-related pattern) to the pathological image at a position that corresponds to the instrument target 22 (the position may be determined based on thereference target 21 and theinstrument target 22 in said one of the first image and the second image), so as to generate the augmented reality image. Then, theprocessing unit 5 outputs the augmented reality image to thedisplay device 32 for display thereby in theAR area 322 thereof. In some embodiments, the first target-related pattern may be the pattern that represents the predetermined instrument, a pattern that represents the reference target, a pattern that represents the instrument target, or a combination thereof, and this disclosure is not limited in this respect. - The
processing unit 5 may further output, to thedisplay goggle 44 based on a result of the image recognition on the second image, image data for generating a second target-related pattern that constitutes part of a mixed reality view. The image data makes thedisplay goggle 44 display the second target-related pattern that corresponds in position to theposition identification unit 2. A combination of the second target-related pattern, the pathological image and an actual view, which is the view themedical staff 6 sees through thedisplay goggle 44, forms the mixed reality view perceived by themedical staff 6. For example, thedisplay goggle 44 may display, based on the image data, the pathological image and a pattern that represents a predetermined surgical instrument (the second target-related pattern) at specific positions of the transparent lens thereof (the positions may be determined based on thereference target 21 and/or theinstrument target 22 in the second image, and the visible light image which is similar to the actual view seen through the transparent lens), so that themedical staff 6 who wears thesecond support 41 may see the pathological image and the pattern that represents the predetermined surgical instrument superimposed on the actual view, thereby perceiving the mixed reality view. In some embodiments, the second target-related pattern may be the pattern that represents the predetermined instrument, the pattern that represents the reference target, the pattern that represents the instrument target, or a combination thereof, and this disclosure is not limited in this respect. In other embodiments, thedisplay goggle 44 may display only the pathological image or only the second target-related pattern, and this disclosure is not limited in this respect. - The
processing unit 5 may further generate a mixed reality image by adding, based on the result of the image recognition on the second image, the pathological image and a third target-related pattern that corresponds in position to theposition identification unit 2 into the visible light image. For example, theprocessing unit 5 may add the pathological image and a pattern that represents a predetermined surgical instrument (the third target-related pattern) into the visible light image at specific positions (the positions may be determined based on thereference target 21 and/or theinstrument target 22 in the second image), so as to generate the mixed reality image. Then, theprocessing unit 5 outputs the mixed reality image to thedisplay device 32 for display thereby in theMR area 323 thereof. In some embodiments, the third target-related pattern may be the pattern that represents the predetermined instrument, the pattern that represents the reference target, the pattern that represents the instrument target, or a combination thereof, and this disclosure is not limited in this respect. In other embodiments, theprocessing unit 5 may add only the pathological image or only the third target-related pattern into the visible light image to form the mixed reality image, and this disclosure is not limited in this respect. - In this embodiment, the
processing unit 5 includes afirst processor 51 and asecond processor 52. Thefirst processor 51 is disposed on theplacement frame 312 and is in signal connection with thedisplay device 32 and the firstinfrared camera device 33. Thesecond processor 52 is in signal connection with thefirst processor 51, the secondinfrared camera device 42, the visiblelight camera device 43 and thedisplay goggle 44, and is adapted to be carried by themedical staff 6. - In this embodiment, the
second processor 52 is configured to receive the second image and the visible light image respectively from the secondinfrared camera device 42 and the visiblelight camera device 43, to transmit the second image and the visible light image to thefirst processor 51 for image recognition, and to perform image processing on the image data for assisting in the creation of the mixed reality view perceived by themedical staff 6. In other embodiments, theprocessing unit 5 may only have thefirst processor 51, in which case the secondinfrared camera device 42, the visible light camera device and thedisplay goggle 44 are in signal connection directly with thefirst processor 51. - Referring to
FIGS. 2 and 8 , an embodiment of an imaging method for surgical navigation according to this disclosure is applied to the surgical navigation system as described above. - Before the embodiment of the imaging method is performed, the
medical staff 6 may use a preoperative planning system to position thereference target 21 in the pathological image, thereby establishing a world coordinate system S0 (seeFIG. 6 ). Since this is not a feature of this disclosure, and relevant details can be derived by a person skilled in the art according to the above description, further description is omitted herein for the sake of brevity. - In
step 901, the first infrared camera device captures the first image of theposition identification unit 2, and the secondinfrared camera device 42 captures the second image of theposition identification unit 2 - In
step 902, theprocessing unit 5 performs image recognition on the first image and the second image with respect to thereference target 21 and theinstrument target 22. In detail, theprocessing unit 5 determines whether the first image has thereference target 21, whether the first image has theinstrument target 22, whether the second image has thereference target 21, and whether the second image has theinstrument target 22 through image recognition techniques. The next step depends on a result of the image recognition, and is exemplified as shown inFIG. 8 . When theprocessing unit 5 determines that one of the first and second images has both of thereference target 21 and the instrument target 22 (including a situation that both of the first and second images are determined as having both of thereference target 21 and the instrument target 22), the flow goes to step 903. When theprocessing unit 5 determines that the second image has one of thereference target 21 and the instrument target 22 (including a situation that the second image is determined as having both of thereference target 21 and the instrument target 22), the flow goes to step 905. It is noted that, when the second image is determined as having both of thereference target 21 and theinstrument target 22, the flow goes tosteps reference target 21 and theinstrument target 22, or to only determine whether the second image has one of thereference target 21 and the instrument target 22) instep 902, and the flow goes to the correspondingnext step - In
step 903, theprocessing unit 5 uses one of the first image and the second image to generate an augmented reality image by adding a first target-related pattern that corresponds to theposition identification unit 2 into the pathological image. In detail, upon determining instep 902 that the first image has both of thereference target 21 and theinstrument target 22, and that the second image does not have both of thereference target 21 and theinstrument target 22, theprocessing unit 5 uses the first image to determine a position of the first target-related pattern in the pathological image, and generates the augmented reality image by adding the first target-related pattern into the pathological image at the position thus determined. Upon determining instep 902 that the second image has both of thereference target 21 and theinstrument target 22, and that the first image does not have both of thereference target 21 and theinstrument target 22, theprocessing unit 5 uses the second image to determine a position of the first target-related pattern in the pathological image, and generates the augmented reality image by adding the first target-related pattern into the pathological image at the position thus determined. Upon determining instep 902 that both of the first and second images have both of thereference target 21 and theinstrument target 22, theprocessing unit 5 may select one of the first and second images which has a higher image resolution to be used in generating the augmented reality image, but this disclosure is not limited to such. In order to determine the position of the first target-related pattern in the pathological image, theprocessing unit 5 calculates spatial coordinates of thereference target 21 and theinstrument target 22 based on thereference target 21 and theinstrument target 22 in the first or second image, and adds the first target-related pattern into the pathological image based on the spatial coordinates thus calculated. - Referring to
FIGS. 2, 6 and 8 , thereference target 21 defines the world coordinate system SO, the first image captured by the firstinfrared camera device 33 is imaged on a first imaging coordinate system S1, and the second image captured by the secondinfrared camera device 42 is imaged on a second imaging coordinate system S2. Before the above steps are performed, theprocessing unit 5 may perform a calibration step to calculate inner and outer matrices for the firstinfrared camera device 33 and inner and outer matrices for the secondinfrared camera device 42, so that theprocessing unit 5 can perform coordinate transformation between the first imaging coordinate system S1 and the world coordinate system S0 and between the second imaging coordinate system S2 and the world coordinate system S0 through the above matrices. Accordingly, relative positions of theinstrument target 22 and thereference target 21 in the first image or the second image can be used to obtain the position of theinstrument target 22 in the world coordinate system S0, and the first target-related pattern can be added into the pathological image at a proper position to form the augmented reality image. The details of the coordinate transformation and the inner and outer matrices of the camera devices are omitted herein since one skilled in the art can deduce the details according to the above description based on known techniques. - In
step 904, theprocessing unit 5 outputs the augmented reality image to thedisplay device 5 for display thereby. - In
step 905, theprocessing unit 5 calculates spatial coordinates (coordinates in the world coordinate system S0) of the one of thereference target 21 and the instrument target 22 (i.e., the spatial coordinates of thereference target 21, of theinstrument target 22, or of both of thereference target 21 and the instrument target 22) based on the second image, and generates, based on the spatial coordinates thus calculated, the mixed reality image and the image data which is used in the creation of the mixed reality view perceived by themedical staff 6. - Briefly, when the
processing unit 5 determines that the second image has only one of thereference target 21 and theinstrument target 22, a pattern that corresponds to said reference orinstrument target display goggle 44 to help create the mixed reality view perceived by themedical staff 6, and will be combined with the visible light image to form the mixed reality image to be displayed by thedisplay device 32. When theprocessing unit 5 determines that the second image has both thereference target 21 and theinstrument target 22, the patterns that correspond to thereference target 21 and theinstrument target 22 will both be displayed by thedisplay goggle 44 to help create the mixed reality view perceived by themedical staff 6, and will both be combined with the visible light image to form the mixed reality image to be displayed by thedisplay device 32. - The visible light image captured by the visible
light camera device 43 is imaged in a third imaging coordinate system S3, and thedisplay goggle 44 displays the second target-related pattern and/or the pathological image in a fourth imaging coordinate system S4. Since relative positions among the secondinfrared camera device 42, the visiblelight camera device 43, and thedisplay goggle 44 are fixed, a calibration step could be performed before the steps of the embodiment to calculate inner and outer matrices of thevisible camera device 43 and a coordinate transformation relationship among the second imaging coordinate system S2, the third imaging coordinate system S3, and the fourth imaging coordinate system S4, so that theprocessing unit 5 can calculate the position of theinstrument target 22 in the world coordinate system S0 and the position of theinstrument target 22 as displayed by thedisplay goggle 44 based on a relative position ΔX (e.g., a vector) that indicates relative positions of theinstrument target 22 and thereference target 21 in the second image when the second image has both of theinstrument target 22 and thereference target 22, such that the pattern that represents theinstrument target 22 and/or the second target-related pattern can be located at a proper position(s) when displayed by thedisplay goggle 44 to create the mixed reality view perceived by themedical staff 6, and at a correct position in the visible light image to form the mixed reality image to be displayed by thedisplay device 32. In addition, since theprocessing unit 5 already has the coordinate transformation relationship between the second imaging coordinate system S2 and the fourth coordinate system S4, theprocessing unit 5 can transform the coordinates of the reference target or theinstrument target 22 in the second imaging coordinate system S2 to coordinates in the fourth imaging coordinate system S4 based on the coordinate transformation relationship as long as the secondinfrared camera device 42 captures either thereference target 21 or theinstrument target 22. That is, theprocessing unit 5 can form the mixed reality image without knowing the relative position ΔX that indicates relative positions of theinstrument target 22 and thereference target 21 in the first image and/or second image, although the mixed reality view perceived by themedical staff 6 may not have both of thereference target 21 and theinstrument target 22 in such a case. - In
step 906, theprocessing unit 5 outputs the image data to thedisplay goggle 44 to facilitate the creation of the mixed reality view as perceived by themedical staff 6, and outputs the mixed reality image to thedisplay device 32 for display thereby. - In some embodiments, the
processing unit 5 may output only the image data to thedisplay goggle 44, without outputting the mixed reality image to thedisplay device 32. - For ease of understanding, Table 1 shows a relationship between the result of the image recognition by the
processing unit 5 and the resultant image types. -
-
TABLE 1 Target(s) recognized in second image R and I R only I only No target Target(s) R and I AR + MR AR + MR AR + MR AR only recognized (MR: pattern in first for I only) image R only MR only MR only N/A I only (pattern for (pattern for No target R only) I only) R: reference target; I: instrument target; AR: augmented reality image; MR: mixed reality view/image - R: reference target; I: instrument target;
- AR: augmented reality image; MR: mixed reality view/image
- In the case that both of the first image and the second image are determined as having both of the
reference target 21 and theinstrument target 22, theprocessing unit 5 may perform a determination as to which one of the first and second images is to be selected for subsequent use based on the resolution of the first and second images, and use the image with a higher resolution to generate the augmented reality image, the mixed reality view, and/or the mixed reality image. - In the case that the first image is determined as having both of the
reference target 21 and theinstrument target 22, and the second image is determined as having only thereference target 21, theprocessing unit 5 can calculate the position of theinstrument target 22 in the world coordinate system S0 based on the relative position ΔX that indicates relative positions of theinstrument target 22 and thereference target 21 in the first image. It should be noted that although no instrument target is recognized in the second image, theprocessing unit 5 can know a direction that the secondinfrared camera device 42 faces based on thereference target 21 in the second image, so the pattern that represents the instrument target can still be shown for generating the mixed reality view and shown in the mixed reality image based on the relative position ΔX and thereference target 21 in the second image. - In the case the second image is determined as having only the
instrument target 22 and not having thereference target 21, theprocessing unit 5 can add the pattern that represents theinstrument target 22 for creating the mixed reality view and into the mixed reality image based on the coordinate transformation relationship among the second imaging coordinate system S2, the third imaging coordinate system S3 and the fourth imaging coordinate system recognized in the second image, the processing unit cannot know which direction the secondinfrared camera device 42 faces. Therefore, theprocessing unit 5 cannot add the pattern that represents thereference target 21 for creating the mixed reality view and into the mixed reality image even when a relative position ΔX that indicates relative positions of thereference target 21 and theinstrument target 22 in the first image is known, and the mixed reality view and the mixed reality image can only have the pattern that represents theinstrument target 22. - In the case that both of the first and second images are determined as not having both of the
reference target 21 and theinstrument target 22, theprocessing unit 5 cannot calculate the position of theinstrument target 22 in the world coordinate system S0 because the relative position ΔX that indicates the relative positions of theinstrument target 22 and thereference target 21 is not available. As a result, the pattern that represents theinstrument target 22 cannot be combined with the pathological image to form the augmented reality image. - In this embodiment, the augmented reality image is displayed on the
augmented reality area 322 of thedisplay device 32, the mixed reality image is displayed on themixed reality area 323 of thedisplay device 32, and thedisplay goggle 44 is used to provide a person wearing thesecond support 41 with the perception of the mixed reality view. - It is noted that the embodiment of the surgical navigation system according to this disclosure is applicable to various types of surgery operations, such as spine surgery (see
FIG. 2 ), knee surgery (seeFIG. 9 ), etc. - The advantages of the foregoing embodiment can be summarized as follows:
- 1. By providing the second
infrared camera device 42 of thesecond tracking unit 4 to capture the second image of the subcutaneousdiseased portion 81, when the firstinfrared camera device 33 of thefirst tracking system 3 cannot capture one or both of thereference target 21 and theinstrument target 22, that is, the first image does not have thereference target 21 and/or theinstrument target 22, theprocessing unit 5 can still use the second image and the pathological image to form the augmented reality image, and output the augmented reality image to thedisplay device 32, so as to reduce the chance that a navigation image (the augmented reality image) cannot be shown because a line of vision between the firstinfrared camera device 33 and the subcutaneousdiseased portion 81 is blocked, thereby achieving the effect of improving the stability or reliability of surgical navigation. - 2. Because of safety reasons, there are few situations that the sight of the
medical staff 6 would be blocked by external objects, so the secondinfrared camera device 42 and the visiblelight camera device 43 that are disposed above the eyes of themedical staff 6 would hardly ever be blocked during a surgical operation, thereby achieving the effect of improving the stability or reliability of the surgical navigation. 3. Theprocessing unit 5 can use the second image to help create the mixed reality view perceived by themedical staff 6 where the medical staff can see the pathological image and the third target-related pattern via thedisplay goggle 44, so themedical staff 6 can focus on the subcutaneousdiseased portion 81 of thepatient 8 without continuously having to look up at thedisplay device 32, thereby achieving the effect of improving the convenience of the operation. - In the description above, for the purposes of explanation, numerous specific details have been set forth in order to provide a thorough understanding of the embodiment(s). It will be apparent, however, to one skilled in the art, that one or more other embodiments may be practiced without some of these specific details. It should also be appreciated that reference throughout this specification to “one embodiment,” “an embodiment,” an embodiment with an indication of an ordinal number and so forth means that a particular feature, structure, or characteristic may be included in the practice of the disclosure. It should be further appreciated that in the description, various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of various inventive aspects, and that one or more features or specific details from one embodiment may be practiced together with one or more features or specific details from another embodiment, where appropriate, in the practice of the disclosure.
- While the disclosure has been described in connection with what is (are) considered the exemplary embodiment(s), it is understood that this disclosure is not limited to the disclosed embodiment(s) but is intended to cover various arrangements included within the spirit and scope of the broadest interpretation so as to encompass all such modifications and equivalent arrangements.
Claims (18)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/145,482 US20230126207A1 (en) | 2020-03-27 | 2022-12-22 | Surgical navigation system, and imaging method of the same |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
TW109110389 | 2020-03-27 | ||
TW109110389A TWI727725B (en) | 2020-03-27 | 2020-03-27 | Surgical navigation system and its imaging method |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/145,482 Continuation-In-Part US20230126207A1 (en) | 2020-03-27 | 2022-12-22 | Surgical navigation system, and imaging method of the same |
Publications (1)
Publication Number | Publication Date |
---|---|
US20210298835A1 true US20210298835A1 (en) | 2021-09-30 |
Family
ID=72913931
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/212,007 Abandoned US20210298835A1 (en) | 2020-03-27 | 2021-03-25 | Surgical navigation system, and imaging method of the same |
US18/145,482 Pending US20230126207A1 (en) | 2020-03-27 | 2022-12-22 | Surgical navigation system, and imaging method of the same |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/145,482 Pending US20230126207A1 (en) | 2020-03-27 | 2022-12-22 | Surgical navigation system, and imaging method of the same |
Country Status (3)
Country | Link |
---|---|
US (2) | US20210298835A1 (en) |
CN (1) | CN111821024B (en) |
TW (1) | TWI727725B (en) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11382712B2 (en) | 2019-12-22 | 2022-07-12 | Augmedics Ltd. | Mirroring in image guided surgery |
US11389252B2 (en) * | 2020-06-15 | 2022-07-19 | Augmedics Ltd. | Rotating marker for image guided surgery |
EP4223171A1 (en) * | 2022-02-02 | 2023-08-09 | Zimmer, Inc. | Mixed reality surgical helmet |
US11750794B2 (en) | 2015-03-24 | 2023-09-05 | Augmedics Ltd. | Combining video-based and optic-based augmented reality in a near eye display |
US11766296B2 (en) | 2018-11-26 | 2023-09-26 | Augmedics Ltd. | Tracking system for image-guided surgery |
US11974887B2 (en) | 2018-05-02 | 2024-05-07 | Augmedics Ltd. | Registration marker for an augmented reality system |
US11980506B2 (en) | 2019-07-29 | 2024-05-14 | Augmedics Ltd. | Fiducial marker |
Family Cites Families (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7753910B2 (en) * | 2003-07-21 | 2010-07-13 | Stephen Ritland | Surgical image tracker mounting base apparatus and method of use |
US7702379B2 (en) * | 2004-08-25 | 2010-04-20 | General Electric Company | System and method for hybrid tracking in surgical navigation |
US9718190B2 (en) * | 2006-06-29 | 2017-08-01 | Intuitive Surgical Operations, Inc. | Tool position and identification indicator displayed in a boundary area of a computer display screen |
JP5355220B2 (en) * | 2009-05-22 | 2013-11-27 | キヤノン株式会社 | Fundus photographing device |
CN102772244A (en) * | 2012-08-13 | 2012-11-14 | 南京航空航天大学 | Intraoperative navigation system used for implanting pedicle screw |
CN102910130B (en) * | 2012-10-24 | 2015-08-05 | 浙江工业大学 | Forewarn system is assisted in a kind of driving of real enhancement mode |
CN103908345B (en) * | 2012-12-31 | 2017-02-08 | 复旦大学 | Volume data visualization method for surgical navigation based on PPC (Panel Personal Computer) |
WO2015057507A1 (en) * | 2013-10-15 | 2015-04-23 | Ellsworth Jeri J | System and method for reconfigurable projected augmented/virtual reality appliance |
NO2944284T3 (en) * | 2014-05-13 | 2018-05-05 | ||
US20150366628A1 (en) * | 2014-06-18 | 2015-12-24 | Covidien Lp | Augmented surgical reality environment system |
CN109416841B (en) * | 2016-07-11 | 2023-03-31 | 台湾骨王生技股份有限公司 | Method for enhancing image fidelity and application thereof method for surgical guidance on wearable glasses |
US10537394B2 (en) * | 2016-12-19 | 2020-01-21 | Ethicon Llc | Hot device indication of video display |
CN107088091A (en) * | 2017-06-08 | 2017-08-25 | 广州技特电子科技有限公司 | The operation guiding system and air navigation aid of a kind of auxiliary bone surgery |
CN208017582U (en) * | 2017-06-16 | 2018-10-30 | 青岛大学附属医院 | Area of computer aided Minimally Invasive Surgery device |
US11272985B2 (en) * | 2017-11-14 | 2022-03-15 | Stryker Corporation | Patient-specific preoperative planning simulation techniques |
WO2019141704A1 (en) * | 2018-01-22 | 2019-07-25 | Medivation Ag | An augmented reality surgical guidance system |
CN108742841B (en) * | 2018-05-30 | 2020-11-06 | 上海交通大学 | Tool real-time positioning device of multi-position tracker |
TWI741196B (en) * | 2018-06-26 | 2021-10-01 | 華宇藥品股份有限公司 | Surgical navigation method and system integrating augmented reality |
US11612438B2 (en) * | 2018-09-05 | 2023-03-28 | Point Robotics Medtech Inc. | Navigation system and method for medical operation by a robotic system using a tool |
-
2020
- 2020-03-27 TW TW109110389A patent/TWI727725B/en active
- 2020-05-19 CN CN202010422466.6A patent/CN111821024B/en active Active
-
2021
- 2021-03-25 US US17/212,007 patent/US20210298835A1/en not_active Abandoned
-
2022
- 2022-12-22 US US18/145,482 patent/US20230126207A1/en active Pending
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11750794B2 (en) | 2015-03-24 | 2023-09-05 | Augmedics Ltd. | Combining video-based and optic-based augmented reality in a near eye display |
US11974887B2 (en) | 2018-05-02 | 2024-05-07 | Augmedics Ltd. | Registration marker for an augmented reality system |
US11980507B2 (en) | 2018-05-02 | 2024-05-14 | Augmedics Ltd. | Registration of a fiducial marker for an augmented reality system |
US11980508B2 (en) | 2018-05-02 | 2024-05-14 | Augmedics Ltd. | Registration of a fiducial marker for an augmented reality system |
US11766296B2 (en) | 2018-11-26 | 2023-09-26 | Augmedics Ltd. | Tracking system for image-guided surgery |
US11980429B2 (en) | 2018-11-26 | 2024-05-14 | Augmedics Ltd. | Tracking methods for image-guided surgery |
US11980506B2 (en) | 2019-07-29 | 2024-05-14 | Augmedics Ltd. | Fiducial marker |
US11382712B2 (en) | 2019-12-22 | 2022-07-12 | Augmedics Ltd. | Mirroring in image guided surgery |
US11801115B2 (en) | 2019-12-22 | 2023-10-31 | Augmedics Ltd. | Mirroring in image guided surgery |
US11389252B2 (en) * | 2020-06-15 | 2022-07-19 | Augmedics Ltd. | Rotating marker for image guided surgery |
EP4223171A1 (en) * | 2022-02-02 | 2023-08-09 | Zimmer, Inc. | Mixed reality surgical helmet |
Also Published As
Publication number | Publication date |
---|---|
CN111821024A (en) | 2020-10-27 |
CN111821024B (en) | 2022-04-05 |
TW202135737A (en) | 2021-10-01 |
US20230126207A1 (en) | 2023-04-27 |
TWI727725B (en) | 2021-05-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20210298835A1 (en) | Surgical navigation system, and imaging method of the same | |
US11276187B2 (en) | Method and system for registration verification | |
US11510750B2 (en) | Leveraging two-dimensional digital imaging and communication in medicine imagery in three-dimensional extended reality applications | |
US20210052348A1 (en) | An Augmented Reality Surgical Guidance System | |
RU2621488C2 (en) | Display fixed on head and method of controlling display fixed on head | |
US6891518B2 (en) | Augmented reality visualization device | |
US7774044B2 (en) | System and method for augmented reality navigation in a medical intervention procedure | |
WO2018165767A1 (en) | Systems and methods for augmented reality display in navigated surgeries | |
US11574446B2 (en) | Digital image reality aligning kit and method applied to mixed reality system for surgical navigation | |
US20220313386A1 (en) | Navigated surgical system with eye to xr headset display calibration | |
US20210386482A1 (en) | Rotating marker | |
KR20130132335A (en) | Camera registration method for augmented reality of surgical navigation system | |
TWI825891B (en) | Augmented reality system for real space navigation and surgical system using the same | |
JP2008018015A (en) | Medical display unit and system | |
King et al. | Stereo augmented reality in the surgical microscope | |
KR102582154B1 (en) | User terminal for providing augmented reality medical image using depth camera and method for providing augmented reality medical image | |
CN111297501B (en) | Augmented reality navigation method and system for oral implantation operation | |
CN110638525B (en) | Operation navigation system integrating augmented reality | |
TWI697317B (en) | Digital image reality alignment kit and method applied to mixed reality system for surgical navigation | |
JP2024505269A (en) | Surgical auxiliary system including surgical microscope and camera and display method | |
KR101923927B1 (en) | Image registration system and method using subject-specific tracker | |
US20230233257A1 (en) | Augmented reality headset systems and methods for surgical planning and guidance | |
TW202223915A (en) | Method and system for register operating space | |
CA3232379A1 (en) | Integrated surgical navigation and visualization system, and methods thereof | |
WO2023203522A2 (en) | Reduction of jitter in virtual presentation |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: TAIWAN MAIN ORTHOPAEDIC BIOTECHNOLOGY CO., LTD., TAIWAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:WANG, MIN-LIANG;REEL/FRAME:055712/0310 Effective date: 20210311 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |