US20220392607A1 - Image acquisition visuals for augmented reality - Google Patents
Image acquisition visuals for augmented reality Download PDFInfo
- Publication number
- US20220392607A1 US20220392607A1 US17/776,042 US202017776042A US2022392607A1 US 20220392607 A1 US20220392607 A1 US 20220392607A1 US 202017776042 A US202017776042 A US 202017776042A US 2022392607 A1 US2022392607 A1 US 2022392607A1
- Authority
- US
- United States
- Prior art keywords
- image
- imaging
- interventional
- sequence
- visual
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 145
- 230000000007 visual effect Effects 0.000 title claims abstract description 125
- 238000003384 imaging method Methods 0.000 claims abstract description 293
- 230000002452 interceptive effect Effects 0.000 claims abstract description 69
- 230000003416 augmentation Effects 0.000 claims abstract description 18
- 230000003993 interaction Effects 0.000 claims abstract description 15
- 238000000034 method Methods 0.000 claims description 15
- 238000009877 rendering Methods 0.000 claims description 3
- 238000002604 ultrasonography Methods 0.000 description 32
- 210000003484 anatomy Anatomy 0.000 description 24
- 239000000523 sample Substances 0.000 description 16
- 238000004891 communication Methods 0.000 description 15
- 230000015654 memory Effects 0.000 description 12
- 238000012163 sequencing technique Methods 0.000 description 12
- 238000003745 diagnosis Methods 0.000 description 11
- 238000012285 ultrasound imaging Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 238000010348 incorporation Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000009420 retrofitting Methods 0.000 description 3
- 230000002123 temporal effect Effects 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 239000003550 marker Substances 0.000 description 2
- 210000000056 organ Anatomy 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 241001481828 Glyptocephalus cynoglossus Species 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 230000009849 deactivation Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 238000002675 image-guided surgery Methods 0.000 description 1
- 230000003340 mental effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000004984 smart glass Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 210000001519 tissue Anatomy 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G16H20/40—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/20—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
- G16H40/63—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for local operation
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
- G16H40/67—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
- A61B2034/2055—Optical tracking systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B2090/364—Correlation of different images or relation of image positions in respect to the body
- A61B2090/365—Correlation of different images or relation of image positions in respect to the body augmented reality, i.e. correlating a live optical image with another image
Definitions
- the present disclosure generally relates to image-guided interventions for a diagnosis and/or a treatment of a three-dimensional (3D) anatomy.
- the present disclosure specifically relates to an incorporation of augmented reality into image-guided interventions.
- Two-dimensional (2D) interventional images are inherently projections of complex three-dimensional (3D) anatomy.
- 3D three-dimensional
- a clinician is provided an opportunity to control a positioning and/or an orientation of the 2D imaging modality to thereby acquire 2D interventional images from a variety of angles (e.g., a clinician manual or programmed control of a C-arm of an X-ray system, a clinician manual or programmed control of a robotic ultrasound probe, etc.).
- the clinician may decide to treat the diseased 3D anatomy by using interventional tools at the disposal of the clinician.
- the imaging modality is designed to be able to find the optimal viewing angle for visualizing the diseased area as well as positioning and/or an orientating of interventional tools within the diseased area.
- the clinician frequently needs to access the 2D interventional images taken from previous positions (or even a previous procedure) to validate the diagnosis of the 3D anatomy, to check for anomalies within the 3D anatomy and/or to validate the accuracy of the treatment of the 3D anatomy.
- an imaging modality stores geometry information with the 2D interventional images (e.g., an angulation/rotation of a C-arm, a mechanical scanning of an ultrasound probe, etc.), such geometry information is only available as a linear list.
- the clinician needs to make a mental picture of the image-guided intervention to know where an image sequence was taken at which time(s) and at which location(s). Especially for novice users and very long procedures, it is sometimes difficult to get an overview of the whole procedure.
- a clinician typically must navigate through an image sequence linearly, one by one, until they find the 2D interventional image(s) for validating the diagnosis of the 3D anatomy, checking for anomalies within the 3D anatomy and/or validating the accuracy of the treatment of the 3D anatomy.
- an image-guided intervention e.g., X-ray interventional imaging for diagnosis and/or treatment of 3D anatomy, ultrasound interventional imaging for diagnosis and/or treatment of 3D anatomy, etc.
- the present disclosure may be embodied as (1) a visual image sequence controller, (2) an augmented reality device incorporating a visual image sequence controller of the present disclosure, (3) an imaging modality incorporating a visual image sequence controller of the present disclosure, (4) an auxiliary intervention device incorporating a visual image sequence controller of the present disclosure, (5) an augmented image-guided intervention system incorporating an augmented reality device of the present disclosure, an imaging modality of the present disclosure and/or an auxiliary intervention device of the present disclosure, (6) an augmented image-guided intervention method utilizing a visual image sequence controller of the present disclosure; and (7) a visual image sequencing method of the present disclosure.
- a visual image sequence controller includes a non-transitory machine-readable storage medium encoded with instructions for execution by one or more processors to control an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence (live or recorded) in a rendered augmented view of an image-guided intervention by an augmented reality device.
- the non-transitory machine-readable storage medium includes instructions to (1) access geometric information of an imaging modality associated with an acquisition of an interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, (2) generate a visual imaging sequence acquisition of the interventional imaging sequence including, for each interventional image of the interventional imaging sequence, an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence and of an imaging parameter of the imaging modality during an acquisition of the corresponding interventional image by the imaging modality, (3) augment the view of the image-guided intervention by the augmented reality device with the interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, and (4) interface with the at least one of the imaging modality and a display responsive to a user interaction with at least one of the interactive virtual indicators.
- imaging parameter can be any information indicative of the geometry (such as direction, rotation or angulation) or status of one or more components of the imaging modality (e.g.
- augmented reality device embodiments of the present disclosure encompass an augmented reality device employing an augmented reality display for viewing an image-guided intervention (live or recorded) and further employing a visual image sequence controller for controlling an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in the rendered augmented view of the image-guided intervention by the augmented reality device.
- the visual image sequence controller is configured to (1) access geometric information of an imaging modality associated with an acquisition of an interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, (2) generate a visual imaging sequence acquisition of the interventional imaging sequence including, for each interventional image of the interventional imaging sequence, an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence and of an imaging parameter of the imaging modality during an acquisition of the corresponding interventional image by the imaging modality, (3) augment the view of the image-guided intervention by the augmented reality device with the interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, and (4) interface with the at least one of the imaging modality and a display responsive to a user interaction with at least one of the interactive virtual indicators.
- imaging parameter can be any information indicative of the geometry (such as direction, rotation or angulation) or status of one or more components of the imaging modality (e.g. a C-arm, an
- imaging modality embodiments of the present disclosure encompass an imaging modality employing an imaging device for generating an interventional imaging sequence of an image-guided intervention, an imaging controller for controlling a generation of the interventional imaging sequence by the imaging device, and further employing a visual image sequence controller for controlling an augmentation of a visual imaging sequence acquisition of the interventional imaging sequence in a rendered augmented view of the image-guided intervention by an augmented reality device.
- the visual image sequence controller is configured to (1) access geometric information of an imaging modality associated with an acquisition of an interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, (2) generate a visual imaging sequence acquisition of the interventional imaging sequence including, for each interventional image of the interventional imaging sequence, an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence and of an imaging parameter of the imaging modality during an acquisition of the corresponding interventional image by the imaging modality, (3) augment the view of the image-guided intervention by the augmented reality device with the interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, and (4) interface with the at least one of the imaging modality and a display responsive to a user interaction with at least one of the interactive virtual indicators.
- imaging parameter can be any information indicative of the geometry (such as direction, rotation or angulation) or status of one or more components of the imaging modality (e.g. a C-arm, an
- auxiliary intervention device embodiments of the present disclosure encompass an auxiliary intervention device operable to communicate with an augmented reality device and an imaging modality.
- the auxiliary intervention further employs a visual image sequence controller for, in communication with the augmented reality device and the imaging modality, controlling an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in a rendered augmented view of an image-guided intervention by the augmented reality device.
- the visual image sequence controller is configured to (1) access geometric information of an imaging modality associated with an acquisition of an interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, (2) generate a visual imaging sequence acquisition of the interventional imaging sequence including, for each interventional image of the interventional imaging sequence, an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence and of an imaging parameter of the imaging modality during an acquisition of the corresponding interventional image by the imaging modality, (3) augment the view of the image-guided intervention by the augmented reality device with the interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, and (4) interface with the at least one of the imaging modality and a display responsive to a user interaction with at least one of the interactive virtual indicators.
- imaging parameter can be any information indicative of the geometry (such as direction, rotation or angulation) or status of one or more components of the imaging modality (e.g. a C-arm, an
- augmented image-guided intervention system embodiments of the present disclosure encompass an augmented image-guided intervention system employing an augmented reality device including an augmented reality display for viewing an image-guided intervention and further employing a visual image sequence controller for controlling an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in the rendered augmented view of the image-guided intervention by the augmented reality device.
- the visual image sequence controller is configured to (1) access geometric information of an imaging modality associated with an acquisition of an interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, (2) generate a visual imaging sequence acquisition of the interventional imaging sequence including, for each interventional image of the interventional imaging sequence, an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence and of an imaging parameter of the imaging modality during an acquisition of the corresponding interventional image by the imaging modality, (3) augment the view of the image-guided intervention by the augmented reality device with the interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, and (4) interface with the at least one of the imaging modality and a display responsive to a user interaction with at least one of the interactive virtual indicators.
- the visual image sequence controller is installable, or is installed within the augmented reality device, the imaging modality or an auxiliary intervention device.
- augmented image-guided intervention method embodiments of the present disclosure encompass an augmented image-guided intervention method involving an augmented reality device rendering an augmented view of an image-guided intervention and further involving a visual image sequence controller controlling an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in the rendered augmented view of the image-guided intervention by the augmented reality device.
- the visual image sequence controller (1) accesses geometric information of an imaging modality associated with an acquisition of an interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, (2) generates a visual imaging sequence acquisition of the interventional imaging sequence including, for each interventional image of the interventional imaging sequence, an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence and of an imaging parameter of the imaging modality during an acquisition of the corresponding interventional image by the imaging modality, (3) augments the view of the image-guided intervention by the augmented reality device with the interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, and (4) interfaces with the at least one of the imaging modality and a display responsive to a user interaction with at least one of the interactive virtual indicators.
- imaging parameter can be any information indicative of the geometry (such as direction, rotation or angulation) or status of one or more components of the imaging modality (e.g. a C-
- Various visual image sequencing method embodiments of the present disclosure encompass visual image sequencing method involving a visual image sequence controller utilizing a linear listing of interventional images to generate an interactive virtual indicator for each interventional image of a visual imaging sequence acquisition by an imaging modality.
- Each interactive virtual indicator indicates a sequence number of a corresponding interventional image within the interventional imaging sequence and an imaging parameter of the imaging modality during an acquisition of the corresponding interventional image.
- the visual image sequencing method further involves the visual image sequence controller controlling an augmentation of augmented view of the image-guided intervention with the generated interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention.
- FIG. 1 illustrates an exemplary embodiment of an augmented image-guided intervention system in accordance with the present disclosure
- FIG. 2 illustrates an exemplary embodiment of augmented image-guided intervention in accordance with the present disclosure
- FIGS. 3 A- 3 C illustrate a first exemplary interventional X-ray imaging in accordance with the present disclosure
- FIG. 4 illustrates an augmented reality of the interventional X-ray imaging of FIGS. 3 A- 3 C in accordance with the present disclosure
- FIG. 5 illustrates an X-ray image display of the interventional X-ray imaging of FIGS. 3 A- 3 C in accordance with the present disclosure
- FIG. 6 illustrates a reorientation of an X-ray modality in accordance with the present disclosure
- FIGS. 7 A- 7 C illustrate a second exemplary interventional X-ray imaging in accordance with the present disclosure
- FIG. 8 illustrates an augmented reality of the interventional X-ray imaging of FIGS. 7 A- 7 C in accordance with the present disclosure
- FIG. 9 illustrates an X-ray image display of the interventional X-ray imaging of FIGS. 7 A- 7 C in accordance with the present disclosure
- FIGS. 10 A- 10 C illustrate a first exemplary interventional ultrasound imaging in accordance with the present disclosure
- FIG. 11 illustrates an augmented reality of the interventional ultrasound imaging of FIGS. 10 A- 10 C in accordance with the present disclosure
- FIG. 12 illustrates an ultrasound image display of the interventional ultrasound imaging of FIGS. 10 A- 10 C in accordance with the present disclosure
- FIG. 13 illustrates a reorientation of an ultrasound modality in accordance with the present disclosure
- FIGS. 14 A- 14 C illustrate a second exemplary interventional ultrasound imaging in accordance with the present disclosure
- FIG. 15 illustrates an augmented reality of the interventional ultrasound imaging of FIGS. 14 A- 14 C in accordance with the present disclosure
- FIG. 16 illustrates an ultrasound image display of the interventional ultrasound imaging of FIGS. 14 A- 14 C in accordance with the present disclosure
- FIGS. 17 A- 17 C illustrate exemplary embodiments of the augmented image-guided intervention system of FIG. 1 in accordance with the present disclosure
- FIG. 18 illustrates an exemplary embodiment of a visual image sequence controller in accordance with the present disclosure.
- FIG. 19 illustrates an exemplary embodiment of a flowchart representative of a visual image sequencing method in accordance with the present disclosure
- the present disclosure is applicable to image-guided intervention implementing minimally-invasive imaging to diagnose and/or treat anatomical diseases (e.g., X-ray interventional imaging of organs, ultrasound interventional imaging of organs, etc.).
- anatomical diseases e.g., X-ray interventional imaging of organs, ultrasound interventional imaging of organs, etc.
- the present disclosure improves upon the prior art of image-guided intervention by providing unique embodiments of 2D/3D interactive virtual indicators in an augmented reality of an image-guided intervention (live or recorded) for graphically displaying, within a physical world, geometric information of an imaging modality associated with a sequential acquisition of 2D/3D interventional images to thereby illustrate a sequence of a portion or an entirety of the image-guided intervention to a clinician in a natural way.
- FIGS. 1 and 2 teaches respective exemplary embodiments of an augmented image-guided intervention system and an augmented image-guided intervention method in accordance with the present disclosure. From the description of FIGS. 1 and 2 , those having ordinary skill in the art of the present disclosure will appreciate how to apply the present disclosure to make and use additional embodiments of augmented image-guided intervention system of the present disclosure and augmented image-guided intervention method of the present disclosure.
- an exemplary augmented image-guided intervention system of the present disclosure employs an imaging modality 20 , an augmented reality device 30 and a visual image sequence controller 40 .
- imaging modality encompasses all systems, as known in the art of the present disclosure and hereinafter conceived, for implementing an image-guided intervention procedure by directing energy (e.g., X-ray beams, ultrasound, radio waves, magnetic fields, light, electrons, lasers, and radionuclides) into an anatomy for purposes of generating images of the anatomy (e.g., biological tissues and bone).
- energy e.g., X-ray beams, ultrasound, radio waves, magnetic fields, light, electrons, lasers, and radionuclides
- an imaging modality include, but are not limited to, interventional X-ray imaging systems and interventional ultrasound systems.
- imaging modality 20 includes an interventional imaging controller 22 for controlling an activation/deactivation of an interventional imaging device 21 (e.g., an X-ray C-arm, an ultrasound probe, etc.) to systematically direct energy into an anatomy via operator-generated commands and/or image guided procedural-generated commands for purposes of generating images of the anatomy as known in the art of the present disclosure.
- an interventional imaging controller 22 for controlling an activation/deactivation of an interventional imaging device 21 (e.g., an X-ray C-arm, an ultrasound probe, etc.) to systematically direct energy into an anatomy via operator-generated commands and/or image guided procedural-generated commands for purposes of generating images of the anatomy as known in the art of the present disclosure.
- an interventional imaging controller 22 for controlling an activation/deactivation of an interventional imaging device 21 (e.g., an X-ray C-arm, an ultrasound probe, etc.) to systematically direct energy into an anatomy via operator-generated commands and/or image guided procedural-generated commands for purposes of generating images
- augmented reality device encompasses all devices, as known in the art of the present disclosure and hereinafter conceived, for implementing an interactive experience of overlaying virtual object(s) in a physical world of image-guided intervention based on an accurate spatial registration of the augmented reality device to the physical world of image-guided intervention that facilitates a consistent positioning and orienting of the virtual object(s) in the real-world image-guided intervention.
- Examples of an augmented reality device include, but are not limited to, Microsoft Hololens®, Microsoft HoloLens®v2, DAQRI Smart Glasses ®, Magic Leap®, Vusix Blade® and Meta2®.
- augmented reality device 30 includes an augmented reality controller 32 for controlling augmented reality display 31 to display virtual object(s) in a physical world of image-guided intervention based on an accurate spatial registration of augmented reality display 31 to imaging modality 20 and/or the subject anatomy that facilitates a consistent positioning and orienting of the virtual object(s) to the subject anatomy and an operater-interaction with the virtual object(s).
- augmented reality controller 32 for controlling augmented reality display 31 to display virtual object(s) in a physical world of image-guided intervention based on an accurate spatial registration of augmented reality display 31 to imaging modality 20 and/or the subject anatomy that facilitates a consistent positioning and orienting of the virtual object(s) to the subject anatomy and an operater-interaction with the virtual object(s).
- the term “visual image sequence controller” encompasses all structural configurations, as understood in the art of the present disclosure and as exemplary described in the present disclosure, of a main circuit board or an integrated circuit for controlling an application of various principles of the present disclosure for an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in a rendered augmented view of the image-guided intervention by an augmented reality device as exemplary described in the present disclosure.
- the structural configuration of the controller may include, but is not limited to, processor(s), computer-usable/computer readable storage medium(s), an operating system, application module(s), peripheral device controller(s), slot(s) and port(s).
- the term “application module” broadly encompasses an application incorporated within or accessible by a controller consisting of an electronic circuit (e.g., electronic components and/or hardware) and/or an executable program (e.g., executable software stored on non-transitory computer readable medium(s) and/or firmware) for executing a specific application associated with implementing an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in a rendered augmented view of the image-guided intervention by the augmented reality device
- visual image sequence controller 40 access an intervention file 41 informative of geometric information of imaging modality 20 associated with an acquisition of 2D/3D interventional images to thereby control an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence by imaging modality 20 in a rendered augmented view of the image-guided intervention by augmented reality device 30 .
- the visual imaging sequence acquisition includes, for each interventional image of the interventional imaging sequence, an interactive virtual indicator of a sequence number and an imaging direction during an acquisition of the interventional image by imaging modality 20 .
- augmented reality controller 32 facilitates a rendering of computer generated content as overlay on an augmented view of an image-guided intervention.
- the visual imaging acquisition 42 can be rendered such that it appears to be part of the image-guided intervention.
- augmented reality device 30 and the subject anatomy need to be accurately tracked. This may be accomplished via a camera (e.g., visual or ToF) of augmented reality device 30 as known in the art of the present disclosure and/or a surgical navigation system as known in the art of the present disclosure. Additional detection of augmented reality device 30 and the subject anatomy adds may be appropriate for additional accuracy, such as, for example, fiducial marker(s), active marker(s), etc.
- visual image sequence controller 40 may interface with a display (not shown), as known in the art of the present disclosure or hereinafter conceived, for displaying an interventional image as commanded by an user-activation of an interactive visual indicator of the present disclosure.
- Visual image sequence controller 40 may interface with interventional imaging controller 22 , as known in the art of the present disclosure or hereinafter conceived, for controlling a re-posing of interventional imaging device 21 as commanded by user-activation of an interactive visual indicator of the present disclosure.
- An augmented reality image-guided intervention method of the present disclosure involves, subsequent to or during an acquisition of an interventional imaging sequence by imaging modality 20 , augmented reality controller 32 controlling an augmentation of an augmented view of the image-guided intervention via an augment reality display 32 of a visual image sequencing acquisition 42 generated by visual image sequencing controller 40 to include interactive virtual indicators of sequence numbers and imaging directions of the interventional imaging sequence.
- FIG. 2 illustrates an exemplary execution of an augmented image-guided intervention method of the present disclosure by the augmented reality image-guided intervention system of FIG. 1 .
- imaging modality 20 is shown in the form of an X-ray imaging system having a C-arm 21 a for acquiring an interventional imaging sequence 23 at different poses of C-arm 21 a .
- the X-ray imaging system has acquired of four (4) sequential X-ray images, each at a different pose of C-arm 21 a and interventional imaging controller 22 generates a database listing of each X-ray image with corresponding C-arm pose information and an optional annotation by a clinician.
- visual image sequence controller 40 accesses an interventional file 41 consisting of the database listing by interventional imaging controller 22 to generate visual imaging sequence acquisition 42 a including, for each interventional image of the interventional imaging sequence 23 , an interactive virtual indicator of a sequence number and an imaging direction.
- Augmented reality controller 32 augments an augmented view of the image-guided intervention via an augmented reality display 31 a with the visual imaging sequence acquisition 42 a of four (4) interactive virtual indicators.
- an interactive virtual indicator may have any form, shape and color to convey a corresponding sequence number and imaging direction.
- the interactive virtual indicators are formed as vectors of an arrow shape starting from the center of the projection on the patient and extending outward to the center of the detector position. Next to the end-point of the vector, the sequence number of the interventional image appears.
- An interactive virtual indicator may be any virtual indicator with witch a user, such as a clinician, is able to interact with; allowing a (two-way) communication or flow of information between a user one or more computers or one or more processors or any device embedding or connected to such computer(s) or processor(s).
- a clinician may interact with an interactive virtual indicator via using gestures, voice commands or other interactive modes of augmented reality display 31 a as known in the art of the present disclosure or hereinafter conceived.
- the interventional images of the interventional image sequence 23 are hidden by default, as not to clutter augmented reality display 31 a . Only a ‘selected’ interactive virtual indicator will show a pictorial image 33 corresponding to the ‘selected’ interactive virtual indicator within a display as known in the art of the present disclosure or hereinafter disclosed (e.g., a display of imaging modality 20 , augmented reality display 31 or an additional display monitor).
- the interactive virtual indicators may correspond to interventional images acquired at different poses of C-arm 21 a
- the image-guided intervention may involve an acquisitions of a temporal series of interventional images at the same pose of C-arm 21 a as will be further discussed in the present disclosure.
- the interactive virtual indicator may be a vector showing the imaging direction of the C-arm 21 a with an associated stack of sequence numbers.
- FIGS. 3 - 16 provides exemplary image-guided intervention in accordance with the present disclosure. From the description of FIGS. 3 - 16 , those having ordinary skill in the art of the present disclosure will appreciate how to apply the present disclosure to any type of image-guided intervention.
- the first exemplary image-guided intervention involves a diagnosis X-ray imaging phase as shown in FIGS. 3 - 6 and a treatment X-ray imaging phase as shown in FIGS. 7 - 9 .
- FIGS. 3 A- 3 C illustrates an acquisition of an interventional imaging sequence of diagnostic X-ray images 23 a - 23 c at different poses of C-arm 21 a .
- FIG. 4 illustrates an augmented view of the diagnostic phase of the image-guided intervention via an augmented reality display 31 a including three (3) interactive virtual indicators of a sequence number and image direction of diagnostic X-ray images 23 a - 23 c , respectively.
- An image 33 a corresponding to a selected interactive virtual indicator along with any annotation 43 a may also be displayed in augmented reality display 31 a and/or the diagnostic X-ray image 23 a - 23 c corresponding to selected interactive virtual indicator may be displayed via a display 24 b.
- a clinician may tap the correct interactive virtual indicator or a virtual object C-arm (not shown) and an interventional imaging controller will move the C-arm to the recalled pose.
- an interventional imaging controller will move the C-arm to the recalled pose.
- the clinician may tap interactive virtual indicator 1 to move the C-arm 21 a from the pose of FIG. 3 C to the pose of FIG. 3 A as exemplary shown in FIG. 6 .
- FIGS. 7 A- 7 C illustrate a temporal acquisition of an interventional imaging sequence of treatment X-ray images 23 d - 23 f at the same pose of C-arm 21 a .
- FIG. 8 illustrates an augmented view of the treatment phase of the image-guided intervention via an augmented reality display 31 a including six (6) interactive virtual indicators of a sequence number and image direction of diagnostic X-ray images 23 a - 23 c and treatment X-ray images 23 d - 23 f , respectively.
- An images sequence 33 b corresponding to a selected interactive virtual indicator of the treatment X-ray images 23 d - 23 f along with any associated annotation 43 b may also be displayed in augmented reality display 31 a and/or treatment X-ray images 23 d - 23 f corresponding to selected interactive virtual indicator may be displayed via a display 24 b.
- the second exemplary image-guided intervention involves a diagnosis ultrasound imaging phase as shown in FIGS. 10 - 13 and a treatment ultrasound imaging phase as shown in FIGS. 14 - 16 .
- FIGS. 10 A- 10 C illustrates an acquisition of an interventional imaging sequence of diagnostic ultrasound images 23 g - 23 i at different poses of robotic ultrasound probe 21 b .
- FIG. 11 illustrates an augmented view of the diagnostic phase of the image-guided intervention via an augmented reality display 31 a including three (3) interactive virtual indicators of a sequence number and image direction of diagnostic ultrasound images 23 g - 23 i , respectively.
- An image 33 c corresponding to a selected interactive virtual indicator along with any annotation 43 c may also be displayed in augmented reality display 31 a and/or the diagnostic ultrasound image 23 g - 23 i corresponding to selected interactive virtual indicator may be displayed via a display 24 c.
- a clinician may tap the correct interactive virtual indicator or a virtual object robotic ultrasound probe (not shown) and an interventional imaging controller will move the robotic ultrasound probe to the recalled pose.
- robotic ultrasound probe 21 b is in the pose of FIG. 10 C
- the clinician may tap interactive virtual indicator 1 to move the robotic ultrasound probe 21 b from the pose of FIG. 10 C to the pose of FIG. 10 A as exemplary shown in FIG. 13 .
- FIGS. 14 A- 14 C illustrate a temporal acquisition of an interventional imaging sequence of treatment ultrasound images 23 j - 23 l at the same pose of robotic ultrasound probe 21 b .
- FIG. 15 illustrates an augmented view of the treatment phase of the image-guided intervention via an augmented reality display 31 a including six (6) interactive virtual indicators of a sequence number and image direction of diagnostic ultrasound images 23 g - 23 i and treatment ultrasound images 23 j - 23 l, respectively.
- An images sequence 33 d corresponding to a selected interactive virtual indicator of treatment ultrasound images 23 j - 23 l along with any associated annotation 43 d may also be displayed in augmented reality display 31 a and/or treatment ultrasound images 23 j - 23 l corresponding to selected interactive virtual indicator may be displayed via a display 24 b.
- FIGS. 17 A- 17 C teach exemplary embodiments of an installation of a visual image sequencing controller of the present disclosure in various devices. From this description, those having ordinary skill in the art will appreciate how to apply various aspects of the present disclosure for making and using installation of a visual image sequencing controller of the present disclosure in numerous and various devices of the present disclosure.
- a visual imaging sequence controller 40 a is in installed within an augmented reality device 30 a , whereby communication channel(s) 50 a between an imaging modality 20 a and augmented reality device 30 a facilitates a communication of a linear listing of interventional images from imaging modality 20 a to visual imaging sequence controller 40 a to thereby support the augmentation of an augmented view of an image-guided intervention as previously described in the present disclosure.
- This embodiment is suitable for a retrofitting of a visual imaging sequence controller of the present disclosure in augmented reality devices known in the art of the present disclosure or for incorporation of a visual imaging sequence controller of the present disclosure in augmented reality devices hereinafter conceived.
- a visual imaging sequence controller 40 b is in installed within an imaging modality 20 b , whereby communication channel(s) 50 b between an imaging modality 20 b and augmented reality device 30 b facilitates a communication of a visual imaging sequence acquisition of an interventional imaging sequence in a rendered augmented view of the image-guided intervention by the augmented reality device 30 b to thereby support the augmentation of an augmented view of an image-guided intervention as previously described in the present disclosure.
- This embodiment is suitable for a retrofitting of a visual imaging sequence controller of the present disclosure in imaging modalities known in the art of the present disclosure or for incorporation of a visual imaging sequence controller of the present disclosure in imaging modalities hereinafter conceived.
- a visual imaging sequence controller 40 c is in installed within in an auxiliary intervention device 50 , whereby communication channel(s) 50 c between auxiliary intervention device 50 and an imaging modality 20 c and communication channel(s) 50 d between auxiliary intervention device 50 and an augmented reality device 30 c facilitates a communication of a linear listing of interventional images from imaging modality 20 c to visual imaging sequence controller 40 c and a communication of a visual imaging sequence acquisition of an interventional imaging sequence in a rendered augmented view of the image-guided intervention by the augmented reality device 30 c to thereby support the augmentation of an augmented view of an image-guided intervention as previously described in the present disclosure.
- This embodiment is suitable for a non-retrofitting/non-incorporation of a visual imaging sequence controller of the present disclosure in imaging modalities and augmented reality devices known in the art of the present disclosure or hereinafter conceived.
- FIGS. 18 and 19 respectively teach an exemplary embodiment of a visual image sequence controller of the present disclosure and an exemplary embodiment of a visual image sequencing method of the present disclosure. From this description, those having ordinary skill in the art will appreciate how to apply various aspects of the present disclosure for making and using additional embodiments of a visual image sequence controller of the present disclosure and a visual image sequencing method of the present disclosure.
- a visual image sequence controller 140 includes one or more processor(s) 141 , memory 142 , a user interface 143 , a network interface 144 , and a storage 145 interconnected via one or more system buses 146 .
- Each processor 141 may be any hardware device, as known in the art of the present disclosure or hereinafter conceived, capable of executing instructions stored in memory 142 or storage or otherwise processing data.
- the processor(s) 141 may include a microprocessor, field programmable gate array (FPGA), application-specific integrated circuit (ASIC), or other similar devices.
- the memory 142 may include various memories, as known in the art of the present disclosure or hereinafter conceived, including, but not limited to, L1, L2, or L3 cache or system memory.
- the memory 142 may include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read only memory (ROM), or other similar memory devices.
- the user interface 143 may include one or more devices, as known in the art of the present disclosure or hereinafter conceived, for enabling communication with a user such as an administrator.
- the user interface may include a command line interface or graphical user interface that may be presented to a remote terminal via the network interface 144 .
- the network interface 144 may include one or more devices, as known in the art of the present disclosure or hereinafter conceived, for enabling communication with other hardware devices.
- the network interface 144 may include a network interface card (NIC) configured to communicate according to the Ethernet protocol.
- NIC network interface card
- the network interface 144 may implement a TCP/IP stack for communication according to the TCP/IP protocols.
- TCP/IP protocols Various alternative or additional hardware or configurations for the network interface 144 will be apparent.
- the storage 145 may include one or more machine-readable storage media, as known in the art of the present disclosure or hereinafter conceived, including, but not limited to, read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash-memory devices, or similar storage media.
- ROM read-only memory
- RAM random-access memory
- magnetic disk storage media magnetic disk storage media
- optical storage media flash-memory devices
- similar storage media may store instructions for execution by the processor(s) 141 or data upon with the processor(s) 141 may operate.
- the storage 145 may store a base operating system for controlling various basic operations of the hardware.
- the storage 145 also stores application modules 147 in the form of executable software/firmware for implementing the various functions of the visual image sequence controller 140 as previously described in the present disclosure including, but not limited to, a virtual object generator 148 and a virtual object augmenter 149 for executing a flowchart 240 representative of a visual imaging sequencing method of the present disclosure as shown in FIG. 19 .
- a stage S 242 of flowchart 240 encompasses virtual object generator 148 utilizing a linear listing of interventional images to generate interactive virtual indicators as previously described in the present disclosure
- a stage S 244 of flowchart 240 encompasses virtual object augmenter 149 augmenting an augmented view of an image-guided intervention with the generated visual object indicators as previously described in the present disclosure.
- FIGS. 1 - 19 those having ordinary skill in the art of the present disclosure will appreciate numerous benefits of the present disclosure including, but not limited to, a more intuitive and less restrictive way of selecting interventional images for diagnostic and/or treatment purposes during an image-guided intervention.
- structures, elements, components, etc. described in the present disclosure/specification and/or depicted in the Figures may be implemented in various combinations of hardware and software, and provide functions which may be combined in a single element or multiple elements.
- the functions of the various structures, elements, components, etc. shown/illustrated/depicted in the Figures can be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software for added functionality.
- the functions can be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which can be shared and/or multiplexed.
- processor or “controller” should not be construed to refer exclusively to hardware capable of executing software, and can implicitly include, without limitation, digital signal processor (“DSP”) hardware, memory (e.g., read only memory (“ROM”) for storing software, random access memory (“RAM”), non-volatile storage, etc.) and virtually any means and/or machine (including hardware, software, firmware, combinations thereof, etc.) which is capable of (and/or configurable) to perform and/or control a process.
- DSP digital signal processor
- ROM read only memory
- RAM random access memory
- non-volatile storage etc.
- machine including hardware, software, firmware, combinations thereof, etc.
- any flow charts, flow diagrams and the like can represent various processes which can be substantially represented in computer readable storage media and so executed by a computer, processor or other device with processing capabilities, whether or not such computer or processor is explicitly shown.
- signal broadly encompasses all forms of a detectable physical quantity or impulse (e.g., voltage, current, or magnetic field strength) as understood in the art of the present disclosure and as exemplary described in the present disclosure for transmitting information and/or instructions in support of applying various inventive principles of the present disclosure as subsequently described in the present disclosure.
- Signal/data/command communication between various components of the present disclosure may involve any communication method as known in the art of the present disclosure including, but not limited to, signal/data/command transmission/reception over any type of wired or wireless datalink and a reading of signal/data/commands uploaded to a computer-usable/computer readable storage medium.
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Public Health (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Primary Health Care (AREA)
- Epidemiology (AREA)
- Biomedical Technology (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Surgery (AREA)
- General Business, Economics & Management (AREA)
- Business, Economics & Management (AREA)
- Life Sciences & Earth Sciences (AREA)
- Radiology & Medical Imaging (AREA)
- Heart & Thoracic Surgery (AREA)
- Veterinary Medicine (AREA)
- Molecular Biology (AREA)
- Animal Behavior & Ethology (AREA)
- Urology & Nephrology (AREA)
- Pathology (AREA)
- Gynecology & Obstetrics (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Robotics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Apparatus For Radiation Diagnosis (AREA)
- Ultra Sonic Daignosis Equipment (AREA)
- Medical Treatment And Welfare Office Work (AREA)
Abstract
Various embodiments of the present disclosure encompass a visual image sequence controller (40) for controlling an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in a rendered augmented view of an image-guided intervention by an augmented reality device (30). The visual imaging sequence acquisition (42) includes, for each inter-ventional image of the interventional imaging sequence (23), an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence (23) and an imaging parameter of the imaging modality (20) (e.g., a direction, a rotation and/or an angulation of the imaging modality (20)) during an acquisition of the corresponding interventional image by the imaging modality (20). The controller (40) interfaces with the imaging modality (20) and/or a display (24, 31) responsive to a user interaction with at least one of the interactive virtual indicators.
Description
- The present disclosure generally relates to image-guided interventions for a diagnosis and/or a treatment of a three-dimensional (3D) anatomy. The present disclosure specifically relates to an incorporation of augmented reality into image-guided interventions.
- Two-dimensional (2D) interventional images (e.g., X-ray images, ultrasound images, etc.) are inherently projections of complex three-dimensional (3D) anatomy. To get a sense of a structure of the 3D anatomy during an imaging guided-diagnosis, multiple projections of the 3D anatomy taken from multiple angles of a 2D imaging modality relative to the anatomy are needed. For this reason, in an image-guided intervention, a clinician is provided an opportunity to control a positioning and/or an orientation of the 2D imaging modality to thereby acquire 2D interventional images from a variety of angles (e.g., a clinician manual or programmed control of a C-arm of an X-ray system, a clinician manual or programmed control of a robotic ultrasound probe, etc.).
- After a diagnosis of a diseased 3D anatomy via the 2D/3D interventional images, the clinician may decide to treat the diseased 3D anatomy by using interventional tools at the disposal of the clinician. For such an image-guided surgery, the imaging modality is designed to be able to find the optimal viewing angle for visualizing the diseased area as well as positioning and/or an orientating of interventional tools within the diseased area.
- In all, during a typical image-guided intervention for diagnosis and/or a treatment of 3D anatomy, a large number of images are taken from a large number of different angles.
- More particularly, the clinician frequently needs to access the 2D interventional images taken from previous positions (or even a previous procedure) to validate the diagnosis of the 3D anatomy, to check for anomalies within the 3D anatomy and/or to validate the accuracy of the treatment of the 3D anatomy. Although an imaging modality stores geometry information with the 2D interventional images (e.g., an angulation/rotation of a C-arm, a mechanical scanning of an ultrasound probe, etc.), such geometry information is only available as a linear list. Thus, the clinician needs to make a mental picture of the image-guided intervention to know where an image sequence was taken at which time(s) and at which location(s). Especially for novice users and very long procedures, it is sometimes difficult to get an overview of the whole procedure. So, a clinician typically must navigate through an image sequence linearly, one by one, until they find the 2D interventional image(s) for validating the diagnosis of the 3D anatomy, checking for anomalies within the 3D anatomy and/or validating the accuracy of the treatment of the 3D anatomy.
- It is an object of the present disclosure to provide novel, unique 2D/3D interactive virtual indicators in an augmented reality of an image-guided intervention (e.g., X-ray interventional imaging for diagnosis and/or treatment of 3D anatomy, ultrasound interventional imaging for diagnosis and/or treatment of 3D anatomy, etc.) for graphically displaying, within a physical world, geometric information of an imaging modality associated with a sequential acquisition of 2D/3D interventional images. This advantageously enable interaction of a user, such as a clinician, with a sequence of a portion or an entirety of the image-guided intervention in a natural, yet intuitive way, thereby improving workflows, decision speed and diagnosis and/or treatment quality by enabling the clinician to ascertain an overview of a diagnostic phase and/or a treatment phase of the image-guided intervention.
- The present disclosure may be embodied as (1) a visual image sequence controller, (2) an augmented reality device incorporating a visual image sequence controller of the present disclosure, (3) an imaging modality incorporating a visual image sequence controller of the present disclosure, (4) an auxiliary intervention device incorporating a visual image sequence controller of the present disclosure, (5) an augmented image-guided intervention system incorporating an augmented reality device of the present disclosure, an imaging modality of the present disclosure and/or an auxiliary intervention device of the present disclosure, (6) an augmented image-guided intervention method utilizing a visual image sequence controller of the present disclosure; and (7) a visual image sequencing method of the present disclosure.
- Various visual image sequence controller embodiments of the present disclosure encompass a visual image sequence controller includes a non-transitory machine-readable storage medium encoded with instructions for execution by one or more processors to control an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence (live or recorded) in a rendered augmented view of an image-guided intervention by an augmented reality device. The non-transitory machine-readable storage medium includes instructions to (1) access geometric information of an imaging modality associated with an acquisition of an interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, (2) generate a visual imaging sequence acquisition of the interventional imaging sequence including, for each interventional image of the interventional imaging sequence, an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence and of an imaging parameter of the imaging modality during an acquisition of the corresponding interventional image by the imaging modality, (3) augment the view of the image-guided intervention by the augmented reality device with the interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, and (4) interface with the at least one of the imaging modality and a display responsive to a user interaction with at least one of the interactive virtual indicators. Such imaging parameter can be any information indicative of the geometry (such as direction, rotation or angulation) or status of one or more components of the imaging modality (e.g. a C-arm, an ultrasound probe, etc.) during the acquisition of the interventional image.
- Various augmented reality device embodiments of the present disclosure encompass an augmented reality device employing an augmented reality display for viewing an image-guided intervention (live or recorded) and further employing a visual image sequence controller for controlling an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in the rendered augmented view of the image-guided intervention by the augmented reality device. The visual image sequence controller is configured to (1) access geometric information of an imaging modality associated with an acquisition of an interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, (2) generate a visual imaging sequence acquisition of the interventional imaging sequence including, for each interventional image of the interventional imaging sequence, an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence and of an imaging parameter of the imaging modality during an acquisition of the corresponding interventional image by the imaging modality, (3) augment the view of the image-guided intervention by the augmented reality device with the interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, and (4) interface with the at least one of the imaging modality and a display responsive to a user interaction with at least one of the interactive virtual indicators. Such imaging parameter can be any information indicative of the geometry (such as direction, rotation or angulation) or status of one or more components of the imaging modality (e.g. a C-arm, an ultrasound probe, etc.) during the acquisition of the interventional image.
- Various imaging modality embodiments of the present disclosure encompass an imaging modality employing an imaging device for generating an interventional imaging sequence of an image-guided intervention, an imaging controller for controlling a generation of the interventional imaging sequence by the imaging device, and further employing a visual image sequence controller for controlling an augmentation of a visual imaging sequence acquisition of the interventional imaging sequence in a rendered augmented view of the image-guided intervention by an augmented reality device. The visual image sequence controller is configured to (1) access geometric information of an imaging modality associated with an acquisition of an interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, (2) generate a visual imaging sequence acquisition of the interventional imaging sequence including, for each interventional image of the interventional imaging sequence, an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence and of an imaging parameter of the imaging modality during an acquisition of the corresponding interventional image by the imaging modality, (3) augment the view of the image-guided intervention by the augmented reality device with the interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, and (4) interface with the at least one of the imaging modality and a display responsive to a user interaction with at least one of the interactive virtual indicators. Such imaging parameter can be any information indicative of the geometry (such as direction, rotation or angulation) or status of one or more components of the imaging modality (e.g. a C-arm, an ultrasound probe, etc.) during the acquisition of the interventional image.
- Various auxiliary intervention device embodiments of the present disclosure encompass an auxiliary intervention device operable to communicate with an augmented reality device and an imaging modality. The auxiliary intervention further employs a visual image sequence controller for, in communication with the augmented reality device and the imaging modality, controlling an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in a rendered augmented view of an image-guided intervention by the augmented reality device. The visual image sequence controller is configured to (1) access geometric information of an imaging modality associated with an acquisition of an interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, (2) generate a visual imaging sequence acquisition of the interventional imaging sequence including, for each interventional image of the interventional imaging sequence, an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence and of an imaging parameter of the imaging modality during an acquisition of the corresponding interventional image by the imaging modality, (3) augment the view of the image-guided intervention by the augmented reality device with the interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, and (4) interface with the at least one of the imaging modality and a display responsive to a user interaction with at least one of the interactive virtual indicators. Such imaging parameter can be any information indicative of the geometry (such as direction, rotation or angulation) or status of one or more components of the imaging modality (e.g. a C-arm, an ultrasound probe, etc.) during the acquisition of the interventional image.
- Various augmented image-guided intervention system embodiments of the present disclosure encompass an augmented image-guided intervention system employing an augmented reality device including an augmented reality display for viewing an image-guided intervention and further employing a visual image sequence controller for controlling an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in the rendered augmented view of the image-guided intervention by the augmented reality device. The visual image sequence controller is configured to (1) access geometric information of an imaging modality associated with an acquisition of an interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, (2) generate a visual imaging sequence acquisition of the interventional imaging sequence including, for each interventional image of the interventional imaging sequence, an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence and of an imaging parameter of the imaging modality during an acquisition of the corresponding interventional image by the imaging modality, (3) augment the view of the image-guided intervention by the augmented reality device with the interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, and (4) interface with the at least one of the imaging modality and a display responsive to a user interaction with at least one of the interactive virtual indicators. The visual image sequence controller is installable, or is installed within the augmented reality device, the imaging modality or an auxiliary intervention device.
- Various augmented image-guided intervention method embodiments of the present disclosure encompass an augmented image-guided intervention method involving an augmented reality device rendering an augmented view of an image-guided intervention and further involving a visual image sequence controller controlling an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in the rendered augmented view of the image-guided intervention by the augmented reality device. In operation, the visual image sequence controller (1) accesses geometric information of an imaging modality associated with an acquisition of an interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, (2) generates a visual imaging sequence acquisition of the interventional imaging sequence including, for each interventional image of the interventional imaging sequence, an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence and of an imaging parameter of the imaging modality during an acquisition of the corresponding interventional image by the imaging modality, (3) augments the view of the image-guided intervention by the augmented reality device with the interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention, and (4) interfaces with the at least one of the imaging modality and a display responsive to a user interaction with at least one of the interactive virtual indicators. Such imaging parameter can be any information indicative of the geometry (such as direction, rotation or angulation) or status of one or more components of the imaging modality (e.g. a C-arm, an ultrasound probe, etc.) during the acquisition of the interventional image.
- Various visual image sequencing method embodiments of the present disclosure encompass visual image sequencing method involving a visual image sequence controller utilizing a linear listing of interventional images to generate an interactive virtual indicator for each interventional image of a visual imaging sequence acquisition by an imaging modality. Each interactive virtual indicator indicates a sequence number of a corresponding interventional image within the interventional imaging sequence and an imaging parameter of the imaging modality during an acquisition of the corresponding interventional image. The visual image sequencing method further involves the visual image sequence controller controlling an augmentation of augmented view of the image-guided intervention with the generated interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence of interventional images by the imaging modality during the image-guided intervention.
- The foregoing embodiments and other embodiments of the present disclosure as well as various structures and advantages of the present disclosure will become further apparent from the following detailed description of various embodiments of the present disclosure read in conjunction with the accompanying drawings. The detailed description and drawings are merely illustrative of the present disclosure rather than limiting, the scope of the present disclosure being defined by the appended claims and equivalents thereof.
- The present disclosure will present in detail the following description of exemplary embodiments with reference to the following Figures wherein:
-
FIG. 1 illustrates an exemplary embodiment of an augmented image-guided intervention system in accordance with the present disclosure; -
FIG. 2 illustrates an exemplary embodiment of augmented image-guided intervention in accordance with the present disclosure; -
FIGS. 3A-3C illustrate a first exemplary interventional X-ray imaging in accordance with the present disclosure; -
FIG. 4 illustrates an augmented reality of the interventional X-ray imaging ofFIGS. 3A-3C in accordance with the present disclosure; -
FIG. 5 illustrates an X-ray image display of the interventional X-ray imaging ofFIGS. 3A-3C in accordance with the present disclosure; -
FIG. 6 illustrates a reorientation of an X-ray modality in accordance with the present disclosure; -
FIGS. 7A-7C illustrate a second exemplary interventional X-ray imaging in accordance with the present disclosure; -
FIG. 8 illustrates an augmented reality of the interventional X-ray imaging ofFIGS. 7A-7C in accordance with the present disclosure; -
FIG. 9 illustrates an X-ray image display of the interventional X-ray imaging ofFIGS. 7A-7C in accordance with the present disclosure; -
FIGS. 10A-10C illustrate a first exemplary interventional ultrasound imaging in accordance with the present disclosure; -
FIG. 11 illustrates an augmented reality of the interventional ultrasound imaging ofFIGS. 10A-10C in accordance with the present disclosure; -
FIG. 12 illustrates an ultrasound image display of the interventional ultrasound imaging ofFIGS. 10A-10C in accordance with the present disclosure; -
FIG. 13 illustrates a reorientation of an ultrasound modality in accordance with the present disclosure; -
FIGS. 14A-14C illustrate a second exemplary interventional ultrasound imaging in accordance with the present disclosure; -
FIG. 15 illustrates an augmented reality of the interventional ultrasound imaging ofFIGS. 14A-14C in accordance with the present disclosure; -
FIG. 16 illustrates an ultrasound image display of the interventional ultrasound imaging ofFIGS. 14A-14C in accordance with the present disclosure; -
FIGS. 17A-17C illustrate exemplary embodiments of the augmented image-guided intervention system ofFIG. 1 in accordance with the present disclosure; -
FIG. 18 illustrates an exemplary embodiment of a visual image sequence controller in accordance with the present disclosure; and -
FIG. 19 illustrates an exemplary embodiment of a flowchart representative of a visual image sequencing method in accordance with the present disclosure - The present disclosure is applicable to image-guided intervention implementing minimally-invasive imaging to diagnose and/or treat anatomical diseases (e.g., X-ray interventional imaging of organs, ultrasound interventional imaging of organs, etc.).
- The present disclosure improves upon the prior art of image-guided intervention by providing unique embodiments of 2D/3D interactive virtual indicators in an augmented reality of an image-guided intervention (live or recorded) for graphically displaying, within a physical world, geometric information of an imaging modality associated with a sequential acquisition of 2D/3D interventional images to thereby illustrate a sequence of a portion or an entirety of the image-guided intervention to a clinician in a natural way.
- To facilitate an understanding of the present disclosure, the following description of
FIGS. 1 and 2 teaches respective exemplary embodiments of an augmented image-guided intervention system and an augmented image-guided intervention method in accordance with the present disclosure. From the description ofFIGS. 1 and 2 , those having ordinary skill in the art of the present disclosure will appreciate how to apply the present disclosure to make and use additional embodiments of augmented image-guided intervention system of the present disclosure and augmented image-guided intervention method of the present disclosure. - Referring to
FIG. 1 , an exemplary augmented image-guided intervention system of the present disclosure employs animaging modality 20, anaugmented reality device 30 and a visualimage sequence controller 40. - For purposes of describing and claiming the present disclosure, the term “imaging modality” encompasses all systems, as known in the art of the present disclosure and hereinafter conceived, for implementing an image-guided intervention procedure by directing energy (e.g., X-ray beams, ultrasound, radio waves, magnetic fields, light, electrons, lasers, and radionuclides) into an anatomy for purposes of generating images of the anatomy (e.g., biological tissues and bone). Examples of an imaging modality include, but are not limited to, interventional X-ray imaging systems and interventional ultrasound systems.
- In practice,
imaging modality 20 includes aninterventional imaging controller 22 for controlling an activation/deactivation of an interventional imaging device 21 (e.g., an X-ray C-arm, an ultrasound probe, etc.) to systematically direct energy into an anatomy via operator-generated commands and/or image guided procedural-generated commands for purposes of generating images of the anatomy as known in the art of the present disclosure. - For purposes of describing and claiming the present disclosure, the term “augmented reality device” encompasses all devices, as known in the art of the present disclosure and hereinafter conceived, for implementing an interactive experience of overlaying virtual object(s) in a physical world of image-guided intervention based on an accurate spatial registration of the augmented reality device to the physical world of image-guided intervention that facilitates a consistent positioning and orienting of the virtual object(s) in the real-world image-guided intervention. Examples of an augmented reality device include, but are not limited to, Microsoft Hololens®, Microsoft HoloLens®v2, DAQRI Smart Glasses ®, Magic Leap®, Vusix Blade® and Meta2®.
- In practice,
augmented reality device 30 includes anaugmented reality controller 32 for controllingaugmented reality display 31 to display virtual object(s) in a physical world of image-guided intervention based on an accurate spatial registration ofaugmented reality display 31 toimaging modality 20 and/or the subject anatomy that facilitates a consistent positioning and orienting of the virtual object(s) to the subject anatomy and an operater-interaction with the virtual object(s). - For purposes of describing and claiming the present disclosure, the term “visual image sequence controller” encompasses all structural configurations, as understood in the art of the present disclosure and as exemplary described in the present disclosure, of a main circuit board or an integrated circuit for controlling an application of various principles of the present disclosure for an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in a rendered augmented view of the image-guided intervention by an augmented reality device as exemplary described in the present disclosure. The structural configuration of the controller may include, but is not limited to, processor(s), computer-usable/computer readable storage medium(s), an operating system, application module(s), peripheral device controller(s), slot(s) and port(s).
- For purposes of describing and claiming the present disclosure, the term “application module” broadly encompasses an application incorporated within or accessible by a controller consisting of an electronic circuit (e.g., electronic components and/or hardware) and/or an executable program (e.g., executable software stored on non-transitory computer readable medium(s) and/or firmware) for executing a specific application associated with implementing an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in a rendered augmented view of the image-guided intervention by the augmented reality device
- In practice, as exemplary described in the present disclosure, visual
image sequence controller 40 access anintervention file 41 informative of geometric information ofimaging modality 20 associated with an acquisition of 2D/3D interventional images to thereby control an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence by imagingmodality 20 in a rendered augmented view of the image-guided intervention byaugmented reality device 30. The visual imaging sequence acquisition includes, for each interventional image of the interventional imaging sequence, an interactive virtual indicator of a sequence number and an imaging direction during an acquisition of the interventional image by imagingmodality 20. - More particularly,
augmented reality controller 32 facilitates a rendering of computer generated content as overlay on an augmented view of an image-guided intervention. When a pose of the eyes are known byaugmented reality controller 32, thevisual imaging acquisition 42 can be rendered such that it appears to be part of the image-guided intervention. For this to happen,augmented reality device 30 and the subject anatomy need to be accurately tracked. This may be accomplished via a camera (e.g., visual or ToF) of augmentedreality device 30 as known in the art of the present disclosure and/or a surgical navigation system as known in the art of the present disclosure. Additional detection ofaugmented reality device 30 and the subject anatomy adds may be appropriate for additional accuracy, such as, for example, fiducial marker(s), active marker(s), etc. - Further in practice, visual
image sequence controller 40 may interface with a display (not shown), as known in the art of the present disclosure or hereinafter conceived, for displaying an interventional image as commanded by an user-activation of an interactive visual indicator of the present disclosure. Visualimage sequence controller 40 may interface withinterventional imaging controller 22, as known in the art of the present disclosure or hereinafter conceived, for controlling a re-posing ofinterventional imaging device 21 as commanded by user-activation of an interactive visual indicator of the present disclosure. - An augmented reality image-guided intervention method of the present disclosure involves, subsequent to or during an acquisition of an interventional imaging sequence by imaging
modality 20,augmented reality controller 32 controlling an augmentation of an augmented view of the image-guided intervention via an augmentreality display 32 of a visualimage sequencing acquisition 42 generated by visualimage sequencing controller 40 to include interactive virtual indicators of sequence numbers and imaging directions of the interventional imaging sequence. - For example,
FIG. 2 illustrates an exemplary execution of an augmented image-guided intervention method of the present disclosure by the augmented reality image-guided intervention system ofFIG. 1 . - Referring to
FIG. 2 , during an image-guided intervention,imaging modality 20 is shown in the form of an X-ray imaging system having a C-arm 21 a for acquiring aninterventional imaging sequence 23 at different poses of C-arm 21 a. For this image-guided intervention example, the X-ray imaging system has acquired of four (4) sequential X-ray images, each at a different pose of C-arm 21 a andinterventional imaging controller 22 generates a database listing of each X-ray image with corresponding C-arm pose information and an optional annotation by a clinician. - Still referring to
FIG. 2 , visualimage sequence controller 40 accesses aninterventional file 41 consisting of the database listing byinterventional imaging controller 22 to generate visualimaging sequence acquisition 42 a including, for each interventional image of theinterventional imaging sequence 23, an interactive virtual indicator of a sequence number and an imaging direction.Augmented reality controller 32 augments an augmented view of the image-guided intervention via anaugmented reality display 31 a with the visualimaging sequence acquisition 42 a of four (4) interactive virtual indicators. - In practice, an interactive virtual indicator may have any form, shape and color to convey a corresponding sequence number and imaging direction. For this example, the interactive virtual indicators are formed as vectors of an arrow shape starting from the center of the projection on the patient and extending outward to the center of the detector position. Next to the end-point of the vector, the sequence number of the interventional image appears.
- An interactive virtual indicator may be any virtual indicator with witch a user, such as a clinician, is able to interact with; allowing a (two-way) communication or flow of information between a user one or more computers or one or more processors or any device embedding or connected to such computer(s) or processor(s). A clinician may interact with an interactive virtual indicator via using gestures, voice commands or other interactive modes of
augmented reality display 31 a as known in the art of the present disclosure or hereinafter conceived. - The interventional images of the
interventional image sequence 23 are hidden by default, as not to clutter augmentedreality display 31 a. Only a ‘selected’ interactive virtual indicator will show a pictorial image 33 corresponding to the ‘selected’ interactive virtual indicator within a display as known in the art of the present disclosure or hereinafter disclosed (e.g., a display ofimaging modality 20,augmented reality display 31 or an additional display monitor). - While the interactive virtual indicators may correspond to interventional images acquired at different poses of C-
arm 21 a, the image-guided intervention may involve an acquisitions of a temporal series of interventional images at the same pose of C-arm 21 a as will be further discussed in the present disclosure. For this embodiment, the interactive virtual indicator may be a vector showing the imaging direction of the C-arm 21 a with an associated stack of sequence numbers. - To facilitate an understanding of the present disclosure, the following description of
FIGS. 3-16 provides exemplary image-guided intervention in accordance with the present disclosure. From the description ofFIGS. 3-16 , those having ordinary skill in the art of the present disclosure will appreciate how to apply the present disclosure to any type of image-guided intervention. - The first exemplary image-guided intervention involves a diagnosis X-ray imaging phase as shown in
FIGS. 3-6 and a treatment X-ray imaging phase as shown inFIGS. 7-9 . - More particularly,
FIGS. 3A-3C illustrates an acquisition of an interventional imaging sequence ofdiagnostic X-ray images 23 a-23 c at different poses of C-arm 21 a.FIG. 4 illustrates an augmented view of the diagnostic phase of the image-guided intervention via anaugmented reality display 31 a including three (3) interactive virtual indicators of a sequence number and image direction ofdiagnostic X-ray images 23 a-23 c, respectively. Animage 33 a corresponding to a selected interactive virtual indicator along with anyannotation 43 a may also be displayed inaugmented reality display 31 a and/or thediagnostic X-ray image 23 a-23 c corresponding to selected interactive virtual indicator may be displayed via adisplay 24 b. - Additionally, if a clinician wishes to return C-arm to a previous pose for purpose of reacquiring a better interventional image at that pose, the clinician may tap the correct interactive virtual indicator or a virtual object C-arm (not shown) and an interventional imaging controller will move the C-arm to the recalled pose. For example, while C-
arm 21 a is in the pose ofFIG. 3C , if the clinician has selected the interactive virtual indicator 1 associated with the C-arm pose ofFIG. 3A , the clinician may tap interactive virtual indicator 1 to move the C-arm 21 a from the pose ofFIG. 3C to the pose ofFIG. 3A as exemplary shown inFIG. 6 . -
FIGS. 7A-7C illustrate a temporal acquisition of an interventional imaging sequence oftreatment X-ray images 23 d-23 f at the same pose of C-arm 21 a.FIG. 8 illustrates an augmented view of the treatment phase of the image-guided intervention via anaugmented reality display 31 a including six (6) interactive virtual indicators of a sequence number and image direction ofdiagnostic X-ray images 23 a-23 c andtreatment X-ray images 23 d-23 f, respectively. Animages sequence 33 b corresponding to a selected interactive virtual indicator of thetreatment X-ray images 23 d-23 f along with any associatedannotation 43 b may also be displayed inaugmented reality display 31 a and/ortreatment X-ray images 23 d-23 f corresponding to selected interactive virtual indicator may be displayed via adisplay 24 b. - The second exemplary image-guided intervention involves a diagnosis ultrasound imaging phase as shown in
FIGS. 10-13 and a treatment ultrasound imaging phase as shown inFIGS. 14-16 . - More particularly,
FIGS. 10A-10C illustrates an acquisition of an interventional imaging sequence ofdiagnostic ultrasound images 23 g-23 i at different poses ofrobotic ultrasound probe 21 b.FIG. 11 illustrates an augmented view of the diagnostic phase of the image-guided intervention via anaugmented reality display 31 a including three (3) interactive virtual indicators of a sequence number and image direction ofdiagnostic ultrasound images 23 g-23 i, respectively. Animage 33 c corresponding to a selected interactive virtual indicator along with anyannotation 43 c may also be displayed inaugmented reality display 31 a and/or thediagnostic ultrasound image 23 g-23 i corresponding to selected interactive virtual indicator may be displayed via adisplay 24 c. - Additionally, if a clinician wishes to return robotic ultrasound probe to a previous pose for purpose of reacquiring a better interventional image at that pose, the clinician may tap the correct interactive virtual indicator or a virtual object robotic ultrasound probe (not shown) and an interventional imaging controller will move the robotic ultrasound probe to the recalled pose. For example, while
robotic ultrasound probe 21 b is in the pose ofFIG. 10C , if the clinician has selected the interactive virtual indicator 1 associated with the robotic ultrasound probe pose ofFIG. 10A , the clinician may tap interactive virtual indicator 1 to move therobotic ultrasound probe 21 b from the pose ofFIG. 10C to the pose ofFIG. 10A as exemplary shown inFIG. 13 . -
FIGS. 14A-14C illustrate a temporal acquisition of an interventional imaging sequence oftreatment ultrasound images 23 j-23 l at the same pose ofrobotic ultrasound probe 21 b.FIG. 15 illustrates an augmented view of the treatment phase of the image-guided intervention via anaugmented reality display 31 a including six (6) interactive virtual indicators of a sequence number and image direction ofdiagnostic ultrasound images 23 g-23 i andtreatment ultrasound images 23 j-23 l, respectively. Animages sequence 33 d corresponding to a selected interactive virtual indicator oftreatment ultrasound images 23 j-23 l along with any associatedannotation 43 d may also be displayed inaugmented reality display 31 a and/ortreatment ultrasound images 23 j-23 l corresponding to selected interactive virtual indicator may be displayed via adisplay 24 b. - To facilitate a further understanding of the various inventions of the present disclosure, the following description of
FIGS. 17A-17C teach exemplary embodiments of an installation of a visual image sequencing controller of the present disclosure in various devices. From this description, those having ordinary skill in the art will appreciate how to apply various aspects of the present disclosure for making and using installation of a visual image sequencing controller of the present disclosure in numerous and various devices of the present disclosure. - Referring to
FIG. 17A , a visualimaging sequence controller 40 a is in installed within anaugmented reality device 30 a, whereby communication channel(s) 50 a between animaging modality 20 a andaugmented reality device 30 a facilitates a communication of a linear listing of interventional images fromimaging modality 20 a to visualimaging sequence controller 40 a to thereby support the augmentation of an augmented view of an image-guided intervention as previously described in the present disclosure. This embodiment is suitable for a retrofitting of a visual imaging sequence controller of the present disclosure in augmented reality devices known in the art of the present disclosure or for incorporation of a visual imaging sequence controller of the present disclosure in augmented reality devices hereinafter conceived. - Referring to
FIG. 17B , a visualimaging sequence controller 40 b is in installed within animaging modality 20 b, whereby communication channel(s) 50 b between animaging modality 20 b andaugmented reality device 30 b facilitates a communication of a visual imaging sequence acquisition of an interventional imaging sequence in a rendered augmented view of the image-guided intervention by theaugmented reality device 30 b to thereby support the augmentation of an augmented view of an image-guided intervention as previously described in the present disclosure. This embodiment is suitable for a retrofitting of a visual imaging sequence controller of the present disclosure in imaging modalities known in the art of the present disclosure or for incorporation of a visual imaging sequence controller of the present disclosure in imaging modalities hereinafter conceived. - Referring to
FIG. 17C , a visual imaging sequence controller 40 c is in installed within in an auxiliary intervention device 50, whereby communication channel(s) 50 c between auxiliary intervention device 50 and animaging modality 20 c and communication channel(s) 50 d between auxiliary intervention device 50 and anaugmented reality device 30 c facilitates a communication of a linear listing of interventional images fromimaging modality 20 c to visual imaging sequence controller 40 c and a communication of a visual imaging sequence acquisition of an interventional imaging sequence in a rendered augmented view of the image-guided intervention by theaugmented reality device 30 c to thereby support the augmentation of an augmented view of an image-guided intervention as previously described in the present disclosure. This embodiment is suitable for a non-retrofitting/non-incorporation of a visual imaging sequence controller of the present disclosure in imaging modalities and augmented reality devices known in the art of the present disclosure or hereinafter conceived. - To facilitate a further understanding of the various inventions of the present disclosure, the following description of
FIGS. 18 and 19 respectively teach an exemplary embodiment of a visual image sequence controller of the present disclosure and an exemplary embodiment of a visual image sequencing method of the present disclosure. From this description, those having ordinary skill in the art will appreciate how to apply various aspects of the present disclosure for making and using additional embodiments of a visual image sequence controller of the present disclosure and a visual image sequencing method of the present disclosure. - Referring to
FIG. 18 , a visualimage sequence controller 140 includes one or more processor(s) 141,memory 142, a user interface 143, anetwork interface 144, and astorage 145 interconnected via one ormore system buses 146. - Each
processor 141 may be any hardware device, as known in the art of the present disclosure or hereinafter conceived, capable of executing instructions stored inmemory 142 or storage or otherwise processing data. In a non-limiting example, the processor(s) 141 may include a microprocessor, field programmable gate array (FPGA), application-specific integrated circuit (ASIC), or other similar devices. - The
memory 142 may include various memories, as known in the art of the present disclosure or hereinafter conceived, including, but not limited to, L1, L2, or L3 cache or system memory. In a non-limiting example, thememory 142 may include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read only memory (ROM), or other similar memory devices. - The user interface 143 may include one or more devices, as known in the art of the present disclosure or hereinafter conceived, for enabling communication with a user such as an administrator. In a non-limiting example, the user interface may include a command line interface or graphical user interface that may be presented to a remote terminal via the
network interface 144. - The
network interface 144 may include one or more devices, as known in the art of the present disclosure or hereinafter conceived, for enabling communication with other hardware devices. In a non-limiting example, thenetwork interface 144 may include a network interface card (NIC) configured to communicate according to the Ethernet protocol. Additionally, thenetwork interface 144 may implement a TCP/IP stack for communication according to the TCP/IP protocols. Various alternative or additional hardware or configurations for thenetwork interface 144 will be apparent. - The
storage 145 may include one or more machine-readable storage media, as known in the art of the present disclosure or hereinafter conceived, including, but not limited to, read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash-memory devices, or similar storage media. In various non-limiting embodiments, thestorage 145 may store instructions for execution by the processor(s) 141 or data upon with the processor(s) 141 may operate. For example, thestorage 145 may store a base operating system for controlling various basic operations of the hardware. Thestorage 145 also storesapplication modules 147 in the form of executable software/firmware for implementing the various functions of the visualimage sequence controller 140 as previously described in the present disclosure including, but not limited to, avirtual object generator 148 and a virtual object augmenter 149 for executing aflowchart 240 representative of a visual imaging sequencing method of the present disclosure as shown inFIG. 19 . - Referring to
FIG. 19 , a stage S242 offlowchart 240 encompassesvirtual object generator 148 utilizing a linear listing of interventional images to generate interactive virtual indicators as previously described in the present disclosure, and a stage S244 offlowchart 240 encompasses virtual object augmenter 149 augmenting an augmented view of an image-guided intervention with the generated visual object indicators as previously described in the present disclosure. - Referring to
FIGS. 1-19 , those having ordinary skill in the art of the present disclosure will appreciate numerous benefits of the present disclosure including, but not limited to, a more intuitive and less restrictive way of selecting interventional images for diagnostic and/or treatment purposes during an image-guided intervention. - Further, as one having ordinary skill in the art will appreciate in view of the teachings provided herein, structures, elements, components, etc. described in the present disclosure/specification and/or depicted in the Figures may be implemented in various combinations of hardware and software, and provide functions which may be combined in a single element or multiple elements. For example, the functions of the various structures, elements, components, etc. shown/illustrated/depicted in the Figures can be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software for added functionality. When provided by a processor, the functions can be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which can be shared and/or multiplexed. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and can implicitly include, without limitation, digital signal processor (“DSP”) hardware, memory (e.g., read only memory (“ROM”) for storing software, random access memory (“RAM”), non-volatile storage, etc.) and virtually any means and/or machine (including hardware, software, firmware, combinations thereof, etc.) which is capable of (and/or configurable) to perform and/or control a process.
- Moreover, all statements herein reciting principles, aspects, and embodiments of the invention, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future (e.g., any elements developed that can perform the same or substantially similar function, regardless of structure). Thus, for example, it will be appreciated by one having ordinary skill in the art in view of the teachings provided herein that any block diagrams presented herein can represent conceptual views of illustrative system components and/or circuitry embodying the principles of the invention. Similarly, one having ordinary skill in the art should appreciate in view of the teachings provided herein that any flow charts, flow diagrams and the like can represent various processes which can be substantially represented in computer readable storage media and so executed by a computer, processor or other device with processing capabilities, whether or not such computer or processor is explicitly shown.
- The terms “signal”, “data” and “command” as used in the present disclosure broadly encompasses all forms of a detectable physical quantity or impulse (e.g., voltage, current, or magnetic field strength) as understood in the art of the present disclosure and as exemplary described in the present disclosure for transmitting information and/or instructions in support of applying various inventive principles of the present disclosure as subsequently described in the present disclosure. Signal/data/command communication between various components of the present disclosure may involve any communication method as known in the art of the present disclosure including, but not limited to, signal/data/command transmission/reception over any type of wired or wireless datalink and a reading of signal/data/commands uploaded to a computer-usable/computer readable storage medium.
- It will be appreciated that the term “comprising” does not exclude other elements or steps and that the indefinite article “a” or “an” does not exclude a plurality. A single processor may fulfil the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to an advantage. Any reference signs in the claims should not be construed as limiting the scope of the claims.
- Although claims have been formulated in this application to particular combinations of features, it should be understood that the scope of the disclosure of the present invention also includes any novel features or any novel combinations of features disclosed herein either explicitly or implicitly or any generalisation thereof, whether or not it relates to the same invention as presently claimed in any claim and whether or not it mitigates any or all of the same technical problems as does the parent invention. The applicants hereby give notice that new claims may be formulated to such features and/or combinations of features during the prosecution of the present application or of any further application derived therefrom.
Claims (15)
1. A visual image sequence controller (40) for an augmented image-guided intervention system including an augmented reality device (30) operable to render an augmented view of an image-guided intervention, the visual image sequence controller (40) comprising:
a non-transitory machine-readable storage medium encoded with instructions for execution by at least one processor to control an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in a rendered augmented view of the image-guided intervention by the augmented reality device (30), wherein the non-transitory machine-readable storage medium includes instructions to:
access geometric information (41) of an imaging modality (20) associated with an acquisition of an interventional imaging sequence (23) of interventional images by the imaging modality (20) during the image-guided intervention;
generate the visual imaging sequence acquisition (42) of the interventional imaging sequence (23) including, for each interventional image of the interventional imaging sequence (23), an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence and of an imaging parameter of the imaging modality (20) during an acquisition of the corresponding interventional image by the imaging modality (20);
augment the view of the image-guided intervention by the augmented reality device (30) with the interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence (23) of interventional images by the imaging modality (20) during the image-guided intervention; and
interface with the at least one of the imaging modality (20) and a display (24, 31) responsive to a user interaction with at least one of the interactive virtual indicators.
2. The visual image sequence controller (40) of claim 1 , wherein the non-transitory machine-readable storage medium further includes instructions to annotate the visual imaging sequence acquisition (42) of the interventional imaging sequence (23).
3. The visual image sequence controller (40) of claim 1 , wherein the non-transitory machine-readable storage medium further includes instructions to control an image display interaction between the visual imaging sequence acquisition (42) of the interventional imaging sequence (23) and a display (24, 31).
4. The visual image sequence controller (40) of claim 1 , wherein the non-transitory machine-readable storage medium further includes instructions to control an imaging modality posing interaction between the visual imaging sequence acquisition (42) of the interventional imaging sequence (23) and the imaging modality (20).
5. The visual image sequence controller (40) of claim 1 , wherein the visual image sequence controller (40) is installable within the augmented reality device (30).
6. The visual image sequence controller (40) of claim 1 ,
wherein the visual image sequence controller (40) is installable within an imaging modality (20); and
wherein, when installed within an imaging modality (20), the non-transitory machine-readable storage medium further includes instructions to communicate the visual imaging sequence acquisition (42) of the interventional imaging sequence (23) from the imaging modality (20) to the augmented reality device (30).
7. The visual image sequence controller (40) of claim 1 ,
wherein the visual image sequence controller (40) is installable within an auxiliary intervention device; and
wherein, when installed within an auxiliary intervention device, the non-transitory machine-readable storage medium further includes instructions to communicate the visual imaging sequence acquisition (42) of the interventional imaging sequence (23) from the auxiliary intervention device to the augmented reality device (30).
8. An augmented image-guided intervention system, comprising:
an augmented reality device (30) operable to render an augmented view of an image-guided intervention; and
a visual image sequence controller (40) configured to control an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in a rendered augmented view of the image-guided intervention by the augmented reality device (30),
wherein the visual image sequence controller (40) is configured to:
access geometric information (41) of an imaging modality (20) associated with an acquisition of an interventional imaging sequence (23) of interventional images by the imaging modality (20) during the image-guided intervention;
generate the visual imaging sequence acquisition (42) of the interventional imaging sequence (23) including, for each interventional image of the interventional imaging sequence (23), an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence and of an imaging parameter of the imaging modality (20) during an acquisition of the corresponding interventional image by the imaging modality (20);
augment the view of the image-guided intervention by the augmented reality device (30) with the interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence (23) of interventional images by the imaging modality (20) during the image-guided intervention; and
interface with the at least one of the imaging modality (20) and a display (24, 31) responsive to a user interaction with at least one of the interactive virtual indicators.
9. The augmented image-guided intervention system of claim 8 , wherein the visual image sequence controller (40) is further configured to annotate the visual imaging sequence acquisition (42) of the interventional imaging sequence (23).
10. The augmented image-guided intervention system of claim 8 , wherein the visual image sequence controller (40) is further configured to control an image display interaction between the visual imaging sequence acquisition (42) of the interventional imaging sequence (23) and a display (24, 31).
11. The augmented image-guided intervention system of claim 8 , wherein the visual image sequence controller (40) is further configured to control an imaging modality posing interaction between the visual imaging sequence acquisition (42) of the interventional imaging sequence (23) and the imaging modality (20).
12. The augmented image-guided intervention system of claim 8 , wherein the augmented reality device (30) comprises the visual image sequence controller (40).
13. The augmented image-guided intervention system of claim 8 ,
wherein the visual image sequence controller (40) is installable within an imaging modality (20); and
wherein the visual image sequence controller (40) is further configured to communicate the visual imaging sequence acquisition (42) of the interventional imaging sequence (23) from the imaging modality (20) to the augmented reality device (30).
14. The augmented image-guided intervention system of claim 8 ,
wherein the visual image sequence controller (40) is installable within an auxiliary intervention device; and
wherein the visual image sequence controller (40) is further configured to communicate the visual imaging sequence acquisition (42) of the interventional imaging sequence (23) from the auxiliary intervention device to the augmented reality device (30).
15. An augmented image-guided intervention method, comprising:
rendering, by an augmented reality device (30), an augmented view of an image-guided intervention; and
controlling, by a visual image sequence controller (40), an augmentation of a visual imaging sequence acquisition of an interventional imaging sequence in a rendered augmented view of the image-guided intervention by the augmented reality device (30) including
accessing, by the visual image sequence controller (40), geometric information (41) of an imaging modality (20) associated with an acquisition of an interventional imaging sequence (23) of interventional images by the imaging modality (20) during the image-guided intervention;
generating, by the visual image sequence controller (40), the visual imaging sequence acquisition (42) of the interventional imaging sequence (23) including, for each interventional image of the interventional imaging sequence (23), an interactive virtual indicator of a sequence number of a corresponding interventional image within the interventional imaging sequence and of an imaging parameter of the imaging modality (20) during an acquisition of the corresponding interventional image by the imaging modality (20);
augmenting, by the visual image sequence controller (40), the view of the image-guided intervention by the augmented reality device (30) with the interactive virtual indicators as virtual objects indicative of the acquisition of the interventional imaging sequence (23) of interventional images by the imaging modality (20) during the image-guided intervention; and
interfacing, by the visual image sequence controller (40), with the at least one of the imaging modality (20) and a display (24, 31) responsive to a user interaction with at least one of the interactive virtual indicators
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP19209345.8A EP3822981A1 (en) | 2019-11-15 | 2019-11-15 | Image acquisition visuals for augmented reality |
EP19209345.8 | 2019-11-15 | ||
PCT/EP2020/081740 WO2021094360A1 (en) | 2019-11-15 | 2020-11-11 | Image acquisition visuals for augmented reality |
Publications (1)
Publication Number | Publication Date |
---|---|
US20220392607A1 true US20220392607A1 (en) | 2022-12-08 |
Family
ID=68583127
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/776,042 Pending US20220392607A1 (en) | 2019-11-15 | 2020-11-11 | Image acquisition visuals for augmented reality |
Country Status (5)
Country | Link |
---|---|
US (1) | US20220392607A1 (en) |
EP (2) | EP3822981A1 (en) |
JP (1) | JP2023502036A (en) |
CN (1) | CN114730628A (en) |
WO (1) | WO2021094360A1 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113778580B (en) * | 2021-07-28 | 2023-12-08 | 赤子城网络技术(北京)有限公司 | Modal user interface display method, electronic device and storage medium |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160191887A1 (en) * | 2014-12-30 | 2016-06-30 | Carlos Quiles Casas | Image-guided surgery with surface reconstruction and augmented reality visualization |
WO2018060304A1 (en) * | 2016-09-30 | 2018-04-05 | Koninklijke Philips N.V. | Anatomical model for position planning and tool guidance of a medical tool |
US20190282311A1 (en) * | 2016-11-11 | 2019-09-19 | Intuitive Surgical Operations, Inc. | Teleoperated surgical system with patient health records based instrument control |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2019040493A1 (en) * | 2017-08-21 | 2019-02-28 | The Trustees Of Columbia University In The City Of New York | Systems and methods for augmented reality guidance |
US20190310819A1 (en) * | 2018-04-10 | 2019-10-10 | Carto Technologies, LLC | Augmented reality image display systems and methods |
-
2019
- 2019-11-15 EP EP19209345.8A patent/EP3822981A1/en not_active Withdrawn
-
2020
- 2020-11-11 CN CN202080079236.2A patent/CN114730628A/en active Pending
- 2020-11-11 WO PCT/EP2020/081740 patent/WO2021094360A1/en unknown
- 2020-11-11 US US17/776,042 patent/US20220392607A1/en active Pending
- 2020-11-11 EP EP20801304.5A patent/EP4059024A1/en active Pending
- 2020-11-11 JP JP2022527737A patent/JP2023502036A/en active Pending
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160191887A1 (en) * | 2014-12-30 | 2016-06-30 | Carlos Quiles Casas | Image-guided surgery with surface reconstruction and augmented reality visualization |
WO2018060304A1 (en) * | 2016-09-30 | 2018-04-05 | Koninklijke Philips N.V. | Anatomical model for position planning and tool guidance of a medical tool |
US20190282311A1 (en) * | 2016-11-11 | 2019-09-19 | Intuitive Surgical Operations, Inc. | Teleoperated surgical system with patient health records based instrument control |
Also Published As
Publication number | Publication date |
---|---|
JP2023502036A (en) | 2023-01-20 |
CN114730628A (en) | 2022-07-08 |
WO2021094360A1 (en) | 2021-05-20 |
EP4059024A1 (en) | 2022-09-21 |
EP3822981A1 (en) | 2021-05-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7774044B2 (en) | System and method for augmented reality navigation in a medical intervention procedure | |
US9681925B2 (en) | Method for augmented reality instrument placement using an image based navigation system | |
US6049622A (en) | Graphic navigational guides for accurate image orientation and navigation | |
JP5417609B2 (en) | Medical diagnostic imaging equipment | |
US11026747B2 (en) | Endoscopic view of invasive procedures in narrow passages | |
US20080074427A1 (en) | Method for display of medical 3d image data on a monitor | |
US10951837B2 (en) | Generating a stereoscopic representation | |
US20220392607A1 (en) | Image acquisition visuals for augmented reality | |
WO2008120136A1 (en) | 2d/3d image registration | |
WO2021150459A1 (en) | Systems and methods for masking a recognized object during an application of a synthetic element to an original image | |
JP6952740B2 (en) | How to assist users, computer program products, data storage media, and imaging systems | |
EP3629932B1 (en) | Device and a corresponding method for providing spatial information of an interventional device in a live 2d x-ray image | |
Vogt et al. | An AR system with intuitive user interface for manipulation and visualization of 3D medical data | |
US11532130B2 (en) | Virtual augmentation of anatomical models | |
Vogt | Real-Time Augmented Reality for Image-Guided Interventions | |
Dey et al. | Mixed reality merging of endoscopic images and 3-D surfaces | |
US11769261B2 (en) | Imaging system | |
US20220218435A1 (en) | Systems and methods for integrating imagery captured by different imaging modalities into composite imagery of a surgical space | |
JP2018061844A (en) | Information processing apparatus, information processing method, and program | |
US20150121276A1 (en) | Method of displaying multi medical image and medical image equipment for performing the same | |
US20160205390A1 (en) | Method for displaying on a screen an object shown in a 3d data set | |
EP3931799B1 (en) | Interventional device tracking | |
De Paolis | Advanced navigation and augmented visualization in minimally invasive surgery | |
JP2023004884A (en) | Rendering device for displaying graphical representation of augmented reality | |
DE202020004674U1 (en) | Device of a mirror with augmented reality |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: KONINKLIJKE PHILIPS N.V., NETHERLANDS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:DEN HARTOG, MARKUS JOHANNES HARMEN;OLIVAN BESCOS, JAVIER;REEL/FRAME:059892/0329 Effective date: 20201111 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |