WO2020041693A1 - Boîte à outils virtuelle pour radiologues - Google Patents

Boîte à outils virtuelle pour radiologues Download PDF

Info

Publication number
WO2020041693A1
WO2020041693A1 PCT/US2019/047891 US2019047891W WO2020041693A1 WO 2020041693 A1 WO2020041693 A1 WO 2020041693A1 US 2019047891 W US2019047891 W US 2019047891W WO 2020041693 A1 WO2020041693 A1 WO 2020041693A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual
volume
dimensional image
sub
image volume
Prior art date
Application number
PCT/US2019/047891
Other languages
English (en)
Inventor
David Byron Douglas
Robert Edwin Douglas
Kathleen Mary DOUGLAS
Original Assignee
David Byron Douglas
Robert Edwin Douglas
Douglas Kathleen Mary
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by David Byron Douglas, Robert Edwin Douglas, Douglas Kathleen Mary filed Critical David Byron Douglas
Priority to JP2021510084A priority Critical patent/JP2021533940A/ja
Priority to CN201980062928.3A priority patent/CN113424130A/zh
Priority to CA3109234A priority patent/CA3109234A1/fr
Priority to AU2019325414A priority patent/AU2019325414A1/en
Priority to EP19852803.6A priority patent/EP3841450A1/fr
Publication of WO2020041693A1 publication Critical patent/WO2020041693A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/50—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for simulation or modelling of medical disorders

Definitions

  • CT computed tomography
  • MRI magnetic resonance imaging
  • Recent advances in diagnostic radiology have provided true 3D viewing of medical images through the use of virtual reality, mixed reality or augmented reality headsets and have enabled a volume -by-volume approach and interactive volume subtending 3D cursor usage (See US Patent 8,384,771 for METHOD AND APPARATUS FOR THREE DIMENSIONAL VIEWING OF IMAGES which is incorporated by reference; US Patent 9,980,691 for METHOD AND APPARATUS FOR THREE DIMENSIONAL VIEWING OF IMAGES which is incorporated by reference; and Douglas, D. B., Wilke, C.
  • a method comprises: for a selected three-dimensional medical image volume loaded in an image processing system, selecting a set of virtual tools from a group of available virtual tools responsive to user input; geo-registering each virtual tool of the selected set with the three-dimensional image volume; and manipulating the three-dimensional image volume in response to manipulation of ones of the set of virtual tools.
  • Some implementations comprise selecting the set of virtual tools from a group of available virtual tools comprising: a virtual focal point pen; a virtual 3D cursor; a virtual transport viewer; a virtual pedestal; a virtual knife; a virtual catheter; a virtual road sign; a virtual ablation tool; a virtual table; a virtual contrast tool; and virtual icons.
  • Some implementations comprise manipulating the three-dimensional image volume responsive to a virtual focal point pen by highlighting a portion of the three-dimensional image volume and adding written notes. Some implementations comprise altering three-dimensional image volume voxels adjacent to a tip of the virtual focal point pen. Some implementations comprise manipulating the three-dimensional image volume responsive to a virtual knife by separating tissue from the three-dimensional image volume. Some implementations comprise manipulating the three-dimensional image volume responsive to the virtual transport viewer by moving the virtual transport viewer within a hollow structure of the three-dimensional image volume and presenting an image from a perspective of the virtual transport viewer. Some implementations comprise using the virtual transport viewer to perform a virtual colonoscopy.
  • Some implementations comprise manipulating the three-dimensional image volume responsive to the virtual contrast material by inserting visible, mobile voxels into the three-dimensional image volume. Some implementations comprise assigning different density values to different ones of the mobile voxels. Some implementations comprise manipulating the three-dimensional image volume in response to manipulation of ones of the set of virtual tools by removing voxels of an outer shell of an organ, which can be performed in a repeated shell-by-shell fashion. Some implementations comprise manipulating the three- dimensional image volume in response to manipulation of ones of the set of virtual tools by spreading apart closely spaced tissue of interest by adjusting coordinates of voxels of the tissue of interest.
  • Some implementations comprise comprising manipulating the three-dimensional image volume responsive to the virtual table by placing tissue of interest in a virtual storage bin of the virtual table. Some implementations comprise manipulating the three-dimensional image volume responsive to the virtual catheter by restricting movement of the virtual catheter to a column of blood voxels within a selected blood vessel. Some implementations comprise automatically displaying information associated with a selected sub-volume of the three-dimensional image volume. Some implementations comprise displaying patient’s meta data and current condition for which obtaining the medical image volume was prompted, patient’s medical history, laboratory results, and pathology results. Some implementations comprise displaying the information with a virtual windshield. Some implementations comprise displaying distances to key metrics with a virtual road sign. Some implementations comprise displaying a visual aid icon indicating viewing perspective.
  • Some implementations comprise displaying a visual aid icon indicating a finding detected by an artificial intelligence algorithm. Some implementations comprise displaying a visual aid icon indicating location of a sub-volume being displayed relative to the three- dimensional medical image volume or patient’s body. Some implementations comprise selecting a sub volume with a volume -subtending three-dimensional cursor. Some implementations comprise selecting the sub-volume from a plurality of sub-volumes of a predetermined list of sub-volumes. Some implementations comprise sequentially displaying each of the sub-volumes of the list. Some implementations comprise selecting the sub-volume from a plurality of sub-volumes defined by sequential search pattern coordinates. Some implementations comprise selecting the sub-volume from a plurality of sub-volumes defined by random search pattern coordinates.
  • manipulating the three-dimensional image volume comprises at least one of: changing voxel size; changing voxel shape; changing voxel position; changing voxel orientation; changing voxel internal parameter; creating a voxel; and eliminating a voxel.
  • manipulating the three- dimensional image volume comprises dividing a sub-volume volume of interest into multiple parts based on common characteristics.
  • manipulating the three-dimensional image volume comprises generating an exploded view by creating multiple enlarged cubes each touching a center point.
  • Some implementations comprise employing a virtual eye tracker symbol to assist in human eye viewing.
  • Some implementations comprise making the virtual eye tracker symbol appear and disappear at spatially separate lactations so that the human eye can perform saccades and jump from one location to another. Some implementations comprise making the virtual eye tracker symbol move smoothly along a path, so that the human eye can perform smooth tracking.
  • an apparatus comprises: an image processing system comprising an interface that presents, for a selected three-dimensional medical image volume loaded in the image processing system, a set of virtual tools for selection from a group of available virtual tools responsive to user input, geo-registers each virtual tool of the selected set with the three-dimensional image volume, and an image processor that manipulates the three-dimensional image volume in response to manipulation of ones of the set of virtual tools.
  • the set of virtual tools is selected from a group of available virtual tools comprising: a virtual focal point pen; a virtual 3D cursor; a virtual transport viewer; a virtual pedestal; a virtual knife; a virtual catheter; a virtual road sign; a virtual ablation tool; a virtual table; a virtual contrast tool; and virtual icons.
  • the set of virtual tools comprises a virtual focal point pen and the image processor manipulates the three-dimensional image volume responsive to the virtual focal point pen by highlighting a portion of the three-dimensional image volume and adding written notes.
  • the image processor alters three- dimensional image volume voxels adjacent to a tip of the virtual focal point pen.
  • the set of virtual tools comprises a virtual knife and the image processor manipulates the three-dimensional image volume responsive to the virtual knife by separating or manipulating (e.g., changing the location of) voxels (e.g., tissue-type voxels) from the three-dimensional image volume.
  • the set of virtual tools comprises a virtual transport viewer and wherein the image processor manipulates the three-dimensional image volume responsive to the virtual transport viewer by moving the virtual transport viewer within a hollow structure of the three-dimensional image volume and presenting an image from a perspective of the virtual transport viewer.
  • the virtual transport viewer is used to perform a virtual colonoscopy via the interface.
  • the set of virtual tools comprises a virtual contrast material and the wherein the image processor manipulates the three-dimensional image volume responsive to the virtual contrast material by inserting visible, mobile voxels into the three-dimensional image volume.
  • the image processor assigns different density values to different ones of the mobile voxels.
  • the image processor manipulates the three-dimensional image volume in response to manipulation of ones of the set of virtual tools by removing voxels of an outer shell of an organ.
  • the image processor manipulates the three-dimensional image volume in response to manipulation of ones of the set of virtual tools by spreading apart closely spaced tissue of interest by adjusting coordinates of voxels of the tissue of interest.
  • the set of virtual tools comprises a virtual table and the image processor manipulating the three-dimensional image volume responsive to the virtual table by placing tissue of interest in a virtual storage bin of the virtual table.
  • the set of virtual tools comprises a virtual catheter and the image processor manipulates the three-dimensional image volume responsive to the virtual catheter by restricting movement of the virtual catheter to a column of blood voxels within a selected blood vessel.
  • the interface automatically displays information associated with a selected sub-volume of the three-dimensional image volume.
  • the interface displays patient’s meta data and current condition for which obtaining the medical image volume was prompted, patient’s medical history, laboratory results, and pathology results.
  • Some implementations comprise the interface displaying the information with a virtual windshield.
  • Some implementations comprise the interface displaying distances to key metrics with a virtual road sign. Some implementations comprise the interface displaying a visual aid icon indicating viewing perspective. Some implementations comprise the interface displaying a visual aid icon indicating a finding detected by an artificial intelligence algorithm. Some implementations comprise the interface displaying a visual aid icon indicating location of a sub- volume being displayed relative to the three-dimensional medical image volume or patient’s body. Some implementations comprise the interface receiving selection of a sub-volume with a volume -subtending three-dimensional cursor. In some implementations the selected sub-volume is one of a plurality of sub volumes of a predetermined list of sub-volumes. Some implementations comprise the interface sequentially displaying each of the sub-volumes of the list.
  • the selected sub volume is one of a plurality of sub-volumes defined by sequential search pattern coordinates. In some implementations the selected sub-volume is one of a plurality of sub-volumes defined by random search pattern coordinates.
  • the image processor manipulates the three-dimensional image volume comprises at least one of: changing voxel size; changing voxel shape; changing voxel position; changing voxel orientation; changing voxel internal parameter; creating a voxel; and eliminating a voxel. In some implementations the image processor manipulates the three-dimensional image volume by dividing a sub-volume volume of interest into multiple parts based on common characteristics.
  • the image processor manipulates the three-dimensional image volume by generating an exploded view by creating multiple enlarged cubes each touching a center point.
  • the interface comprises a virtual eye tracker symbol.
  • the virtual eye tracker symbol appears and disappears at spatially separate lactations so that the human eye can perform saccades and jump from one location to another.
  • the virtual eye tracker symbol moves smoothly along a path.
  • a method and apparatus for preparing software to implement the virtual tool kit for enhanced medical image analysis comprises the following steps: loading volumetric medical imaging dataset in accordance with a checklist, converting the medical images to a 3D volume and importing a virtual tool kit; performing registration and calibration of each virtual tool with/within the geo-registered volumetric medical image; performing filtering, segmentation and voxel manipulations (e.g. as described in US Patent application 15/904,092 for PROCESSING 3D
  • An apparatus comprising: an IO device; an image processor in communication with the IO device, the image processors comprising a program stored on computer-readable non-transitory media, the program comprising: instruction to perform of segmentation of the voxels into distinct tissue types; instruction to perform voxel manipulations; instructions to creation and insertion of at least one additional voxel into the medical imaging dataset; instructions to eliminate voxel into the medical imaging datasets; instructions to perform voxel annotations to the above user-altered radiologic image; instructions to recording steps for review.
  • virtual tool maneuvers can also be accomplished via the controller/joystick input.
  • the controller / joystick input can direct the 3D cursor to change size, shape, or orientation (roll, pitch and yaw).
  • controller/joystick input can direct the left and the right eye viewing perspectives to be zoomed inward towards the 3D cursor or out away from the cursor.
  • controller/joystick input can direct convergence to a focal point.
  • controller/joystick input can direct raising or lowering of the 3D cursor within the head display unit, or moving the 3D cursor from side to side.
  • controller/joystick input can change the color of the 3D cursor.
  • controller / joystick input can invoke filtering, segmentation, sequencing, statistical analysis and reporting, which were discussed in US Patent Application 15/904,092, which is incorporated by reference.
  • controller/joystick input can direct movement of a virtual focal point pen through the volume of interest.
  • controller/joystick/keyboard input can direct annotations of one or more 3D cursors within the volume of interest.
  • controller / joystick input can direct icon options as related to the volumetric medical imaging to keep the radiologist organized in his/her checklist approach to a complex exam.
  • presenting 3D medical images comprises a method of improved user controller interface for medical personnel reviewing 3D medical images comprised of joystick and functional buttons.
  • Functionality would include, but is not limited to, the following when interfacing with the 3D cursor: change the orientation of the 3D cursor - roll, pitch and yaw; zoom the medical person viewpoint in toward the 3D cursor and out away from the cursor; invoke convergence; raise and lower the 3D cursor as to where it is displayed on the headset; change the size, shape, and color of the 3D cursor; invoke filtering, segmentation, sequencing, statistical, and reporting operations; invoke virtual focal point pen and movement control thereof; annotate one or more 3D cursors within the volume of interest; invoke icon options; and, invoke advanced viewing options of interest (e.g., explosion, ablation, slice-type viewing).
  • change the orientation of the 3D cursor - roll, pitch and yaw zoom the medical person viewpoint in toward the 3D cursor and out away from the cursor
  • invoke convergence raise and lower the 3D cursor as to where it is displayed on the headset
  • change the size, shape, and color of the 3D cursor invoke filtering, segmentation, sequencing
  • a representative sample of virtual tools includes, but is not limited to the following: virtual focal point pen; virtual 3D cursor; virtual transport viewer; virtual pedestal; virtual knife; virtual catheter; virtual road signs; virtual ablation; virtual table; virtual contrast; and, virtual icons.
  • virtual tools are possible, but are not illustrated, which include, but are not limited to, the following: drill; cup; string; mirror; lens; metallic devices; non-metallic devices; and other tools, commonly used by medical personnel, construction workers or engineers.
  • the medical person conducting the review of volumetric medical images could invoke a process whereby a virtual radiology assistant type icon is displayed.
  • the purpose of the virtual radiology assistant type icon is to portray all relevant/ important information relating to inter alia: where the medical person conducting the examination is on the medical institution’s checklist and what’s next; patient’s meta data and current condition triggering obtaining medical image(s); patient’s medical history; laboratory results; results, if any, from application of artificial intelligence (AI) routines and indicators of condition.
  • AI artificial intelligence
  • the medical person conducting the review could command display of the virtual windshield at any time.
  • the medical person conducting the review could also modify items to be shown on the windshield.
  • virtual tools can direct voxel alterations.
  • voxel manipulations include changing the size, shape, position, orientation or internal parameter of a voxel.
  • voxels can be created or eliminated at the direction of the virtual tool.
  • a virtual focal point pen is utilized to enhance visualization of a structure of interest within the medical imaging volume.
  • the focal point pen can be used to point to areas within the structure, which may contain anomalies.
  • the focal point pen can use symbols (e.g., arrows) to point to the areas of interest.
  • notes can be written in the volumetric data.
  • voxels in proximity of the point of the focal point pen can be highlighted in conjunction with modifying the transparency of tissues at a specified distance away from the tip of the focal point pen.
  • the focal point pen can be used in conjunction with the 3D cursor.
  • a virtual eye tracker symbol is employed with the 3D medical imaging dataset to facilitate the human in viewing structures.
  • the virtual eye tracker symbol appears and disappears at spatially separate lactations so that the human eye can perform saccades and jump from one location to another.
  • the virtual eye tracker symbol is continuously visible and has a smooth movement along a path, so that the human eye can perform smooth tracking.
  • the virtual eye tracker symbol can be controlled by, but not limited to, the following: virtual tool (e.g. virtual focal point pen); geo-registered tool; or, a pre-programmed sequence.
  • a virtual knife can be co-registered with the volumetric medical images and used to cut virtual tissue in a medical imaging volume.
  • the virtual knife would have a moveable geo-registration point (e.g., tip of the virtual knife), additional points to indicate the cutting surface of the virtual knife and the control unit would provide changes in the X, Y, Z coordinate system and also roll, pitch and yaw of the knife.
  • the virtual knife could be used, but not limited to, as follows: the medical person viewing the medical images could: pick up the virtual knife and move it to the 3D digital structure of current interest, then pass the virtual knife through the 3D geo-registered structure, and tissue may be deleted (or set aside) which is external to the surface created by virtual knife when it passed through the 3D geo-registered structure (the side of the virtual knife pre-selected by the medical person viewing the medical images).
  • the virtual knife would have an exact registration point (e.g., tip of geo-registered knife), additional geo-registration points to indicate the cutting surface of the knife.
  • tactile or auditory feedback can be provided to the user.
  • presenting 3D medical images comprises a method to facilitate viewing medical images consisting of using a visual transport tool to achieve a ride through a blood vessel.
  • the virtual catheter could be used in conjunction with a visual transport tool to optimize viewing of the vascular structure within the patient.
  • Example procedures include but, are not limited to the following: the ride in a tunnel could be used during an assessment of the vascular structure and potential need for insertion of one or more stents.
  • the ride in a blood vessel entry could be through the groin into the common femoral artery, external iliac artery, common iliac artery, abdominal aorta, thoracic aorta, aortic arch, and finally the coronary artery of interest. Then, the medical person viewing the medical images visualize what it looks like to travel within a blood vessel as viewed from a 3D headset. Blood within the blood vessel could be digitally subtracted and a virtual light could shine on the blood vessel walls. Constrictors within the vascular structure would show up as a narrowing of the tunnel and X, Y, Z coordinates of the constrictor be recorded.
  • presenting 3D medical images comprises a method to facilitate viewing medical images consisting of a 3D geo-registered ride in the blood vessel.
  • this 3D geo-registered ride in the tunnel would be used in conjunction geo-registration of patients as described in US Patent Application 15/949,202 for SMART OPERATING ROOM EQUIPPED WITH SMART SURGICAL DEVICES and US
  • the interventionist could switch back and forth between the geo-registered 3D system using the 3D head mounted display and standard displays currently available in interventional operations. This permits viewing in near real time the constrictors identified during pre-operative planning. Further, alerts could be given in near real time as critical junctions were being approached.
  • presenting volumetric medical images comprises a method to facilitate viewing medical images consisting of a virtual catheter.
  • the virtual catheter could be used in conjunction with a 3D digital image of the vascular structure within the patient.
  • the catheter could continuously compute the total distance travelled which could be displayed, and also time tagged and recorded for later review.
  • the virtual catheter could be used during pre-operative planning of an interventional procedure such as, but not limited to, treatment of a cerebral aneurysm.
  • Implementation of the virtual catheter in treatment of an aneurysm could be as follows: insert the virtual catheter into the 3D digital vascular structure at a pre-determined point such as the groin of the patient into the common femoral artery, then the external iliac artery, then the common iliac artery, then the abdominal aorta, then the thoracic aorta, then the brachiocephalic artery, then the common carotid artery, then the internal carotid artery then the middle cerebral artery, and finally into the aneurysm.
  • a pre-determined point such as the groin of the patient into the common femoral artery, then the external iliac artery, then the common iliac artery, then the abdominal aorta, then the thoracic aorta, then the brachiocephalic artery, then the common carotid artery, then the internal carotid artery then the middle cerebral artery, and finally into the aneury
  • Augmented reality distance markers could be added to the 3D virtual catheter for each intersection the interventionalist would need to take care and be prepared to change from one blood vessel to another; screen captures of all key vascular junctures could be annotated angular changes from current path in coordinate system X-Y, X-Z and Y-Z planes.
  • presenting 3D medical images comprises a method to facilitate viewing medical images consisting of an explosion within the 3D digital imagery.
  • the medical person viewing the medical images e.g., using segmentation techniques outlined in US Patent Application 15/904,092
  • One of the ways, but not limited to, is as follows: create eight large cubes each touching the center point and each parallel to the X, Y, Z axes (e.g., the first cube would be positive in X, positive in Y and positive in Z; the second cube could be positive in X, negative in Y and positive in Z; and so on). Then the medical person viewing the medical images establishes a distance factor for sub-volumes close to the center point, a larger distance factor for those further away. Then these factors are applied to all voxels within each specific sub volume of the 3D digital image based on which cube the center voxel of the sub-volume was in.
  • the virtual transport process viewer could be used to precede a colonoscopy.
  • the patient would: first receive a CT scan of the colon; a 3D volume of the colon would be created from the CT 2D slices (US Patent 8,384,771); segmentation (US Patent Application 15/904,092) would identify and subtraction would extract the contents of the colon (e.g., air, excrement) and in so doing the colon would maintain its original shape and any polyps would be visible; then the virtual platform could be inserted and enable examination back and forth and side to side within the colon.
  • the diameter of the colon could be increased thru voxel manipulation (US Patent Application 16/195,251) to an optimal diameter from the center point of the enlarged colon. If no polyps were found, the patient could go home confident of continued good health and he/ she would have avoided the unpleasantness and discomfort of a colonoscopy and the cost of the colonoscopy avoided. Otherwise, treatment would be known to be required. Personnel who would have avoided the colonoscopy would likely not shy away from the virtual transport viewer process and the fraction of cases wherein detection occurred at an early stage would increase. Thereby, the general health of the populace would be improved.
  • the medical person conducting the review of volumetric medical images of the colon could invoke a process whereby a virtual colonoscopy is performed.
  • a CT scan of the colon with/ without contrast is performed.
  • a 3D virtual image is constructed from the CT 2D slices (US Patent 8,384,771). Segmentation (US Patent Application 15/904,092) is performed and tissue subtracted external to the colon. Also, the non-tissue contents within the colon are subtracted.
  • the colon is‘stretched’ so that folds which can obscure polyps are elongated and, thereby, obscuration of polyps by folded colon tissue is eliminated.
  • This stretching process involves voxel manipulation as described in US Patent Application 16/195,251.
  • This elongated, straight virtual colon is split in 2 along the length axis so that the internal structure can be viewed via the head display unit as illustrated in this figure.
  • presenting 3D medical images comprises a method to facilitate viewing medical images consisting of insertion of virtual contrast material within the vascular system.
  • the virtual contrast could be used in conjunction with a 3D digital image of the vascular structure within the patient.
  • Example procedures include, but are not limited to, the following: search for blockage as in a pulmonary embolism.
  • blood vessels would be selected to receive the virtual contrast material in a time step fashion. Insertion of the virtual contrast could be in a manner and velocity as if it were actually being inserted into the blood vessel. The duration of the time step would be under the control of medical person viewing the medical images; ‘freeze frame’ would be available together with replay of the contrast flow.
  • Nearby and possibly overlapping blood vessels other than the one receiving the virtual contrast could be extracted from the 3D imagery displayed to medical person viewing the medical images.
  • presenting 3D medical images comprises a method to facilitate viewing medical images consisting of an ablation technique.
  • the ablation technique could be used in conjunction with 3D digital structure, transported by the 3D cursor as described above.
  • the method underlying the ablation technique consists of, but not limited to, the following procedures: first determine the outer‘shell’ of organ of interest to the medical person viewing the medical images (e.g., using segmentation techniques outlined in US Patent Application 15/904,092); sequentially eliminate one voxel deep from all of the voxels on the outer surface - repeat this step multiple times on the remaining outer layer of tissue at the direction of the medical person viewing the medical images; alternatively, select one layer in the X, Y, Z coordinate system (e.g., select the X-Y layer with the highest Z coordinate and eliminate that layer - repeat this step multiple times on the remaining 3D digital volume at the direction of the medical person viewing the medical images. This process can be repeated multiple times and each time the outer shell is removed, leaving
  • the virtual tool can be utilized to alter the location of tissue of interest that is closely spaced such that the separation distance of these tissues can be increased through voxel manipulation (e.g., insertion of additional voxels of varying transparency) and concurrently adjusting the coordinates of the tissues of interest.
  • voxel manipulation e.g., insertion of additional voxels of varying transparency
  • Sub-volumes can be made from varying number or combination of voxels.
  • presenting 3D medical images comprises a method to sequence the movements of the 3D cursor through the volume of interest.
  • the sequence could, for example but not limited to, start the 3D cursor located at the 0_X, 0_Y, 0_Z coordinate, increment X to move the 3D cursor in the X direction and continue to increment to the maximum value of X, then increment Y and decrement X back to the OX coordinate. This would continue through the X - Y plane until it was completed, then Z would be incremented, and the process continued until completion. The increments would continue until the entire volume of interest had been reviewed. Changing from one increment to another would be controlled by the medical personnel reviewing the medical images.
  • the medical personnel could annotate the 3D cursor for further review. Further, at any time the medical personnel could choose to display the location of where the 3D cursor is within the volume of interest and the volume of interest that had already been examined. In the event suspicious tissue appeared in more than one 3D cursor, the totality of these 3D cursors could be displayed simultaneously.
  • the medical person reviewing the 3D virtual medical image may invoke a type of search pattern.
  • a sequential search pattern may be selected.
  • An example of sequential search is, but not limited to, a virtual windshield wiper search. This type of search pattern helps ensure a thorough search has been conducted.
  • Image processing aids such as changing organ transparency and using false color can help identify items of potential interest (US Patent 8,384,771).
  • This type of search pattern can expedite the review process.
  • Both types of search patterns employ a 3D cursor (US Patent 9,980,691 for METHOD AND APPARATUS FOR THREE DIMENSIONAL VIEWING OF IMAGES and US Patent Application 15/878,463 for INTERACTIVE 3D CURSOR FOR USE IN MEDICAL IMAGING).
  • the eyes jump from one spot to another following the reviewing individual’s saccadian path and substantial portions of a slice may not be observed and, consequently, small masses may be missed.
  • these small masses subtend a larger fraction of presented image and the probability of detection increases proportionally.
  • the medical person conducting the review of volumetric medical images could invoke a process whereby a volumetric medical image is examined using a step-by-step process of selection of sub-volumes (of the total volume) that are encased within the 3D cursor (US Patent 9,980,691 and US Patent Application 15/878,463).
  • a volumetric medical image is examined using a step-by-step process of selection of sub-volumes (of the total volume) that are encased within the 3D cursor (US Patent 9,980,691 and US Patent Application 15/878,463).
  • Each 3D cursor’s contents would be independently reviewed.
  • the question could arise as to whether the entire volume has been examined.
  • the volume contained in each of the 3D cursors which had been examined could be totaled and subtracted from the total original volume. This could result is some portions of the original volume being missed.
  • presenting 3D medical images comprises a method to sequence the movements of the 3D cursor through the volume of interest.
  • the sequence could, for example but not limited to, start the 3D cursor located at the 0_X, 0_Y, 0_Z coordinate, increment X to move the 3D cursor in the X direction and continue to increment to the maximum value of X, then increment Y and decrement X back to the OX coordinate.
  • virtual icons can be used in conjunction with viewing of the medical imaging volume to facilitate orientation.
  • an organ e.g. liver
  • the icon can assist with orientation at all times during the viewing of the examination.
  • an arrow annotation marker can show the path from the initial viewing perspective to the current viewing perspective.
  • an auto-recenter technique can be used to quickly reorient the user.
  • a virtual table could be added to the tool kit which would have virtual storage bins on the table.
  • Sub-volume(s) of the virtual medical image which is currently being examined and which contain tissue of concern/ interest could be placed in the virtual storage bin(s).
  • the bins for the sub-volumes could correspond to checklist items of the medical institution.
  • an emergency bin could be added which could be accessed by both reviewing medical person and treatment personnel, thereby facilitating and expediting collaboration between these persons.
  • the report preparation could be expedited by automatically sequencing through the bins and, extracting items, and adding these items to the report. The added items (e.g., an annotated figure containing the tissue in question) would add to both the quality and completeness of the report.
  • a radiologist finds something anomalous, he/she puts it on the report virtual table.
  • the item placed on the report virtual table may include the 2D slice or the 3D volume containing the anomalous finding.
  • the radiologist has options to determine the size of the virtual table and virtual bins. One radiologist can pass an item onto another radiologist’s table or bin for collaboration.
  • a radiology report may include images processed via virtual tools.
  • Figure 1 illustrates a flow diagram for the use of virtual tools to optimize display of medical imaging examinations.
  • Figure 2 illustrates an apparatus for implementing the process of figure 1.
  • Figure 3 illustrates virtual toolkit options in viewing volumetric medical images.
  • FIG 4 illustrates a flow diagram and an example heads-up display type icon delivering pertinent information to the radiologist during the interpretation of the imaging examination.
  • the heads-up display we refer to as a virtual windshield (analogous to a‘heads-up’ display in an aircraft) which could be called at any time during the examination process and would display on a single virtual windshield items relevant to the patient’s entire case and/ or the particular item being examined on the medical institution’s checklist. Alternatively, it can be referred to as a virtual radiology assistant type icon.
  • Figure 5 illustrates flow diagram and illustration of virtual tool kit inputs leading to voxel alterations.
  • Figure 6 illustrates voxel manipulation based on interaction with a virtual tool.
  • Figure 7 illustrates virtual eye tracker symbol with variable display patterns.
  • Figure 8 illustrates the virtual knife, which can be used by medical personnel to‘carve away tissue’ from an existing 3D medical imaging volume to permit enhanced viewing of the interior structure.
  • Figure 9 illustrates a virtual ride thru virtual blood vessel using a visual transport tool.
  • Figure 10 illustrates the virtual catheter could be used in conjunction with a volumetric medical image of the vascular structure within the patient with the assistance of virtual icons.
  • Figure 11 illustrates the general concept of 3D medical images and example techniques behind the explosion of 3D medical images into multi separate organs.
  • Figure 12 illustrates use of virtual transport viewer to perform a more accurate virtual colonography review.
  • Figure 13 illustrates a portion of a virtual 3D volumetric medical image which contains the colon portion of the large intestine which has, through voxel manipulation, been stretched in such a manner that is one long straight tube. And then, the contents within the tube have segmented and subsequently/ eliminated from the tube. And, finally, the tube is split along the length axis and opened to permit viewing of the internal colon structure.
  • Figure 14 illustrates insertion of virtual contrast and its flow through the vascular system.
  • Figure 15 illustrates an ablation technique could be used, inter alia, in conjunction with 3D digital structure within a 3D cursor. This permits careful examination of the interior structure and elements of an organ.
  • Figure 16 illustrates virtual focal point pen guiding voxel manipulation.
  • Figure 17 illustrates the total imaging volume and multiple types of sub-volumes.
  • Figure 18 illustrates sequencing of movement of the 3D cursor through the volume of interest in a random pattern.
  • Figure 19 illustrates an example of a systematic pattern of viewing of medical images (e.g., sequential virtual windshield wiper type pattern).
  • Figure 20 illustrates volume of interest to be reviewed and a process whereby any areas which intended for review which were missed could be highlighted to the medical person performing the review. These identified sub-volumes could be reviewed subsequently, thereby ensuring completeness of the review.
  • Figure 21 illustrates an icon of a human with the location of the 3D virtual cursor included at the approximate location within the body.
  • Figure 22 illustrates a virtual moveable table for storing virtual images of suspect tissue stored by checklist category.
  • Figure 23 illustrates a sample radiology report including an image processed with virtual tools.
  • Some aspects, features and implementations described herein may include machines such as computers, electronic components, radiological components, optical components, and processes such as computer-implemented steps. It will be apparent to those of ordinary skill in the art that the computer- implemented steps may be stored as computer-executable instructions on a non-transitory computer- readable medium. Furthermore, it will be understood by those of ordinary skill in the art that the computer-executable instructions may be executed on a variety of tangible processor devices. For ease of exposition, not every step, device or component that may be part of a computer or data storage system is described herein. Those of ordinary skill in the art will recognize such steps, devices and components in view of the teachings of the present disclosure and the knowledge generally available to those of ordinary skill in the art. The corresponding machines and processes are therefore enabled and within the scope of the disclosure.
  • FIG. 1 illustrates a flow diagram for the use of geo-registered tools to optimize display of medical imaging examinations.
  • Step A 100 load volumetric medical imaging dataset in accordance with checklist, convert the medical images to a 3D volume and import virtual tool kit.
  • Step B 102 perform registration and calibration of each virtual tool with/within the geo-registered volumetric medical image.
  • step C 104 perform filtering, segmentation and voxel manipulations in accordance with US Application 15/904,092 and US Patent Application 16/195,251, which are incorporated by reference, and advanced viewing options discussed in this disclosure.
  • Step D 106 for every time step, provide displays in accordance with movement and operation of the virtual tool kit listed in steps above.
  • step E 108 the user must answer the question of“is the examination of this element of the checklist complete?” If the answer is no 110, then the user should proceed to step D 106. If the answer is yes 114, then the radiologist should proceed to step F 116 and review the next set of medical images in accordance with the checklist. Then the radiologist should proceed to step G 118 and answer the question of“is the review complete?” If the answer is no 120, then the radiologist should go 122 to step A 100. If the answer is yes 124, then the radiologist should stop 126.
  • FIG. 2 illustrates an apparatus for implementing the process illustrated in figure 1.
  • a radiologic imaging system 200 e.g., X-ray, ultrasound, CT (computed Tomography), PET (Positron Emission Tomography), or MRI (Magnetic Resonance Imaging)
  • the 2D medical images 202 are provided to an image processor 206, that includes processors 208 (e.g., CPUs and GPUs), volatile memory 210 (e.g., RAM), and non-volatile storage 212 (e.g. HDDs and SSDs).
  • a program 214 running on the image processor implements one or more of the steps described in figure 1.
  • 3D medical images are generated from the 2D medical images and displayed on an IO device 216.
  • the IO device 216 may include a virtual reality headset, mixed reality headset, augmented reality headset, monitor, tablet computer, PDA (personal digital assistant), mobile phone, or any of a wide variety of devices, either alone or in combination.
  • the 10 device 216 may include a touchscreen, and may accept input from external devices (represented by 218) such as a keyboard, mouse, and any of a wide variety of equipment for receiving various inputs. However, some or all the inputs could be automated, e.g. by the program 214.
  • a series of virtual tools 220 are implemented, which facilitate viewing of medical images by medical personnel.
  • FIG. 3 illustrates virtual toolkit options in viewing volumetric medical images.
  • representative examples of viewing options available through the use of a virtual tools are illustrated. Options by which the virtual tools can be guided/selected could be presented on the display and the user would click on desired option.
  • the virtual tool 300 i.e., virtual focal point pen
  • the virtual focal point pen is superimposed within the region containing the virtual 3D medical image 302 located inside of the 3D cursor 304.
  • a button (e.g., on the keyboard) plus a movement of a virtual tool can be coupled together to size the 3D cursor 304 (e.g., select the center of the 3D cursor 304 and then move the virtual focal point pen 300 a distance away to correspond to the radius).
  • the user views the virtual tool using a headset 306 (e.g., augmented reality, mixed reality or virtual reality) glasses with a left eye display 308 and right eye display 310.
  • the virtual focal point pen can be registered within the virtual image by touching specific spots (e.g., comers) of the medical imaging volume 302.
  • the medical personnel can select to only show the tip of the focal point pen in the display, enlarge the tip of the focal point pen as desired, and/or show the virtual image of the focal point pen in its entirety as it is oriented within the volume. Movement of the virtual focal point pen 300 would be controlled by medical person viewing the medical images.
  • the virtual focal point pen 300 is useful when smooth pursuit eye movements are necessary. For example, smooth pursuit eye movements are necessary when examining arteries for any blockages, wherein using the virtual focal point pen to trace along arteries looking for blockages. Saccadian eye movement could result in skipping over portions of the artery and a serious blockage go undetected; therefore, the virtual focal point pen 300 could be helpful in aiding this search pattern.
  • Multiple colored/ shaped virtual focal point pens 300 could be used to trace the different flows of arteries and veins.
  • the virtual focal point pen is shown with an initial position and orientation 312 with respect to the volume of interest 314. Then, the user can move the virtual focal point pen to a subsequent position and orientation 316 with respect to the volume of interest 314. Proceeding clockwise, next the virtual focal point pen 318 performs grabbing of the volume of interest 320 at an initial distance from the head display unit 322. Then, the virtual vocal point pen 324 pulls the volume of interest 326 closer to the head display unit 322 for improved visualization.
  • a virtual dot can be placed on or next to a portion of the virtual image 330 (e.g., carotid artery) being examined in a fixed or dynamic manner.
  • the dot can appear and disappear at multiple spots along the vascular structure to facilitate Saccadian viewing where the eyes jump short distances to view the most important portions of the vascular structure.
  • a first virtual dot 332 appears and no other virtual dots are shown in the field of view at this time.
  • a second virtual dot 334 appears and no other virtual dots are shown in the field of view at this time.
  • a third virtual dot 336 appears and no other virtual dots are shown in the field of view at this time.
  • a fourth virtual dot 338 appears and no other virtual dots are shown in the field of view at this time.
  • the virtual dot 342 can be moving along a portion of the virtual image 340 (e.g., carotid artery) to help the human eye perform smooth tracking and enhanced viewing of the vascular structure.
  • the virtual focal point pen 344 is used to perform convergence to a focal point 346.
  • a left eye view point 348 is shown.
  • a line illustrating the look angle of the left eye 350 is also shown.
  • a right eye view point 352 is shown.
  • a line illustrating the look angle of the right eye 354 is also shown.
  • a virtual dissection is performed by using a virtual knife 356 and the aorta 358 and pulmonary artery 360 are cut and moved away from the rest of the heart 364. Note the cutting plane 362 is shown.
  • a virtual catheter 366 is being placed through the aorta 368 within the medical imaging volume. Note that a virtual road sign 370 is shown to guide the medical personnel. The focal point pen 372 is shown. The dotted line blue line 374 is the desired catheter trajectory, which can be at different time setting. The virtual catheter 366 can be pulled through the vascular system.
  • a ride through the blood vessel type viewing is shown 376 with the desired path highlighted in a dotted red circle 378.
  • the last three examples illustrate advanced viewing options enabled by the virtual tools.
  • An explosion-type viewing where the organs are separated is illustrated wherein the various organs are separate. For example, the amount of spacing between the various abdominal organs including the aorta 380, left kidney 382, pancreas 384 and liver 386 is increased.
  • a virtual ablation is performed wherein the outer shell 390 of a virtual tissue are sequentially removed over multiple time points.
  • the anatomic structure in which virtual ablation is performed can be placed in a 3D cursor 388 to help direct the ablation.
  • a structure such as the colon 392 can be sliced (e.g., using a virtual knife) and opened such that the inner aspects including a polyp 394 inside the hollow viscus can be more carefully examined. Voxel manipulation would be required to achieve this aspect.
  • FIG. 4 illustrates a flow diagram and an example heads-up display type icon delivering pertinent information to the radiologist during the interpretation of the imaging examination.
  • First a flow diagram is illustrated to show how the virtual windshield 406 can be used to assist with interpretation of the radiology images.
  • the first step 400 is for the radiologist (or other medical personnel) to move to a checklist item on the report.
  • the second step 402 is for the virtual assistant type icon (also known as the virtual windshield 406) to display the pertinent information.
  • the third step 404 is for the radiologist to review the virtual assistant type icon display, review the images and input that section of the report.
  • the heads-up display we refer to as a virtual windshield 406 (analogous to a‘heads-up’ display in an aircraft) which could be called at any time during the examination process and would display on a single virtual windshield items relevant to the patient’s entire case and/ or the particular item being examined on the medical institution’s checklist.
  • a virtual windshield 406 analogous to a‘heads-up’ display in an aircraft
  • questions relevant to the review could include: what is next on the checklist; what was the impetus for obtaining medical images; is there prior history of current condition; what are any laboratory results; and, if artificial intelligence routines have been applied, what were the results and any listed indictors.
  • an example virtual windshield 406 is shown.
  • Figure 5 illustrates flow diagram and illustration of virtual tool kit inputs leading to voxel alterations.
  • voxels can be manipulated in size, shape, position, orientation, or internal parameter. Further, voxels can be created or eliminated at the direction of the virtual tool.
  • the original voxel 500 is illustrated.
  • the size of the original voxel 500 can be decreased to yield a smaller voxel 502.
  • the size of the original voxel 500 can be increased to yield a larger voxel 504.
  • the original cube shaped voxel 500 can be altered such that eight smaller voxels 506 are created, such that the each of the eight smaller voxels 506 have one eighth of the volume of the original voxel 500.
  • the original voxel 500 can also be eliminated 508.
  • the internal data unit (e.g., grayscale value) of the original voxel 500 can be altered 510.
  • Additional internal data units e.g., texture, tissue type property, etc.
  • the orientation of the original voxel can be changed 514.
  • the location of the original voxel 516 can be shifted (i.e., the voxel is moved), which can be performed by altering the x, y, z coordinate 518 of the original voxel 516 to a new x, y, z coordinate 522 of the shifted voxel 520 such that it has been moved a particular x-distance 524, y- distance 526 and z-distance 528.
  • the shape of the original voxel 500 can change, for example changing from a cube to an octahedron 530 or to that of a cylinder 532.
  • Figure 6 illustrates voxel manipulation based on interaction with a virtual tool.
  • Figure A illustrates a 3D cursor 600 containing a volume of interest 602. Note that the volume of interest is of homogeneous mid-gray color. Also, note that the tip 604 of the virtual tool (i.e., in this case, the virtual focal point pen) 606 is located outside of the volume of interest 604.
  • Figure B a 3D cursor 608 containing a volume of interest 610 with a change in position and orientation of the virtual tool (i.e., in this case, the focal point pen) 612 with a portion of the virtual tool including the tip of the virtual tool 614 now entering both the virtual 3D cursor 608 and the volume of interest (e.g., contains selected tissue from the volumetric medical image) 610.
  • the volume of interest e.g., contains selected tissue from the volumetric medical image
  • multiple voxels 616 in close proximity to the tip 614 of the virtual tool 612 have changed/highlighted to a light gray color.
  • the transparency of the tissue 610 within the 3D cursor 608 has changed to better visualize the tissue 616 highlighted by the virtual focal point pen 612 and the virtual focal point pen 612 itself.
  • Figure C illustrates another change in position and orientation of the virtual focal point pen 618 and corresponding alterations of the visual appearance of the nearby voxels 620.
  • the volume of interest e.g., contains selected tissue from the volumetric medical image
  • Figure 7 illustrates virtual eye tracker symbol with variable display patterns.
  • the human eye can perform saccades to quickly switch from fixed object to fixed object.
  • Figure 7A illustrates the carotid bifurcation 700 with a virtual eye tracker symbol 701 (e.g., a blue dot) at multiple positions over multiple time points.
  • the virtual eye tracker symbol 701 is located over the inferior aspect of the common carotid artery 704 portion of the carotid bifurcation 700.
  • the virtual eye tracker symbol 701 is located over the carotid bulb 708 portion of the carotid bifurcation 700.
  • the virtual eye tracker symbol 701 is located over the mid aspect of the internal carotid artery 712 portion of the carotid bifurcation 700.
  • the virtual eye tracker symbol 701 is located over the external carotid artery 716 portion of the carotid bifurcation 700.
  • Figure 7B illustrates the virtual eye tracker symbol 718 located over the inferior aspect of the common carotid artery 720 portion of the carotid bifurcation 700.
  • the virtual eye tracker dot 718 moves in a smooth manner up to the internal carotid artery at time point #N where the virtual tracker dot 718 reaches the final destination, such as the mid portion of the internal carotid artery 722.
  • the user could select a frame rate of 60 frames per second and a movement rate of 2 cm/second. The distance that the virtual tracker dot travels would determine the total time for the particular segment.
  • a high frame rate facilitates smooth eye tracking of the human eye and potential avoidance of skipped areas by the user leading to a more comprehensive viewing.
  • the virtual eye tracer dot 718 can take on many shapes, sizes, colors, speed of the movement, etc.
  • Figure 8 illustrates the virtual knife, which can be used by medical personnel to‘carve away tissue’ from an existing 3D medical imaging volume to permit enhanced viewing of the interior structure.
  • the virtual knife is used to investigate the patient’s heart. This task is performed in conjunction with a 3D cursor encasing a 3D medical volume of the heart.
  • Figure 8A illustrates the virtual knife 800 with a virtual cutting surface 802 and associated registration point(s) 804. The medical person viewing the medical images could: pick up the virtual knife 800 and move it to the volume of interest shown as heart 806 encased in a 3D cursor 808 with tissue external to the heart subtracted.
  • Figure 8B illustrates passing the knife 800 equipped with a cutting surface 802 and registration points 804 through the 3D volume of interest 806 such that a portion of the tissue 810 (i.e., aorta and pulmonary artery) is cut and then displaced.
  • Figure 8C illustrates removal of the aorta and pulmonary artery to allow the medical personnel to peer into the aortic valve 812 and pulmonary valve 814. Further carving could allow inspection of the tricuspid valve (not shown).
  • 4D datasets can be viewed in conjunction with the virtual tool kit to provide even more enhanced viewing of the heart.
  • Figure 9 illustrates a virtual ride through a blood vessel using a visual transport tool.
  • the virtual transport tool is a means/pathway by which the medical personnel can move within a hollow structure and visualize the conditions therein.
  • the virtual transport tool can be used alone or in conjunction with the virtual catheter.
  • the virtual transport tool could be used in conjunction with the virtual catheter to producing a 3D digital image of the interior of the vascular structure within the patient or to train the interventionalist to treat vascular conditions.
  • the virtual transport tool provides a vision of what is ahead within the blood vessels. In this example, a blood vessel is shown.
  • Blood within the blood vessel could be digitally subtracted and a virtual light could shine on the blood vessel walls for some distance forward from the current viewing perspective within the tunnel and X, Y, Z coordinates of the constrictor be recorded. Note that during the segmentation process, blood would be removed from the blood vessels, but the blood vessel structure would remain. This allows the virtual transport tool to visualize the internal structure of the blood vessel unobscured. Typically, the virtual transport tool will be within the center of the blood vessel and looking ahead. Note that the center of the viewing point is within the center of the blood vessel, but the actual viewing perspectives would be offset in accordance with the left and right eye viewing perspectives.
  • the medical person viewing the medical images could visualize what it looks like to travel within a blood vessel, as viewed from a 3D headset (e.g., augmented reality). Note that it is possible to expand the diameter of the virtual blood vessel to enhance viewing in accordance with voxel manipulations (US Patent Application 16/195,251, which is incorporated by reference). As an analogy, it may be difficult to look into a small pipe or difficult to simultaneously view all portions of a large tunnel from within; therefore, the ability to adjust the size of the tunnel offers great viewing flexibility. If the user identifies an anomalous condition, then the user could take a differing position and orientation to study the condition. The distance of the blood vessel displayed would be selected by the medical personnel and the lighting intensity of the structure would also be selected by the medical personnel.
  • a common clinical application anticipated through the use of these techniques includes measurement of a carotid atherosclerotic plaque (e.g., measuring the lumen at the narrowed regions as well as the length of the narrowing, which may prove to be a better metric for determining stent type and placement than current methods, such as the North American Symptomatic Carotid Endarterectomy Trial (NASCET) measurement technique).
  • NASH North American Symptomatic Carotid Endarterectomy Trial
  • a rolling computation of the lumen of each vessel would be performed with metrics provided to the medical professional. For example, this ride could be used during an assessment of the vascular structure and potential need for insertion of stents.
  • Figure A illustrates a normal blood vessel interior surface with blood deleted 900 without plaque.
  • This largest circle 900 represents the inner aspect of the blood vessel at the current viewing location.
  • the texture of the inner mucosal surface 901 is shown.
  • the middle sized square dot circle 902 represents the inner aspect of the blood vessel at an intermediate distance from the current viewing location, such as 5 cm from the current viewing location.
  • the smallest round dot circle 904 represents the farthest distance that the user could see from the current viewing location.
  • a virtual marker for example, the double-headed arrow 906 could indicate the length within the blood vessel actively being viewed, such as 10 cm from the current position of the blood vessel 900 to the farthest position within the blood vessel that can be seen 904.
  • Figure B illustrates narrowing of the blood vessel lumen due to atherosclerotic plaque and a road sign providing a description of a measurement.
  • This largest circle 910 represents the inner aspect of the blood vessel at the current viewing location.
  • the texture of the inner mucosal surface 911 is shown.
  • the middle sized square dot circle 912 represents the inner aspect of the blood vessel at an intermediate distance from the current viewing location, such as 5 cm from the current viewing location.
  • the middle circle 912 at the 2 o’clock location is shown to bulge inward 916.
  • Both the round portion of the middle circle 912 and the portion of the middle circle that bulges inward 916 are located at 5 cm from the current viewing location.
  • the entirety of the dotted line including 912 and 916 is located at 5 cm from the current viewing location; thus, it represents an“isodistance line”.
  • the smallest round dot circle 914 represents the farthest distance that the user could see from the current viewing location, such as 10 cm from the current viewing location.
  • a virtual marker for example, the large double-headed arrow 918 could indicate the length within the blood vessel actively being viewed, such as 10 cm from the current position of the blood vessel 910 to the farthest position within the blood vessel that can be seen 914.
  • a smaller double-headed arrow 920 is shown from the expected position of the dotted line (demarcating the 5 cm distance away assuming no plaque/ narrowing) to the actual position that is 5 cm away and is more inwardly located 916. Please note that when the radius of a particular“isodistance line” decreases, this would indicate an area of narrowing. Please note that when the radius of a particular“isodense line” increases, this would indicate an area of expansion / ectasia / aneurysm.
  • Figure C illustrates the visual transport tool approaching a branching juncture of three blood vessels. During pre-panning, the medical professional could select which of the blood vessels the catheter should enter and this blood vessel could be highlighted in false color for verification of the correct path for the catheter.
  • the largest circle 922 represents the inner aspect of the blood vessel at the current viewing location. The texture of the inner mucosal surface 923 is shown.
  • a medium sized half circle at the 3 o’clock position 924 represents a branch vessel (e.g., internal carotid artery), which would be the desired option to travel into.
  • a medium sized half circle at the 9 o’clock position 926 represents an additional branch vessel (e.g., external carotid artery) would be a second option to travel into, but not desired in this example scenario.
  • a dotted red line 928 is illustrated as an example of a virtual tool to be displayed over the image as a visual cue to help notify the medical personnel of the branch that they are desiring to enter.
  • a virtual road sign 930 is illustrated, which states“5.0 cm away from the carotid bifurcation. Head towards 3 o’clock to enter the internal carotid artery.”
  • Figure 10 illustrates the virtual catheter, which could be used in conjunction with a volumetric medical image of the vascular structure within the patient with the assistance of virtual icons.
  • the 3D virtual catheter could be used during pre-operative planning of an interventional operation, such as acquiring important distance or angle measurements.
  • an interventional procedure a 3D virtual catheter is used in treatment of an aneurysm.
  • Figure A shows the solid blue line 1000 as the catheter located in the right groin region entering at the common femoral artery and extending into the right external iliac artery and to the aorta.
  • the tip of the catheter 1002 is a small black solid circle. Note that path that has been traversed is shown as a solid lines and the planned path is shown in hashed line 1004.
  • the planned routes can be implemented via placement of location markers in a particular blood vessel that the medical professional wants to target.
  • location markers can be at way points along the desired paths or at the final target lesion 1006 (e.g., brain aneurysm).
  • the path connecting the markers can be marked (e.g., blue dotted line).
  • measurements along the blue dotted line can be performed.
  • a road sign 1008 can be displayed to inform medical personnel of the distance to the vascular junction to be used in an actual interventional medical procedure.
  • a virtual icon 1010 is also shown, such as a 2D or 3D object.
  • Figure B shows virtual catheter 1012 extending into the thoracic aorta.
  • the dashed line 1016 represents the desired pathway of the catheter, which is through the brachiocephalic artery, then the common carotid artery, then the internal carotid artery then the middle cerebral artery, and finally into the aneurysm 1018.
  • a road sign can be displayed to inform the medical personnel of the distance to the vascular junction to be used in an actual interventional medical procedure.
  • Augmented reality distance markers could be added to the 3D virtual catheter for each intersection the interventionalist would need to take care and be prepared to change from one blood vessel to another. Screen captures of all key vascular junctures could be annotated angular changes from current path in coordinate system X-Y, X-Z and Y-Z planes.
  • Figure C shows a blow up of a vascular junction wherein multiple path options occur and the medical personnel must take care in moving the catheter to the correct blood vessel.
  • the descending thoracic aorta 1022, brachiocephalic artery 1024, left common carotid artery 1026, left subclavian artery 1028 and ascending thoracic aorta 1030 are shown.
  • the virtual catheter 1032 is shown.
  • the tip of the virtual catheter 1034 is shown.
  • the blue dotted line 1036 represents the desired catheter path.
  • Figure 11 illustrates the general concept of 3D medical images and example techniques behind the explosion of 3D medical images into multi separate organs.
  • the medical person viewing the medical images could divide the 3D digital volume of interest into multiple parts based on their common characteristics (e.g., similar Hounsfeld units, anatomical atlas, etc.).
  • the general process is illustrated in this figure wherein the desire is to examine key organs individually. Such a process could be structured, for example, in accordance with items on an image review checklist.
  • Figure A illustrates a generalized illustration of organs within the abdomen.
  • liver 1100, right adrenal gland 1102, right kidney 1104, inferior vena cava 1106, right iliac vein 1108, spleen 1110, aorta 1112, pancreas 1114, left adrenal gland 1116, left kidney 1118, gastrointestinal tract 1120 and left iliac artery 1122 are shown.
  • the process would be to expand these organs outward in X, Y, Z directions from an approximate center point in the torso to facilitate the individual inspection without visual interference from adjacent organs.
  • Figure B illustrates the organs after segmentation has been applied, noting dashed lines around the organs to illustrate the segmentation process.
  • liver 1124, right adrenal gland 1126, right kidney 1128, inferior vena cava 1130, right iliac vein 1132, spleen 1134, aorta 1136, pancreas 1138, left adrenal gland 1140, left kidney 1142, gastrointestinal tract 1144 and left iliac artery 1146 are shown. Note that a dashed line is shown to better show the segmentation. Figure C shows the exploded view. The coordinates (X, Y, Z) of the organs would be modified to new positions indicated by the dashed lines.
  • the software for implementation of the concept is, but not limited to, the following procedures.
  • the medical person viewing the medical images could select a point within the 3D digital volume (ideally near the center of the 3D digital volume and between segmented sub-volumes), which would act at the origin point for the explosion.
  • Figure D illustrates one of multiple ways the sub-volumes of the 3D digital can be separated as if an explosion occurred.
  • One of the ways, but not limited to, is as follows: create eight large cubes 1168 each touching the center point and each parallel to the X, Y, Z axes (e.g., the first cube would be positive in X, positive in Y and positive in Z; the second cube could be positive in X, negative in Y and positive in Z; and so on). Then the medical person viewing the medical images establishes a distance factor for sub-volumes close to the center point, a larger distance factor for those further away. Then these factors are applied to all voxels within each specific sub-volume of the 3D digital image based on which cube the center voxel of the sub-volume was in.
  • the X, Y, Z coordinates of voxels within that sub volume would increase by the specified factor in the positive X, positive Y and positive Z direction.
  • the increases would be in positive X, negative Y and positive Z directions).
  • the medical person viewing the medical images modify the factors changing the spread between the sub-volumes during the course of the examination. For example, a moderate spread is shown 1170. Alternatively, a larger spread is shown 1172.
  • Figure 12 illustrates use of virtual transport viewer to perform a more accurate virtual colonography review.
  • One of the alternatives is to go through a virtual colonography, wherein a CT scan is performed and the inner mucosal surfaces of the colon is reviewed. If no polyps are found, a treatment phase is not necessary. If, however, polyps are found, then at some later date, the preparation phase is repeated and a treatment phase (i.e., colonoscopy) is performed to remove the polyps.
  • the virtual transport viewer is employed to view the inside of the colon and determine if polyps are present.
  • FIG. 12A illustrates the view of the interior surface of the colon with the air and stool deleted 900 without a polyp.
  • This largest circle 1200 represents the inner aspect of the colon at the current viewing location. The texture of the inner mucosal surface 1201 is shown.
  • the middle sized square dot circle 902 represents the inner aspect of the colon at an intermediate distance from the current viewing location, such as 5 cm from the current viewing location.
  • the smallest round dot circle 1204 represents the farthest distance that the user could see from the current viewing location.
  • a virtual marker for example, the double-headed arrow 1206 could indicate the length within the blood vessel actively being viewed, such as 10 cm from the current position of the blood vessel 1200 to the farthest position within the blood vessel that can be seen 1204.
  • Figure 12B illustrates the view of the interior surface of the colon with the air and stool deleted 900 with three polyps. This largest circle 1210 represents the inner aspect of the colon at the current viewing location.
  • the texture of the inner mucosal surface 1211 is shown.
  • the middle sized square dot circle 1212 represents the inner aspect of the colon at an intermediate distance from the current viewing location, such as 5 cm from the current viewing location.
  • the smallest round dot circle 1214 represents the farthest distance that the user could see from the current viewing location.
  • a virtual marker, for example, the double-headed arrow 1216 could indicate the length within the blood vessel actively being viewed, such as 10 cm from the current position of the blood vessel 1210 to the farthest position within the blood vessel that can be seen 1214.
  • a villous polyp 1218 is illustrated.
  • a virtual road sign 1220 with distance to key landmark is shown, such as“3 cm to villous polyp at 10 o’clock”.
  • a sessile polyp 1222 is shown.
  • FIG. 13 illustrates a portion of a virtual 3D volumetric medical image which contains the colon portion of the large intestine which has, through voxel manipulation, been stretched in such a manner so that is one long straight tube. And then, the contents within the tube have segmented and subsequently/ eliminated from the tube. And, finally, the tube is split along the length axis and opened to permit viewing of the internal structure.
  • There are methods to physically examine the internal structure of the colon which involve: preparation, insertion of air to fill and expand the colon, insertion of a camera with light and movement of this system along the length of the colon to observe and record the internal structure.
  • a rendered TV recording can be presented to medical personnel and patient.
  • a limitation of the rendered TV recording is that polyps can be occluded from the TV view by folds along the colon. Further, if polyps are found, the patient must return at a later date for a colonoscopy which entails another preparation and subsequent removal of polyp tissue.
  • the process invoked in this virtual process does not require the unpleasant preparation phase in the preliminary examination.
  • a CT image with/ without contrast is performed.
  • a 3D virtual image is constructed from the CT 2D slices (US Patent 8,384,771, which is incorporated by reference). Segmentation (US Patent Application 15/904,092) is performed and tissue subtracted external to the colon.
  • Figure 14 illustrates insertion of virtual contrast and its flow through the vascular system. Initially, the blood within the affected vessels has been removed.
  • the blood vessels are in their normal, non-pathologic state and normal blood flow is illustrated through the placement of virtual contrast.
  • the proximal portion of the blood vessel 1400, mid portions of the blood vessel 140 la and 140 lb and distal portions of the blood vessel l402a, l402b and l402c are shown.
  • virtual contrast when virtual contrast is inserted, it would mimic normal blood flow where it to be imaged.
  • Three time points are shown including: an initial time point 1404; a subsequent time point 1406; and, a final time point 1408. At the initial time point 1404, all of the native blood voxels have been removed and no virtual contrast has been inserted.
  • the blood vessels are in a pathologic state (i.e., a blood clot 1413 is lodged into one of the distal artery branches).
  • the proximal portion of the blood vessel 1400, mid portion of the blood vessel l40la and 140 lb and distal portions of the blood vessel l402a, l402b and l402c are again shown.
  • the virtual contrast would mimic an altered blood flow pattern.
  • Three time points are shown including: an initial time point 1414; a subsequent time point 1416; and, a final time point 1418. At the initial time point 1414, all of the native blood voxels have been removed and no virtual contrast has been inserted.
  • virtual contrast 1412 shown in gray has been inserted into the proximal portion of the blood vessel 1400, mid portions of the blood vessel 140 la and 140 lb, and two of the distal branches of the blood vessels l402b and l402c; however, one of the distal portion of the blood vessel l402a does not fill with virtual contrast 1412 because it is blocked by a blood clot 1413.
  • blood clot which is assigned a blockage-type interactive voxel parameter.
  • insertion of virtual contrast is shown in the normal setting of a blood vessel and in the altered setting of a blood vessel (i.e., with a blood clot).
  • the virtual contrast can progress from proximal to distal up to the point of the clot, but not beyond the clot. The remaining branches experience the insertion of virtual contrast.
  • assigning a blockage-type interactive voxel parameter stops flow of virtual contrast.
  • a surgical clip blockage-type interactive voxel parameter can be used.
  • Figure 15 illustrates an ablation technique could be used, inter alia, in conjunction with 3D digital structure within a 3D cursor. This permits careful examination of the interior of an organ.
  • the method underlying the ablation technique consists of, but not limited to, the following procedures:
  • Figure 15A illustrates the first step of the process which is determining the outer‘shell’ of organ of interest to the medical person viewing the medical images (e.g., using segmentation techniques outlined in US Patent Application 15/904,092) (Note: in this figure a liver 1500 is illustrated which is encased in a 3D cursor 1502 (US Patent 9,980,691).
  • Figure 15B illustrates the next step of the process which is sequentially eliminate one voxel deep from all of the voxels on the outer surface. For example, the outer shell of voxels is shown in black 1510. Then, repeat this step multiple times on the remaining outer layer(s) of tissue at the direction of the medical person viewing the medical images.
  • Figure 15C illustrates the next step of the process, which is as the layers are subtracted, the internal tissue types are revealed.
  • the original outer shell 1512 is shown.
  • the shell after N steps of the ablation 1514 is shown. Note that methods by which to display voxels may include sparse display or translucent appearance of non-prioritized voxels whereas voxels that are prioritized would not be sparsely displayed.
  • Figure 15D normal liver tissue is ablated away and abnormal liver tissue remains.
  • focal nodular hyperplasia 1518 a benign lesion called focal nodular hyperplasia 1518 is revealed after the ablation process.
  • the focal nodular hyperplasia 1518 is shown with the characteristic imaging feature of the hypoattenuating central scar 1520.
  • Such a process would be a more useful method for viewing of tumors inside of a solid organ as compared to the process of searching on each slice.
  • the volume of liver tissue has decrease from the original volume as a result of repeated ablative steps.
  • Figure 16 illustrates a virtual focal point pen guiding voxel manipulation.
  • the movement of the virtual focal point pen would be controlled by medical person viewing the medical images.
  • This figure illustrates expanding distance between blood vessels which are closely space, overlapping and difficult to differentiate and a process to expand the distance between blood vessels. If an arterioveous malformation occurs in the brain in a region where multiple blood vessels are close together, it is difficult to identify which of these blood vessels to inject treatment material. Expanding the distance digitally can help identify the proper blood vessel for injection. Two conditions will be discussed. First, non-blood vessel tissue separates the blood vessels. Second, several blood vessels are in a cluster with little or no non-blood vessel tissue separating the blood vessels.
  • tissue segmentation to determine types of tissue present in the volume of interest; b) for all non-blood and non blood vessel type of tissue, expand the volume by a multiplicative or additive factor; c) adjust the coordinates of blood and blood vessels to account for the expansion of the non-blood and non-blood vessel type of tissue.
  • d) perform segmentation to determine which voxels are predominately blood and which voxels are tissue (i.e., blood vessels); e) temporarily eliminate the tissue voxels; c) then use a multiplicative (or additive factor) to all coordinates of blood voxels; f) apply smoothing routine to blood vessels (optional); g) encase the blood voxels with tissue vessels.
  • the display for the medical person viewing the medical images show the expanded blood vessel structure and thereby facilitate treatment.
  • ACM cerebral arteriovenous malformation
  • a complex cerebral AVM can consist of multiple tortuous feeding arteries, a tangle of nidus with aneurysms and multiple draining veins. It is extremely difficult to understand the precise anatomy of this complex structure.
  • a process is illustrated as follows.
  • the structures that are desired to separate are the two pink blood vessels, which are extremely close together.
  • the user moves the virtual focal point pen tip to the space in between the two blood vessels.
  • the process establishes a virtual red dot, which shows the location of voxels which will later become replicated and inserted as a manipulated tissue type.
  • the tissue properties in between the two structures of interest e.g., cerebrospinal fluid.
  • the third step 1604 perform voxel manipulation (e.g., insertion of additional cerebrospinal fluid type voxels) and concurrently changing the location of the two blood vessels, such that the distance between the two blood vessels is increased.
  • a first blood vessel 1606 and a second blood vessel 1608 are closely spaced with only a sliver of intervening cerebrospinal fluid type voxels 1610.
  • a virtual pointer 1612 is shown.
  • the tip of the virtual pointer 1614 is also shown.
  • a virtual symbol e.g., red dot 1616
  • the tissue properties in between the two structures of interest can be assigned a particular tissue property.
  • each of these voxels has changed to a light blue 1618.
  • a first blood vessel 1606 and a second blood vessel 1608 are still closely spaced to one another.
  • three additional columns of cerebrospinal fluid voxels 1620, 1622 and 1624 are inserted. Note that the spacing between the first blood vessel 1606 and the second blood vessel 1608 has been increased. This is useful in that 3D viewing may now be improved with better ability to see and understand the relationship between closely spaced structures.
  • Figure 17 illustrates the total imaging volume and multiple types of sub-volumes.
  • slices can be arranged to build a volume as described in US Patent 8,384,771.
  • a medical professional or computer program
  • a medical professional or computer program
  • the sub-volume can pulled out of the stack of 2D slices and be viewed and analyzed.
  • a first sub-volume 1710 created from a first set of 2D images 1700 is pulled to upward and to the side of the first set of 2D images 1700.
  • a second sub-volume 1712 created from a second set of 2D images 1704 is pulled to the side of the second set of 2D images 1704.
  • a third sub-volume 1714 created from the third set of 2D images 1708 is pulled to the side and downward in relation to the third set of 2D images. Please note that multiple methods of assigning boundaries of stacks could be performed.
  • sub-volume size e.g., 20 2D slices
  • dividing the sub-volumes accordingly e.g., slices 1-20 are assigned to sub-volume #1, slices 21-40 are assigned to sub-volume #2, etc.
  • the number of slices in each stack can vary.
  • the user could use the virtual pen to touch and select the boundaries, such as points 1702 and 1706.
  • the total imaging volume can be divided into many different combinations of sub-volumes of varying size, which can be viewed from many different angles (US Patent 8,384,771).
  • sub-volumes can be made from arranging a collection of slices.
  • Figure 18 illustrates sequencing of movement of the 3D cursor through the volume of interest in a random pattern.
  • a random pattern is used based on items of possible interest, the reviewing medical person can view the totality of the virtual medical image volume at once and apply techniques such as changing transparency and applying false color to structures of differing density than the nominal density of the organ being examined. The reviewer can then move and re-size the 3D cursor to the various objects of potential interest for detailed examination.
  • a total scanned volume 1800 is shown.
  • a first sub-volume displayed in a 3D cursor is shown at the first time point 1802.
  • the 3D cursor is subsequently moved in direction 1804.
  • a subsequent sub-volume displayed in a second re-sized 3D cursor is shown at a subsequent time point 1806.
  • the 3D cursor is subsequently moved in direction 1808.
  • a subsequent sub-volume displayed in a second re-sized 3D cursor is shown at a subsequent time point 1810.
  • the 3D cursor is subsequently moved in direction 1812.
  • a subsequent sub-volume displayed in a second re-sized 3D cursor is shown at a subsequent time point 1814.
  • This figure shows multiple moves and resizes the 3D cursor to view example tissues of interest.
  • This can type of search pattern can expedite the review process.
  • This search patterns employ a 3D cursor (US Patent 9,980,691 and US Patent Application 15/878,463).
  • sub-volumes are displayed to the medical personnel in an automated pattern, which includes, but is not limited to, the following: windshield wiper pattern or layer-by-layer pattern.
  • the 3D cursor and/or sub-volume within the 3D cursor can be copied and pasted to a virtual movable table for later review.
  • the radiologist may want to round up all of the potentially abnormal or definitely abnormal findings first. Then, the radiologist may want to study each of the abnormal findings in great detail at a later time period.
  • Figure 19 illustrates an example of a systematic pattern of viewing of medical images (e.g., sequential virtual windshield wiper type pattern).
  • the x-direction 1900, y-direction 1902 and z-direction 1904 are shown.
  • the total imaging volume 1906 is shown.
  • a virtual windshield wiper can have several implementations with the cursor moving in multiple systematic fashions.
  • a first sub-volume 1908 being examined in the 3D cursor at the initial time point where one of the coordinates of the 3D cursor has one comer at position (0, 0, 0) 1910 is shown.
  • the 3D cursor first moves in a fashion wherein the x-direction is increased 1912 and the y-coordinates and z-coordinates of the 3D cursor are unchanged as illustrated by the dashed arrow and the sub-volumes along this movement direction can be optimized.
  • the pattern is to sequentially increase the X coordinate while holding the Y and Z coordinates constant.
  • the 3D cursor is moved in a fashion wherein the y-direction is increased 1916 and the x-coordinates and z-coordinates of the 3D cursor are unchanged, as illustrated by the dashed arrow and the sub-volumes along this movement direction can be optimized.
  • the max value for the X coordinate is reached, the y coordinate is increased an increment and the X coordinate is sequentially decreased until the min X coordinate is reached, then Y coordinate is again incremented.
  • This process of moving the 3D cursor in the x-direction 1900 and y-direction 1902 is then repeated 1918 until at which point a bottom layer of the total imaging volume 1906 has been fully examined by the 3D cursor.
  • the Z coordinate is incremented.
  • the 3D cursor can be shifted 1920 upwards in the z-direction 1904.
  • an abnormality 1922 may be discovered at a particular 3D cursor position 1924. Such an abnormality can be placed in to a virtual bucket or virtual 3D virtual movable table for further analysis.
  • sub volumes are displayed to the medical personnel in an automated pattern, which includes, but is not limited to, the following: windshield wiper pattern or layer-by-layer pattern.
  • An automated search pattern through the volume of interest may prove to increase the probability of detection.
  • an automated search pattern is shown as the 3D cursor moves through the volume of interest. Note that a mass is identified in a later sub-volume. Future automated search pattern through the volume of interest is performed back and forth in each layer (similar to a windshield wiper) and then back and forth in the next layer.
  • Figure 20 illustrates volume of interest to be reviewed and a process whereby any areas, which intended for review which were missed, could be highlighted to the medical person performing the review. These identified sub-volumes could be reviewed subsequently, thereby ensuring completeness of the review.
  • This process invokes the sequential selection of sub-volumes of the volume of interest through use of the 3D cursor (US Patent 9,980,691 and US Patent Application 15/878,463) to sequentially step through the volume being examined (e.g., per following medical institution checklist). Further, after the step-by-step process has been completed, the question could arise as to whether the entire volume has been examined. In this implementation, the volume contained in each of the 3D cursors which had been examined could be totaled and subtracted from the total original volume.
  • This figure illustrates sequencing of movement of the 3D cursor through an organ of interest. A volume of interest (i.e., liver) 2000 is shown. A sub-volume 2002 displayed at time point #1 is illustrated.
  • the 3D cursor moves 2004 in a systematic fashion through the volume of interest 2000 such as described in Figure 19.
  • the final sub-volume 2006 would be displayed at time point #N.
  • Control of viewing of the medical images e.g., changing from one increment to another
  • the user could move the control the movement of the 3D cursor by the joystick or other geo-registered tools, as discussed in US Patent Application 16/524,275.
  • volumes displayed in the 3D cursor can be tracked and then reviewed at a later time (i.e., prior to the completion of the exam).
  • Sub-volume size changes based on indication routine screening versus cancer.
  • recording the position of the 3D cursor over time and comparing the sub-volumes displayed with the total volume would enable accounting for sub-volumes that have not yet been displayed to the medical professional 2008.
  • subtracting the sub volumes have been displayed from the whole volume it is possible to determine which sub-volumes have not yet been displayed.
  • a few areas of the structure were missed (i.e., not included in the 3D cursor volumes) 2008; these can be tracked and the radiologist has the option to review these areas prior to the completion of the exam.
  • These missed sub-volumes 2008 can be moved to a new position and examined.
  • the user can select the size of the 3D cursor, rate of the movement of the cursor and the computer performs automated movement through the volume of interest on the checklist item. If the organ is unremarkable, the sub-volumes within the cubes could be changed in a way that an the imaged structures are unremarkable (e.g., deleted, changed in Hounsfield unit, etc).
  • Figure 21 illustrates an icon of a human with the location of the 3D virtual cursor included at the approximate location within the body. This icon could be used in conjunction with viewing display of 3D medical images. During the course of the examination of the volume by the medical person viewing the medical images, it may be useful to quickly refer to an icon in order to re-orient where exactly in the body is some tissue of interest/ concern. The icon would also be of utility in discussions between medical personnel.
  • Figure 21A an icon of the body in a vertical position facing forward 2100 is shown. The icon is marked up with an outline of the sub-volume being examined.
  • Such a markup includes, but is not limited to, the following: markup of region of area of concern as indicated by the ordering physician; markup of the segmented volume that the radiologist is actively working on (e.g., radiologist is actively working on the liver item on the checklist, so the segmented liver is marked up on the icon); markup of sub-volume being examined by the radiologist (e.g., radiologist is actively working on a sub-volume within the liver within the confines of the volume -subtending 3D cursor); markup of viewing perspectives in relation to the icon.
  • the ordering physician may indicate an area of concern (e.g., sending patient specific image to the radiologist as described in US Provisional Patent Application 62/843,612 for METHOD OF CREATING A COMPUTER-GENERATED PATIENT SPECIFIC IMAGE) and this area can be marked up on the virtual icon 2102.
  • the segmented volume 2104 that the radiologist is actively working on e.g., liver
  • a sub-volume within a 3D cursor 2106 can be marked up.
  • An additional symbol may be shown external to the icon. For example, a initial viewing perspective symbol 2108 is shown to denote the initial viewing perspective.
  • the user’s extended reality headset 2114 is illustrated with a left eye display 2116 and a left eye view of the 3D cursor 2118 and a right eye display 2120 with a right eye view of the 3D cursor 2122.
  • a left eye view of the marked up 3D icon 2124 is shown in the left eye display 2116 and a right eye view of the marked up 3D icon 2126 is shown in the right eye display.
  • the outline of the sub volume being examined can be one of the markups of the icon.
  • the approximate location of the 3D cursor(s) within the human body icon is another example markup of the icon. Orientation of the body would be under the control of the medical person viewing the medical images, as would whether to display the icon or not. For example, the icon could be rotated, translated, warped (with corresponding voxel manipulation if desired) or other alterations as directed by the radiologist. Furthermore, adding a marked up icon of a 3D cursor to the diagnostic 2D radiology monitor could be performed.
  • the tissue contained in the 3D cursor As the medical person viewing the medical images rotate, tilt, and zoom, the tissue contained in the 3D cursor, it may be useful to see where the current viewpoint is relative to the initial viewpoint (e.g., voxels’ locations have changed from initial orientation through rolling, pitching and/ or yaw commands to a new orientation).
  • This illustration shows a cured arrow originating at the initial viewing point and terminating at the current viewing point. Whether to display the icon or not would be under the control of the medical person viewing the medical images.
  • the icon of a 3D cursor displays the contents of the 3D cursor which have been rotated and viewed from different viewpoints, it is useful to simultaneously see where the current position and view point are with respect to the original position.
  • Figure 22 illustrates a virtual moveable table for storing virtual images of suspect tissue stored by checklist category.
  • This figure depicts a virtual movable table 2200 on which there are virtual storage bins that correspond to items on the medical institution checklist 2202, plus a bin for emergency items 2204, and a general/ miscellaneous bin 2206 (e.g., image artifact, teaching case, quality improvement, etc.).
  • the emergency bin 2204 could be used for placing findings for items of critical time sensitive information.
  • the virtual mobile table is mobile in the sense that the user could view the virtual mobile table on an extended reality headset off to the side away from the imaging volume that the radiologist is currently working on. Then, the radiologist could move it or size it, such that it convenient for the work space.
  • Figure 23 illustrates a sample radiology report including an image processed with virtual tools. Please note that the sample radiology report 2300 includes a 3D cursor and image finding of the abnormality.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Public Health (AREA)
  • Physics & Mathematics (AREA)
  • Primary Health Care (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Epidemiology (AREA)
  • Databases & Information Systems (AREA)
  • Pathology (AREA)
  • Data Mining & Analysis (AREA)
  • Biomedical Technology (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Apparatus For Radiation Diagnosis (AREA)
  • Processing Or Creating Images (AREA)
  • Image Generation (AREA)
  • Magnetic Resonance Imaging Apparatus (AREA)

Abstract

L'invention concerne des outils virtuels utilisés pour manipuler des aspects d'un volume d'image médicale en trois dimensions. Les outils virtuels sont géo-enregistrés avec le volume d'image. La présentation du volume d'image est manipulée par un processeur d'image en réponse à l'utilisation des outils virtuels. Les outils virtuels peuvent être utilisés pour faciliter l'analyse du volume d'image. Les outils virtuels peuvent comprendre : un stylo à point focal virtuel ; un curseur 3D virtuel ; un visualiseur de transport virtuel ; un socle virtuel ; un couteau virtuel ; un cathéter virtuel ; un panneau de signalisation virtuel ; un outil d'ablation virtuel ; une table virtuelle ; un outil de contraste virtuel ; et des icônes virtuelles.
PCT/US2019/047891 2018-08-24 2019-08-23 Boîte à outils virtuelle pour radiologues WO2020041693A1 (fr)

Priority Applications (5)

Application Number Priority Date Filing Date Title
JP2021510084A JP2021533940A (ja) 2018-08-24 2019-08-23 放射線科医向けの仮想ツールキット
CN201980062928.3A CN113424130A (zh) 2018-08-24 2019-08-23 放射医师的虚拟工具套件
CA3109234A CA3109234A1 (fr) 2018-08-24 2019-08-23 Boite a outils virtuelle pour radiologues
AU2019325414A AU2019325414A1 (en) 2018-08-24 2019-08-23 A virtual tool kit for radiologists
EP19852803.6A EP3841450A1 (fr) 2018-08-24 2019-08-23 Boîte à outils virtuelle pour radiologues

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201862722513P 2018-08-24 2018-08-24
US62/722,513 2018-08-24

Publications (1)

Publication Number Publication Date
WO2020041693A1 true WO2020041693A1 (fr) 2020-02-27

Family

ID=69591478

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2019/047891 WO2020041693A1 (fr) 2018-08-24 2019-08-23 Boîte à outils virtuelle pour radiologues

Country Status (6)

Country Link
EP (1) EP3841450A1 (fr)
JP (1) JP2021533940A (fr)
CN (1) CN113424130A (fr)
AU (1) AU2019325414A1 (fr)
CA (1) CA3109234A1 (fr)
WO (1) WO2020041693A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11207133B1 (en) * 2018-09-10 2021-12-28 David Byron Douglas Method and apparatus for the interaction of virtual tools and geo-registered tools
EP4134971A1 (fr) * 2021-08-09 2023-02-15 Ai Medical AG Procédé et dispositifs pour prendre en charge l'observation d'une anomalie dans une partie du corps
US11941761B2 (en) 2019-12-31 2024-03-26 Novocure Gmbh Methods, systems, and apparatuses for image segmentation

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140088941A1 (en) * 2012-09-27 2014-03-27 P. Pat Banerjee Haptic augmented and virtual reality system for simulation of surgical procedures
WO2018022523A1 (fr) * 2016-07-25 2018-02-01 Magic Leap, Inc. Modification, affichage et visualisation d'images à l'aide de lunettes de réalité augmentée et virtuelle
US20180139518A1 (en) * 2004-11-02 2018-05-17 Touma Pierre A Wireless motion sensor system and method

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180139518A1 (en) * 2004-11-02 2018-05-17 Touma Pierre A Wireless motion sensor system and method
US20140088941A1 (en) * 2012-09-27 2014-03-27 P. Pat Banerjee Haptic augmented and virtual reality system for simulation of surgical procedures
WO2018022523A1 (fr) * 2016-07-25 2018-02-01 Magic Leap, Inc. Modification, affichage et visualisation d'images à l'aide de lunettes de réalité augmentée et virtuelle

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11207133B1 (en) * 2018-09-10 2021-12-28 David Byron Douglas Method and apparatus for the interaction of virtual tools and geo-registered tools
US11941761B2 (en) 2019-12-31 2024-03-26 Novocure Gmbh Methods, systems, and apparatuses for image segmentation
EP4134971A1 (fr) * 2021-08-09 2023-02-15 Ai Medical AG Procédé et dispositifs pour prendre en charge l'observation d'une anomalie dans une partie du corps
WO2023016997A1 (fr) * 2021-08-09 2023-02-16 Ai Medical Ag Procédé et dispositifs de prise en charge de l'observation d'une anomalie dans une partie du corps

Also Published As

Publication number Publication date
CA3109234A1 (fr) 2020-02-27
CN113424130A (zh) 2021-09-21
AU2019325414A1 (en) 2021-03-25
EP3841450A1 (fr) 2021-06-30
JP2021533940A (ja) 2021-12-09

Similar Documents

Publication Publication Date Title
US11304759B2 (en) Systems, methods, and media for presenting medical imaging data in an interactive virtual reality environment
US10878639B2 (en) Interactive voxel manipulation in volumetric medical imaging for virtual motion, deformable tissue, and virtual radiological dissection
JP2020191130A (ja) 自動医療画像注釈の検証及び修正のためのシステム及び方法
US11798249B1 (en) Using tangible tools to manipulate 3D virtual objects
US11547499B2 (en) Dynamic and interactive navigation in a surgical environment
US11594002B2 (en) Overlay and manipulation of medical images in a virtual environment
US11183296B1 (en) Method and apparatus for simulated contrast for CT and MRI examinations
US20070237369A1 (en) Method for displaying a number of images as well as an imaging system for executing the method
US10712837B1 (en) Using geo-registered tools to manipulate three-dimensional medical images
WO2020041693A1 (fr) Boîte à outils virtuelle pour radiologues
CN109559801A (zh) 图像处理结果的智能编辑
US11207133B1 (en) Method and apparatus for the interaction of virtual tools and geo-registered tools
CN113645896A (zh) 手术计划、手术导航和成像用系统
US20190019433A1 (en) Method and system for a preoperative surgical intervention simulation
US11763934B1 (en) Method and apparatus for a simulated physiologic change for CT and MRI examinations
US20240024029A1 (en) Systems and method of planning thoracic surgery
US20150320507A1 (en) Path creation using medical imaging for planning device insertion
CA3105430A1 (fr) Systeme et methode pour lier un graphique de segmentation a des donnees volumetriques
EP4181152A1 (fr) Traitement de données d'image pour évaluer une question clinique
US11295537B1 (en) Method and apparatus for inserting virtual contrast into CT and MRI examinations
Pelanis Assessment of treatment and visualization methods for minimally invasive liver surgery
JP2022123698A (ja) 医用画像処理装置、医用画像処理方法及びプログラム
Foo A framework for tumor segmentation and interactive immersive visualization of medical image data for surgical planning

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19852803

Country of ref document: EP

Kind code of ref document: A1

DPE1 Request for preliminary examination filed after expiration of 19th month from priority date (pct application filed from 20040101)
ENP Entry into the national phase

Ref document number: 3109234

Country of ref document: CA

ENP Entry into the national phase

Ref document number: 2021510084

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019325414

Country of ref document: AU

Date of ref document: 20190823

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2019852803

Country of ref document: EP

Effective date: 20210324