JP2021533940A - Virtual toolkit for radiologists - Google Patents

Abstract

Virtual tools are used to manipulate aspects of the 3D medical image volume. Virtual tools are geo-registered with image volumes. The presentation of the image volume is manipulated by the image processor in response to the use of virtual tools. Virtual tools can be used to assist in the analysis of image volumes. Virtual tools may include virtual focus pens, virtual 3D cursors, virtual transport viewers, virtual pedestals, virtual knives, virtual catheters, virtual load signs, virtual ablation tools, virtual tables, virtual contrast agent tools, and virtual icons.

Description

Aspects of this disclosure generally relate to the display of volume medical images.

Traditionally, computed tomography (CT) and magnetic resonance imaging (MRI) scans are observed by a radiologist on a slice-by-slice approach. Recent advances in diagnostic radiology have provided true 3D observations of medical images through the use of virtual reality, mixed reality or augmented reality headsets, a volume-by-volume approach and an interactive approach. Patent Document 1 which has made it possible to use a volume subtending 3D cursor (incorporated METHOD AND APPARATUS FOR THREE DIMENSIONAL VIEWING OF IMAGES; Patent Document 2 which is incorporated METHOD AND APPARATUS FOR THREE DIMENSIONAL VIEWING OF IMAGES; And Non-Patent Document 1). Interactive volume-containing 3D cursors offer greater potential for more in-depth examination of subvolumes in imaging datasets. Other recent advances include the US Patent Application No. 16 / 195,251, INTERACTIVE VOXEL MANIPULATION IN VOLUMETRIC MEDICAL IMAGING FOR VIRTUAL MOTION, DEFORMABLE TISSUE, AND VIRTUAL RADIOLOGICAL DISSECTION. It provides the ability to perform a variety of organizational operations to improve understanding. In addition, a set of geo-registrated medical imaging tools has been developed to manipulate the volume to improve display (USING GEO-REGISTERED TOOLS TO MANIPULATE THREE-DIMENSIONAL MEDICAL IMAGES, US Patent Application No. 16 / See 524,275). However, the ability of radiologists to interact with volume medical images still has its own limitations.

U.S. Pat. No. 8,384,771 U.S. Pat. No. 9,980,691

Douglas, D. B., Wilke, C. A., Gibson, J. D., Boone, J. M., & Wintermark, M. (2017). Augmented Reality: Advances in Diagnostic Imaging. Multimodal Technologies and Interaction, 1 (4), 29)

All examples, embodiments, and features referred to in this document can be combined in a technically possible manner.

According to some embodiments, the method responds to user input to a selected 3D medical image volume loaded into an image processing system from a set of available virtual tools to a set of virtual tools. It has to select and geo-register each virtual tool in the selected set with a 3D image volume and manipulate the 3D image volume in response to the operation of the virtual tool in the set of virtual tools. Some implementations include a virtual focus pen, a virtual 3D cursor, a virtual transport viewer, a virtual pedestal, a virtual knife, a virtual catheter, a virtual load sign, a virtual ablation tool, a virtual table, a virtual contrast agent tool, and a virtual icon. Have to select a set of virtual tools from the available virtual tools. Some implementations include manipulating the 3D image volume in response to a virtual focus pen by highlighting a portion of the 3D image volume and adding written annotations. Some implementations have to modify the 3D image volume voxels adjacent to the tip of the virtual focus pen. Some implementations have to manipulate the 3D image volume in response to a virtual knife by separating the tissue from the 3D image volume. Some implementations move the 3D image volume within the hollow structure of the 3D image volume and present the image from the perspective of the virtual transport viewer in response to the 3D image volume. Have to operate. Some implementations include performing a virtual colonoscopy with a virtual transport viewer. Some implementations include manipulating the 3D image volume in response to a virtual contrast agent by inserting a visible moving voxel into the 3D image volume. Some implementations have different density values assigned to different voxels in motion. Some implementations can be done by repeating shell-by-shell, by removing voxels in the outer shell of an organ, in response to the operation of virtual tools in a set of virtual tools. Has to manipulate the volume. Some implementations separate 3D image volumes in response to the operation of virtual tools in a set of virtual tools by separating adjacent tissues of interest by adjusting the coordinates of multiple voxels of the tissues of interest. Have to operate. Some implementations have the ability to manipulate a 3D image volume in response to a virtual table by placing the organization of interest in a virtual storage box (storage bin) of the virtual table. Some implementations have the ability to manipulate the 3D image volume in response to a virtual catheter by limiting the movement of the virtual catheter to columnar blood voxels within selected blood vessels. Some implementations have the ability to automatically display information related to a selected subvolume of a 3D image volume. Some implementations include displaying the patient's current state, patient history, test results, and pathological results that were prompted to obtain patient metadata and medical image volume. Some implementations have the use of virtual windshields to display information. Some implementations have a virtual load sign to display the distance to a metric. Some implementations have to display a visual auxiliary icon indicating the viewpoint. Some implementations have the ability to display visual auxiliary icons that indicate findings detected by artificial intelligence algorithms. Some implementations include displaying a three-dimensional medical image volume of the displayed subvolume or a visual auxiliary icon indicating the position of the displayed subvolume with respect to the patient's body. Some implementations have volume-containing 3D cursors to select subvolumes. Some implementations have to select a subvolume from multiple subvolumes in a given list of subvolumes. Some implementations have to display each of the multiple subvolumes of the list sequentially. Some implementations have to select a subvolume from a plurality of subvolumes defined by sequential search pattern coordinates. Some implementations have to select a subvolume from a plurality of subvolumes defined by random search pattern coordinates. In some implementations, manipulating the 3D image volume can change voxel size, change voxel shape, change voxel position, change voxel orientation, change voxel internal parameters. Having at least one of that, creating voxels, and removing voxels. In some implementations, manipulating a 3D image volume has to divide the subvolume of interest into multiple parts based on common characteristics. In some implementations, manipulating a 3D image volume has the ability to generate an exploded view by creating multiple magnified cubes, each contacting a center point. Some implementations have the use of virtual eye tracker symbols to assist the human eye in observing. Some implementations make the virtual eye tracker symbol appear and disappear at spatially separated locations so that the human eye can make jumping movements and jump from one position to another. Have. Some implementations include moving the virtual eye tracker symbol smoothly along the path so that the human eye can perform smooth tracking.

According to some embodiments, the apparatus has an image processing system, which is a set of selected three-dimensional medical image volumes loaded into the image processing system in response to user input. An interface that presents a set of virtual tools for selection from available virtual tools and geo-registers each virtual tool in the selected set with a three-dimensional image volume, and operation of the virtual tools in the set of virtual tools. It has an image processor that operates a three-dimensional image volume in response to. In some implementations, the set of virtual tools includes virtual focus pen, virtual 3D cursor, virtual transport viewer, virtual pedestal, virtual knife, virtual catheter, virtual load sign, virtual ablation tool, virtual table, virtual contrast agent tool, And a set of available virtual tools, including virtual icons. In some implementations, the set of virtual tools includes a virtual focus pen, and the image processor responds to the virtual focus pen by highlighting and annotating a portion of the 3D image volume. Manipulate the 3D image volume. In some implementations, the image processor modifies the 3D image volume voxel adjacent to the tip of the virtual focus pen. In some implementations, the set of virtual tools includes a virtual knife, and the image processor separates or manipulates (eg, repositions) voxels (eg, tissue-type voxels) from a 3D image volume. Manipulate the 3D image volume in response to a virtual knife. In some implementations, the set of virtual tools includes a virtual transport viewer, where the image processor moves the virtual transport viewer within the hollow structure of the 3D image volume and presents the image from the perspective of the virtual transport viewer. By doing so, the 3D image volume is manipulated in response to the virtual transport viewer. In some implementations, a virtual transport viewer is used to perform a virtual colonoscopy through the interface. In some implementations, the set of virtual tools includes a virtual contrast agent, and the image processor manipulates the 3D image volume in response to the virtual contrast agent by inserting a visible moving voxel into the 3D image volume. do. In some implementations, the image processor assigns different density values to different moving voxels. In some implementations, the image processor manipulates the 3D image volume in response to the operation of the virtual tools in the set of virtual tools by removing the voxels of the outer shell of the organ. In some implementations, the image processor is tertiary in response to the operation of a virtual tool in a set of virtual tools by separating adjacent interested tissues by adjusting the coordinates of multiple voxels of the interested tissue. Operate the original image volume. In some implementations, the set of virtual tools includes a virtual table, and the image processor manipulates the three-dimensional image volume in response to the virtual table by placing the organization of interest in the virtual storage box of the virtual table. In some implementations, the set of virtual tools includes a virtual catheter, and the image processor is three-dimensional in response to the virtual catheter by limiting the movement of the virtual catheter to the columnar blood voxels in the selected blood vessel. Manipulate the image volume. In some implementations, the interface automatically displays information related to the selected subvolume of the 3D image volume. In some implementations, the interface displays the current state, patient history, test results, and pathological results for which the patient's metadata and medical image volume were prompted to be acquired. Some implementations have an interface that uses a virtual windshield to display information. Some implementations have the interface display the distance to the metric using a virtual load sign, some implementations have the interface display a visual auxiliary icon indicating the viewpoint. Some implementations have the interface display a visual auxiliary icon indicating the findings detected by the artificial intelligence algorithm. Some implementations have an interface that displays a three-dimensional medical image volume of the displayed subvolume or a visual auxiliary icon indicating the position of the displayed subvolume with respect to the patient's body. Some implementations have an interface that receives a selection of at least one subvolume using a volume containing 3D cursor. In some implementations, the selected subvolume is one of a plurality of subvolumes in a given list. Some implementations have an interface that sequentially displays each of a plurality of subvolumes in a list. In some implementations, the selected subvolume is one of a plurality of subvolumes defined by sequential search pattern coordinates. In some implementations, the selected subvolume is one of a plurality of subvolumes defined by random search pattern coordinates. In some implementations, the image processor manipulating a 3D image volume can change voxel size, change voxel shape, change voxel position, change voxel orientation, voxel internal parameters. Have at least one of modifying, creating voxels, and removing voxels. In some implementations, the image processor manipulates the 3D image volume by dividing the subvolume of interest into multiple parts based on common characteristics. In some implementations, the image processor manipulates the 3D image volume by generating an exploded view by creating multiple magnified cubes, each contacting a center point. In some implementations, the interface has a virtual eye tracker symbol. In some implementations, the virtual eye tracker symbol appears and disappears at spatially separated positions so that the human eye can make jumping movements and jump from one position to another. .. In some implementations, the virtual eye tracker symbol moves smoothly along the path.

In some implementations, methods and devices are used to provide software that implements a virtual toolkit for enhanced medical image analysis, which follows the steps below, ie, volume medical imaging data according to the checklist. Load the set, convert the medical image to a 3D volume, import the virtual toolkit, perform registration and calibration of each virtual tool with / in the geo-registered volume medical image (eg, for example. As described in the incorporated PROCESSING 3D MEDICAL IMAGES TO ENHANCE VISUALIZATION U.S. Patent Application No. 15 / 904,092 and U.S. Patent Application No. 16 / 195,251 and the advanced observation options taught in this disclosure. ) Perform filtering, segmentation and voxel operations and provide a display according to the movement and operation of the virtual toolkit listed in the above steps for all time steps, where decision points are reached and checked. When the list is complete, you can proceed to the next step, if the test is not complete, you go back to the previous step, and finally, when the review of the entire medical set is complete, the test is finished. At the same time, a report is created and filed.

The device has an IO device and an image processor that communicates with the IO device, the image processor having a program stored in a computer-readable non-temporary medium, and the program to a different tissue type. Instructions for performing boxel segmentation, instructions for performing boxel operations, instructions for creating at least one additional boxel and inserting it into the medical imaging data set, and to the medical imaging data set. It has an instruction to remove the boxel, an instruction to perform a boxel annotation on the user-modified radiographic image above, and an instruction to a recording step for review.

In some embodiments, virtual tool operations can also be performed via controller / joystick input. In some implementations, the controller / joystick input can instruct the 3D cursor to change size, shape, or direction (roll, pitch, yaw). In some implementations, the controller / joystick input can instruct the left and right eye viewpoints to zoom inward or away from the 3D cursor. In some implementations, the controller / joystick input can indicate convergence to focus. In some implementations, the controller / joystick input can instruct the 3D cursor to move up or down, or to move the 3D cursor left or right within the head display unit. In some implementations, the controller / joystick input can change the color of the 3D cursor. In some implementations, the controller / joystick input can invoke filtering, segmentation, sequencing, statistical analysis and reporting, which are described in U.S. Patent Application No. 15 / 904,092, which is incorporated herein by reference. do. In some implementations, the controller / joystick input can direct the movement of the virtual focus pen throughout the volume of interest. In some implementations, the controller / joystick / keyboard input can indicate the annotation of one or more 3D cursors in the volume of interest. In some implementations, the controller / joystick input directs icon options related to volume medical imaging to keep the radiologist systematized in his checklist approach for complex examinations. Can be done. In some implementations, presenting a 3D medical image has an improved user controller interface method for medical personnel to review the 3D medical image, consisting of a joystick and a functional button. Functions include, but are not limited to, when interacting with a 3D cursor, including: change the direction of the 3D cursor with roll, pitch, yaw; zoom in on the 3D cursor and from the cursor Zoom out; trigger convergence; move the 3D cursor up and down with respect to what is displayed on the headset; change the size, shape, and color of the 3D cursor; filtering, segmentation, sequencing, statistics, and reporting Invoke processing; Invoke virtual focus pen and its movement control; Annotate one or more 3D cursors in the area of interest; Invoke icon options; and advanced display options of interest (eg, decompose, cut, slice type) View). Typical samples of virtual tools are not limited to the following, but are limited to virtual focus pens, virtual 3D cursors, virtual transport viewers, virtual pedestals, virtual knives, virtual catheters, virtual load signs, virtual ablation, virtual tables, and virtual contrast. Includes agents and virtual icons. Many other virtual tools are possible, including but not limited to drills, cups, strings, mirrors, lenses, metal devices, non-metal devices, and commonly used by healthcare professionals, construction workers or engineers. Includes other tools used by

In some implementations, medical personnel reviewing volume medical images may invoke the process by which the virtual radiologist assistant type icon is displayed. The purpose of the virtual radiologist assistant type icon is, among other things: where the medical personnel performing the test are on the medical institution's checklist and what's next; By drawing all relevant / important information about the triggering patient's metadata and current status; patient's medical history; test results; results from application of artificial intelligence (AI) routines if present and status indicators. be. In this implementation, the reviewing medical personnel may order the display of the virtual windshield (windshield) at any time. In this implementation, the reviewing medical personnel may also modify the items shown on the windshield.

In some implementations, virtual tools can direct voxel changes. Examples of voxel operations include changing voxel size, shape, position, orientation, or internal parameters. In addition, voxels can be created or deleted at the direction of the virtual tool.

In some embodiments, a virtual focus pen is utilized to enhance the visualization of the structure of interest within the medical imaging volume. In some implementations, the focus pen can be used to point to areas within the structure that may contain anomalies. In some implementations, the focus pen can use a symbol (eg, an arrow) to point to the area of interest. In some implementations, notes can be written in the volume data. In some implementations, it is possible to change the transparency of tissue at a certain distance from the tip of the focus pen and highlight voxels close to the point of the focus pen. In some implementations, the focus pen can be used with a 3D cursor.

In some embodiments, a virtual eye tracker symbol is used with a 3D medical imaging dataset to facilitate the observation of the structure by humans. In some embodiments, the virtual eye tracker symbol appears and disappears at spatially separated positions so that the human eye can make jumping movements and jump from one position to another. do. In another embodiment, the virtual eye tracker symbol is continuously visible and has smooth movement along the path so that the human eye can perform smooth tracking. The virtual eye tracker symbol can be controlled by a virtual tool (eg, a virtual focus pen), a georesisted tool, or a pre-programmed sequence, including but not limited to:

In some implementations, the virtual knife can be used to collate with the volume medical image to cut the virtual tissue within the medical imaging volume. In a further implementation, the virtual knife has a movable georegistration point (eg, the tip of the virtual knife), an additional point pointing to the cut plane of the virtual knife, and a control unit with X, Y, Z coordinates. It provides variations in the system, as well as in the roll, pitch, and yaw of the knife. Virtual knives can be used, but not limited to, as follows: A medical practitioner viewing a medical image picks up a virtual knife, moves it to a 3D digital structure of current interest, and then moves the virtual knife. Outside the surface created by the virtual knife as it passes through the 3D georegistered structure and when it passes through the 3D georegistered structure (a virtual knife preselected by the medical practitioner viewing the medical image). Tissues on the side of) can be removed (or set aside). In a further implementation, the virtual knife has a strict registration point (eg, the tip of the geo-registered knife), an additional geo-registration point pointing to the cut surface of the knife. In some implementations, tactile or auditory feedback can be provided to the user.

In some implementations, presenting a 3D medical image has a way to assist in viewing the medical image by achieving a ride in a blood vessel using a visual transport tool. In a further implementation, a virtual catheter may be used with a visual transport tool to optimize the observation of vascular structure within the patient. Examples of procedures include, but are not limited to, the use of intra-tunnel rides in assessing vascular structure and the potential need for insertion of one or more stents. In this case, the vascular entry ride is of interest through the inguinal region to the common femoral artery, to the external iliac artery, to the common iliac artery, to the abdominal aorta, to the thoracic aorta, to the aortic arch, and finally. It can be to the coronary artery. Then, it is visualized so that the medical person who sees the medical image moves in the blood vessel when viewed from the 3D headset. Blood in blood vessels can be digitally removed and virtual light can shine on the walls of blood vessels. The contractions in the vascular structure appear as tunnel stenosis and the X, Y, Z coordinates of the contractions are recorded. Viewing medical images Medical personnel can always view the vascular structure as a whole, along with the current position of the ride within the indicated vessel. In some implementations, voxels can be manipulated by increasing the diameter of the blood vessel so that optimal observation of the internal structure is achieved. In some implementations, presenting a 3D medical image has a method of assisting in observing a medical image consisting of 3D georesisted rides within a blood vessel. In a further implementation, the 3D georesisted ride in this tunnel will be US Patent Application No. 15 / 949,202, SMART OPERATING ROOM EQUIPPED WITH SMART SURGICAL DEVICES, and US Patent Application No. 16 IMPLANTABLE MARKERS TO AID SURGICAL OPERATIONS. Used with patient georegistration as described in / 509,592. Interventors can switch back and forth between a geo-registered 3D system with a 3D head-mounted display and the standard display currently available for intervention surgery. This makes it possible to observe the squeezed contractions identified during preoperative planning in near real time. In addition, warnings can be given in near real time when approaching a critical junction.

In some implementations, presenting a volume medical image has a method of assisting in viewing a medical image composed of a virtual catheter. In a further implementation, the virtual catheter can be used with a 3D digital image of the vascular structure within the patient. In a further implementation, the catheter can continuously calculate the total distance traveled, which can be displayed and time-tagged and recorded for later review. Virtual catheters may be used during preoperative planning of intervention procedures, such as, but not limited to, the treatment of cerebral aneurysms. Implementations of virtual catheters in the treatment of aneurysms can be as follows: for example, from the patient's inguinal to the total femoral artery, then the external iliac artery, then the common iliac artery, then the abdominal aorta, then the thoracic aorta, then the wrist. A virtual catheter is inserted into the 3D digital vascular structure at predetermined locations, such as the artery, then the common carotid artery, then the internal carotid artery, then the middle cerebral artery, and finally the aneurysm. Augmented reality distance markers are added to the 3D virtual catheter for each intersection that the intervener needs to pay attention to, prepared to change from one vessel to another, and all important vascular bifurcation connections. Screen captures of points can be annotated with angular changes from the current path in the coordinate systems XY, XY, and YY planes.

In some implementations, presenting a 3D medical image has a method of assisting in viewing a medical image consisting of decompositions within a 3D digital image. In further implementations, medical personnel viewing medical images (eg, using the segmentation techniques outlined in US Patent Application No. 15 / 904,092) have their common features (eg, equivalent Hansfield units). ), The 3D digital volume of interest can be divided into multiple parts. The medical person viewing the medical image then sees a point in the 3D digital volume that acts as the origin of the decomposition (ideally near the center of the 3D digital volume and between the segmented subvolumes). You can choose. Then there are multiple ways to separate the 3D digital subvolumes as if an explosion had occurred. One of those methods, but not limited to:, creates eight large cubes, each touching the center point and each parallel to the X, Y, Z axes (eg,). , The first cube can be positive in X, positive in Y, positive in Z, the second cube can be positive in X, negative in Y, positive in Z, etc.). Then, the medical person who sees the medical image establishes a distance coefficient for the subvolume near the center point, and establishes a large distance coefficient for the subvolume farther away. Then, based on which cube the central voxel of the subvolume is in, these coefficients are applied to all voxels in each particular subvolume of the 3D digital image (note that the first cube described above). Now, for all subvolumes whose central voxel enters this cube, the X, Y, Z coordinates of the voxels in that subvolume increase by the specified coefficient in the positive X, positive Y, positive Z directions. In the subvolume in the second cube, the increase is in the positive X, negative Y, positive Z directions). Medical personnel viewing medical images change coefficients to change the gap between subvolumes during the course of the examination.

In some implementations, the virtual transport process viewer may be used prior to colonoscopy. Under the virtual transport viewer process, the patient first undergoes a CT scan of the colon, a 3D volume of the colon is created from the CT 2D slices (US Pat. No. 8,384,771), and segmentation (US Patent Application No. 15). / 904/092) identifies the contents of the colon (eg, air, excreta), the subtraction extracts the contents of the colon, and in doing so, the polyp retains its original shape. Will be visible, and a virtual platform will be inserted to allow anterior-posterior-left-right examination within the colon. For ease of 3D observation, the diameter of the colon can be increased from the center point of the enlarged colon to the optimal diameter by voxel manipulation (US Patent Application No. 16 / 195,251). If no polyps were found, the patient could continue to go home with confidence that he was in good health, and the patient would have escaped the discomfort and anxiety of colonoscopy. The cost of colonoscopy is avoided. If this is not the case, it will be known that treatment is needed. Those who escape colonoscopy are unlikely to be reluctant to engage in the virtual transport viewer process and will increase the proportion of cases detected early. This will improve the overall health of all inhabitants.

In some implementations, medical personnel reviewing volume medical images of the colon may call the process by which virtual colonoscopy is performed. In this implementation, a CT scan of the colon with / without contrast is performed. A 3D virtual image is then constructed from the CT 2D slice (US Pat. No. 8,384,771). Segmentation (US Patent Application No. 15 / 904,092) is performed and the tissue outside the colon is removed. Also, non-tissue contents in the colon are removed. The folds that can obscure the polyp are then stretched, thereby'stretching'the colon so that the fold-like colon tissue does not obscure the polyp. This stretching process involves voxel manipulation as described in US Patent Application No. 16 / 195,251. This stretched linear virtual colon is divided in two along the length axis so that the internal structure can be seen through the head display unit as shown in this figure.

In some implementations, presenting a 3D medical image has a method of assisting in viewing a medical image consisting of insertion of a virtual contrast agent into the vascular system. In a further implementation, the virtual contrast agent can be used with a 3D digital image of the vascular structure within the patient. Examples of procedures include, but are not limited to, the search for obstructions such as in pulmonary embolism. In this example, the blood vessels that receive the virtual contrast agent are selected in a time-stepped manner. Insertion of the virtual contrast agent can be performed in a manner and speed as if it were actually placed in a blood vessel. The duration of the time step is under the control of the medical personnel viewing the medical image, and a'freeze frame'is available along with the reproduction of the flow of contrast agent. From the 3D image displayed to the medical personnel viewing the medical image, nearby (possibly partially matching) blood vessels that are not the blood vessels that receive the virtual contrast agent can be extracted.

In some implementations, presenting a 3D medical image has a method of assisting in viewing the medical image consisting of ablation techniques. In further implementations, ablation techniques can be used with 3D digital structures transported by 3D cursors as described above. The method underlying ablation technology is not limited to: (Using the segmentation technique outlined in 15 / 904,092); removing one voxel depth in order from all voxels on the outer surface and viewing the medical image The remaining outer layer of tissue in the direction of the medical practitioner. Repeat this step several times above; or select one layer in the X, Y, Z coordinate system (eg, select the XY layer with the largest Z coordinate and remove that layer for medical use. Repeat this step several times on the remaining 3D digital volume in the direction of the medical personnel viewing the image). This process can be repeated many times, leaving a slightly smaller volume each time the outer shell is removed. The process is then repeated again, removing new, smaller shells once more, leaving smaller volumes, and continuing.

In some implementations, virtual tools are used to determine the location of dense interest tissues, the separation distance of those organizations, voxel manipulation (eg, insertion of additional voxels of various transparency), and of interest organizations. It can be changed so that it can be increased through simultaneous adjustment of coordinates.

In some embodiments, a subvolume-by-subvolume observation approach is enabled. Subvolumes can consist of various numbers or combinations of voxels.

In some implementations, presenting a 3D medical image has a way to sequence the movement of the 3D cursor in the volume of interest. The sequence is not limited to the following, but starts by, for example, placing the 3D cursor at the 0_X, 0_Y, 0_Z coordinates, increasing X so as to move the 3D cursor in the X direction, and continuing to increase up to the maximum value of X. Then, X can be reduced back to the 0X coordinate to increase Y. This continues in the XY plane until it is complete, then Z is incremented and this process continues until it is complete. The increment continues until the entire volume of interest has been reviewed. Changing from one increase to another is controlled by the medical practitioner reviewing the medical image. In addition, if suspicious tissue is detected during a review of the contents of the 3D cursor, medical personnel may annotate the 3D cursor for further review. In addition, the medical practitioner may choose to display the location of the 3D cursor within the volume of interest and the volume of interest that has already been examined at any time. If suspicious tissue appears in more than one 3D cursor, the whole of those 3D cursors can be displayed at the same time. In some implementations, medical personnel reviewing 3D virtual medical images may recall certain types of search patterns. In some implementations, sequential search patterns may be selected. An example of a sequential search is, but is not limited to, a virtual windshield wiper search. This type of search pattern helps ensure that a thorough search is done. In some implementations, it is a random pattern based on examining items that may be of interest. Image processing aids, such as changing the transparency of organs and using pseudocolors, can help identify items of interest (US Pat. No. 8,384,771). This type of search pattern can facilitate the review process. Both types of search patterns use 3D cursors (METHOD AND APPARATUS FOR THREE DIMENSIONAL VIEWING OF IMAGES US Patent Nos. 9,980,691 and INTERACTIVE 3D CURSOR FOR USE IN MEDICAL IMAGING US Patent Application No. 15 / 878,463). Note that when reviewing medical images in the original 2D format, the eye may jump from one spot to another according to the jumping movement of the individual being reviewed, and a significant portion of the slice may not be observed. As a result, small chunks may be missed. In the use of 3D cursors, those small chunks extend to the larger portion of the presented image, increasing the probability of detection proportionally.

In some implementations, medical personnel reviewing volume medical images have a step-by-step process of selecting subvolumes (of the entire volume) wrapped in a 3D cursor (US Pat. No. 9). , 980, 691 and US Patent Application No. 15 / 878,463) may be used to invoke the process by which volume medical images are examined. The contents of each 3D cursor are reviewed independently. In addition, questions may arise as to whether the entire volume was inspected after this step-by-step process was completed. In this implementation, the volumes contained in each of the inspected 3D cursors are summed and removed from the original total volume. This can result in some of the original volume being overlooked. In this implementation, those missed parts may be highlighted to the reviewing healthcare professional and warned to continue reviewing and examining the missed parts. Under this implementation, the detection rate of small chunks will be increased compared to the 2D slice inspection process. Under this implementation, more thorough inspection would have been done. In some implementations, presenting a 3D medical image has a way to sequence the movement of the 3D cursor within the volume of interest. Sequencing is not limited to the following, but starts with, for example, placing the 3D cursor at the 0_X, 0_Y, 0_Z coordinates, increasing X to move the 3D cursor in the X direction, and increasing it to the maximum value of X. Then, X can be reduced back to the 0X coordinate to increase Y. This continues in the XY plane until it is complete, then Z is incremented and this process continues until it is complete. The increment continues until the entire volume of interest has been reviewed. Changing from one increase to another is controlled by the medical practitioner reviewing the medical image. In addition, if suspicious tissue is detected during a review of the contents of the 3D cursor, medical personnel may annotate the 3D cursor for further review. In addition, the medical practitioner may choose to display the location of the 3D cursor within the volume of interest and the volume of interest that has already been examined at any time. If suspicious tissue appears in more than one 3D cursor, the whole of those 3D cursors can be displayed at the same time.

In some implementations, virtual icons may be used with the display of the medical imaging volume to facilitate location. As an example, during a scrutiny of a small area of an organ (eg, the liver), one may lose some orientation as to exactly where the 3D cursor is located within the organ. Therefore, the icon may always assist in positioning while observing the inspection. In another embodiment, the arrow annotation marker can indicate the path from the original point of view to the current point of view. In an alternative embodiment, automatic recentering techniques may be used to quickly adapt the user.

In some implementations, a virtual table may be attached to a toolkit that has a virtual storage box on the table. (One or more) subvolumes of virtual medical images containing tissues of concern / interest that are currently being examined may be placed in (one or more) virtual storage boxes. Subvolume boxes (bins) may correspond to medical institution checklist items. In some implementations, emergency boxes may be added that can be accessed by both the medical and therapeutic personnel to be reviewed, thereby supporting and facilitating cooperation between these parties. In some implementations, reporting can be facilitated by automatically sequencing through boxes, extracting items, and adding those items to the report. The added items (eg, annotated figures containing the organization in question) enhance both the quality and completeness of the report. Whenever a radiologist discovers something unusual, he puts it in the virtual table of the report. Items placed on the virtual table of the report may include 2D slices or 3D volumes containing anomalous findings. The radiologist has the option to determine the size of the virtual table and virtual box. For collaborative work, one radiologist may pass one item to another radiologist's table or box.

In some implementations, the radiology report may include images processed by virtual tools.

This patent file or application file contains at least one color-created figure. A copy of this patent publication or patent application publication with a (one or more) color figures will be provided by the Agency upon request and payment of the required fees.
It shows a flow chart for the use of virtual tools to optimize the display of medical imaging tests. The device which implements the process of FIG. 1 is shown. Shows virtual toolkit options for observing volume medical images. An example of a flow diagram and a head-up display type icon that delivers relevant information to the radiologist during the interpretation of the imaging test is shown. Head-up display refers to a virtual windshield (windshield) (similar to a'head-up'display in an aircraft), which can be called at any time during the inspection process and on a single virtual windshield, patient. Display items that are relevant to the entire case and / or specific items that are examined in the medical institution's checklist. Instead, it can be referenced as a virtual radiation assistant type icon. Shows a flow diagram and explanation of virtual toolkit input that leads to voxel changes. It shows voxel operations based on interaction with virtual tools. Shows a virtual eye tracker symbol with a variable display pattern. Demonstrates a virtual knife that can be used by medical personnel to'separate'tissue from an existing 3D medical imaging volume to facilitate observation of internal structures. It shows a virtual ride in a virtual blood vessel using a visual transport tool. It shows a virtual catheter that can be used with a volume medical image of the vascular structure in a patient with the help of a virtual icon. It shows the general concept of 3D medical images and technical examples behind the decomposition of 3D medical images into multiple separate organs. Demonstrates the use of a virtual transport viewer for more accurate virtual colonography reviews. A portion of a virtual 3D volume medical image containing the colonic portion of the large intestine stretched into one long linear tube through voxel manipulation is shown. The contents in the tube are then segmented and removed from the tube. Finally, the tube is split along the length axis and opened to allow observation of the internal colonic structure. It shows the insertion of a virtual contrast agent and its flow in the vascular system. In particular, it shows ablation techniques that can be used with 3D digital structures within 3D cursors. This allows for careful examination of the internal structure and elements of the organ. It shows a virtual focus pen that guides voxel operations. The total imaging volume and multiple types of subvolumes are shown. It shows the sequence of movement of the 3D cursor in the volume of interest in a random pattern. An example of an orderly pattern for observing a medical image (eg, a continuous virtual windshield wiper type pattern) is shown. It shows the volume of interest to be reviewed and the process by which the missed review intent area can be highlighted to the reviewing healthcare professional. These identified subvolumes are subsequently reviewed, thereby ensuring the integrity of the review. It shows a human icon that includes the position of the 3D virtual cursor at the approximate position inside the body. Shows a virtual movable table for storing virtual images of suspicious organizations stored by the checklist category. A sample radiology report containing images processed by a virtual tool is shown.

Some aspects, features, and implementations described herein may include machines such as, for example, computers, electronic components, radiological components, optical components, and processes such as, for example, computer-mounted steps. As will be apparent to those of skill in the art, computer-implemented steps can be stored on non-transient computer-readable media as computer-executable instructions. Also, as understood by those skilled in the art, computer executable instructions can be executed on a variety of tangible processor devices. For ease of description, not all steps, devices or components that can be part of a computer or data storage system are listed here. Those skilled in the art will recognize such steps, devices and components in light of the teachings of this disclosure and the knowledge generally available to those of skill in the art. Therefore, the corresponding machines and processes are possible and within the scope of disclosure.

FIG. 1 shows a flow chart relating to the use of georegistered tools to optimize the display of medical imaging examinations. In step A 100, the volume medical imaging dataset is loaded according to the checklist, the medical image is converted to a 3D volume, and the virtual toolkit is imported. At step B 102, registration and calibration of each virtual tool with / in the geo-registered volume medical image is performed. Filtering, segmentation and according to the incorporated US Patent Application No. 15 / 904,092 and US Patent Application No. 16 / 195,251 and the advanced observation options described in this disclosure at step C 104. Perform a voxel operation. In step D 106, all time steps are provided with a display according to the movement and operation of the virtual toolkit listed in the above step. At step E 108, the user must answer the question "Is this element of the checklist checked?" If the answer is no 110, the user should proceed to step D 106. If the answer is yes 114, the radiologist should proceed to step F 116 and review the next set of medical images according to the checklist. The radiologist should then proceed to step G 118 to answer the question "Is the review complete?" If the answer is no 120, the radiologist should proceed to step A 100 (122). If the answer is yes 124, the radiologist should end (126).

FIG. 2 shows an apparatus that implements the process shown in FIG. Using a radiological imaging system 200 (eg, X-ray, ultrasound, CT (Computed Tomography), PET (Positron Emission Tomography), or MRI (Magnetic Resonance Imaging)) of the anatomical structure 204 of interest. A 2D medical image 202 is generated. The 2D medical image 202 is provided to the image processor 206, which includes a processor 208 (eg, CPU and GPU), a volatile memory 210 (eg, RAM), and a non-volatile storage 212 (eg, HDD and SSD). Program 214 running on the image processor implements one or more of the steps shown in FIG. A 3D medical image is generated from the 2D medical image and displayed on the IO device 216. The IO device 216 includes either a virtual reality headset, a mixed reality headset, an augmented reality headset, a monitor, a tablet computer, a PDA (mobile information terminal), a mobile phone, or a wide variety of devices, either alone or in combination. obtain. The IO device 216 may include a touch screen and also accepts input from an external device (represented by 218), such as a keyboard, mouse, and any of the various devices for receiving various inputs. You may. However, some or all inputs may be automated, for example by program 214. Finally, as further described in FIG. 3 and the rest of this patent, a series of virtual tools 220 are implemented that help medical personnel observe medical images.

FIG. 3 shows virtual toolkit options for observing volume medical images. This figure shows a representative example of the display options available through the use of virtual tools. It presents an option on the display that allows the virtual tool to be guided / selected and the user clicks on the desired option. At the center of this figure, the virtual tool 300 (ie, the virtual focus pen) is geo-registered within the medical imaging volume. The virtual focus pen is superimposed in the area containing the virtual 3D medical image 302 located inside the 3D cursor 304. Buttons (eg on the keyboard) and virtual tool movements can be combined together to determine the size of the 3D cursor 304 (eg, select the center of the 3D cursor 304 and then correspond to the radius. Move the virtual focus pen 300 a certain distance). The user views a virtual tool using a headset 306 (eg, augmented reality, mixed reality or virtual reality glasses) with a left eye display 308 and a right eye display 310. The virtual focus pen can be registered in the virtual image by touching a specific spot (eg, a corner) of the medical imaging volume 302. For display purposes, the medical practitioner may magnify the tip of the focus pen as desired, to show only the tip of the focus pen in the display, and / or display a virtual image of the focus pen in its entirety. Can be selected to be oriented within the volume. The movement of the virtual focus pen 300 is controlled by a medical person viewing a medical image. The virtual focus pen 300 is useful when smooth tracking eye movements are required. For example, smooth tracking eye movements are needed when examining an artery for some obstruction, and a virtual focus pen is used to trace along the artery to look for the obstruction. Jumping eye movements can result in jumping over a portion of an artery and undetecting a serious occlusion, and therefore the virtual focus pen 300 can help assist in this search pattern.
Multiple color / shape virtual focus pens 300 may be used to trace different flows of arteries and veins. In the image above, the position and orientation of the virtual tool changes with respect to the volume of interest. The virtual focus pen is indicated by the initial position and orientation 312 with respect to the volume of interest 314. The user can then move the virtual focus pen to a subsequent position and orientation 316 with respect to the volume of interest 314. Proceeding clockwise, the virtual focus pen 318 then grabs the volume of interest 320 at an initial distance from the head-mounted display unit 322. The virtual focus pen 324 is then pulled so that the region of interest 326 is closer to the head display unit 322 for improved visualization. Alternatively, the volume of interest 320 may be moved to another position or orientation by the focus pen 318. Virtual dots can then be fixedly or dynamically placed on or next to a portion of the virtual image 330 being inspected (eg, the carotid artery). For example, dots can appear and disappear at multiple spots along the vascular structure to assist in observation by jumping movements in which the eye jumps a short distance to see the most important parts of the vascular structure. .. At time point # 1, the first virtual dot 332 appears, and at this point no other virtual dots are shown in the field of view. At time point # 2, a second virtual dot 334 appears, and at this point no other virtual dots are shown in the field of view. At time point # 3, a third virtual dot 336 appears, and at this point no other virtual dots are shown in the field of view. At time point # 4, a fourth virtual dot 338 appears, at which point no other virtual dots are shown in the field of view. Alternatively, the virtual dot 342 may move along a portion of the virtual image 340 (eg, carotid artery) to help the human eye perform smooth tracking and enhanced observation of vascular structure. .. Next, the virtual focus pen 344 is used to perform convergence to the focus 346. The left eye viewpoint 348 is shown. A straight line 350 indicating the viewing angle of the left eye is also shown. The right eye viewpoint 352 is shown. A straight line 354 showing the viewing angle of the right eye is also shown. The view angle 350 from the left eye viewpoint 348 and the view angle 354 from the right eye viewpoint 352 intersect at the convergence point 346. A virtual incision is then made by using a virtual knife 356 to cut the aorta 358 and the pulmonary artery 360 away from the rest of the heart 364. The cut surface 362 is shown. Next, a virtual catheter 366 is placed through the aorta 368 within the medical imaging volume. A virtual road sign (road sign) 370 for guiding medical personnel is shown. A focus pen 372 is shown. The dotted blue line 374 is the desired catheter trajectory, which can be at different time settings. The virtual catheter 366 can be pulled through the vascular system. Intravascular ride type indication 376 is shown with the desired route highlighted by a dotted red circle 378. The last three examples show the advanced display options enabled by virtual tools. Degradation type indications are shown in which the organs are separated, where various organs are separated. For example, the amount of spacing between various abdominal organs including the aorta 380, left kidney 382, pancreas 384 and liver 386 is increased. Next, virtual ablation is performed to sequentially remove the outer shell 390 of the virtual tissue over a plurality of time points. The anatomical structure on which the virtual ablation is performed can be placed within the 3D cursor 388 to help direct the ablation. Finally, structures such as the colon 392 can be sliced and opened (eg, using a virtual knife) so that the medial aspect containing the polyp 394 inside the hollow organ can be examined more carefully. A voxel operation is required to achieve this aspect.

FIG. 4 shows an example of a flow diagram and a head-up display type icon that delivers relevant information to the radiologist during the interpretation of the imaging examination. First, a flow diagram showing how the virtual windshield 406 can be used to assist in the interpretation of the radiographic image will be described. The first step 400 is for the radiologist (or other healthcare professional) to move the checklist items on the report. In the second step 402, a virtual assistant type icon (also known as a virtual windshield 406) displays relevant information. The third step 404 is for the radiologist to review the virtual assistant type icon display, review the image, and enter that section of the report. Head-up display refers to a virtual windshield 406 (similar to a'head-up'display in an aircraft), which can be called at any time during the inspection process and on a single virtual windshield for the entire patient case. Display items related to and / or specific items that are examined in the medical institution's checklist. In the process of reviewing a virtual volume medical image, calling the virtual windshield may assist the medical practitioner in conducting the review. Questions related to the review are: What's next on the checklist; What triggered the acquisition of medical images; Is there a medical history of the current state; What are the test results; Artificial It may include what the results and enumeration were, if the intelligence routine was applied. This figure shows an example of a virtual windshield 406. In the checklist example, some items have been examined and others remain. Age, gender, current status, and medical history associated with the current status. Results and status indicators from the application of AI routines, if any. Pathology, imaging and examination result data, if any, are also shown. Other items may also be of interest to the person performing the inspection, and the one shown in this figure is an example. Also, having all the relevant information on the virtual front glass 406 means that the patient does not have to go from the radiology PACS system to the electronic medical reporting system, all the important information is there and at the command of the inspector. Keep in mind that it can save medical personnel time as it can be displayed at any time.

FIG. 5 shows a flow diagram and description of virtual toolkit input leading to voxel changes. Voxels can be manipulated by size, shape, position, orientation, or internal parameters. In addition, voxels can be created or deleted in the direction of virtual tools. The original voxel 500 is shown. The size of the original voxel 500 can be reduced to produce a smaller voxel 502. The size of the original voxel 500 can be increased to produce a larger voxel 504. Modify the original cube-shaped voxel 500 so that each of the eight smaller voxels 506 has one-eighth of the volume of the original voxel 500 so that eight smaller voxels 506 are generated. be able to. The original voxel 500 can also be removed 508. The internal data unit (eg, grayscale value) of the original voxel 500 can be modified 510. Additional internal data units (eg, textures, tissue type characteristics, etc.) can be added 512. The original voxel orientation can be changed 514. The position of the original voxel 516 can be shifted (ie, the voxel is moved), which shifts the x, y, z coordinates 518 of the original voxel 516 to the new x, y of the shifted voxel 520. , Z coordinate 522, which can be performed by making it traveled by specific x-distance 524, y-distance 526 and z-distance 528. The shape of the original voxel 500 can be changed, for example, from a cube to an octahedron 530 or a cylinder 532.

FIG. 6 shows voxel operations based on interaction with virtual tools. FIG. A shows a 3D cursor 600 including a volume of interest 602. Note that this volume of interest is a uniform middle gray. Also note that the tip 606 of the virtual tool (ie, the virtual focus pen in this case) 604 is located outside the volume of interest 602. FIG. B shows a 3D cursor 608 containing a volume of interest 610 with changes in the position and orientation of the virtual tool (ie, focus pen in this case) 612, where the virtual tool contains the tip 614 of the virtual tool. Is contained in both the virtual 3D cursor 608 and the volume of interest (eg, including tissue selected from the volume medical image) 610. Note that multiple voxels 616 close to the tip 614 of the virtual tool 612 have been changed / highlighted in light gray. It should also be noted that the transparency of the tissue 610 within the 3D cursor 608 has been altered to better visualize the tissue 616 highlighted by the virtual focus pen 612 and the virtual focus pen 612 itself. FIG. C shows another change in the position and orientation of the virtual focus pen 618, as well as the corresponding change in the visual appearance of the nearby voxel 620. The 3D cursor 622 containing the volume of interest 624 is accompanied by further changes (compared to FIG. 6B) in the position and orientation of the virtual tool 618 (ie, the focus pen in this case) 618, in this case the virtual tool. A portion of the virtual tool 618, including the tip 626, is contained in both the virtual 3D cursor 622 and the volume of interest (eg, including tissue selected from the volume medical image) 624. Note that multiple voxels 620 close to the tip 626 of the virtual tool 618 have been changed / highlighted in light gray. Also, the transparency of the tissue 624 within the 3D cursor 622 has changed (compared to FIG. 6A) to better visualize the tissue 620 highlighted by the virtual focus pen 618 and the virtual focus pen 618 itself. Please also note that there is. This serves to assist the radiologist in the exact location of the virtual tool within the volume of interest.

FIG. 7 shows a virtual eye tracker symbol with a variable display pattern. The human eye can perform jumping movements to quickly switch from a fixed object to a fixed object. FIG. 7A shows a carotid bifurcation 700 with a virtual eye tracker symbol 701 (eg, a blue dot) at multiple locations over multiple time points. At the first time point 702, the virtual eye tracker symbol 701 is located below the common carotid 704 portion of the carotid bifurcation 700. At a second time point 706, the virtual eye tracker symbol 701 is located at the carotid valve 708 portion of the carotid bifurcation 700. At a third time point 710, the virtual eye tracker symbol 701 is located in the middle of the internal carotid 712 portion of the carotid bifurcation 700. At the fourth time point 714, the virtual eye tracker symbol 701 is located at the external carotid artery 716 portion of the carotid bifurcation 700. This assists the movement of the human eye so that the eye can jump to each of the new positions when detecting the movement of a new spot. Such a system may be combined with a line-of-sight tracking system to ensure that a person has seen the line-of-sight tracking symbol. The user can control the disappearance of one virtual eye tracker symbol and the appearance of another virtual eye tracker symbol by the IO device. FIG. 7B shows a virtual eye tracker symbol 718 located below the common carotid 720 portion of the carotid bifurcation 700. Over a number of time steps, the virtual eye tracker dot 718 moves smoothly down to the internal carotid artery, and at time point #N, the virtual tracker dot 718 reaches its final destination, for example, the middle part of the internal carotid artery 722. .. For example, the user may choose a frame rate of 60 frames per second and a movement rate of 2 cm / sec. The distance traveled by the virtual tracker dot determines the total time for a particular segment. The high frame rate facilitates smooth gaze tracking of the human eye and potential avoidance of areas jumped by the user, leading to a more comprehensive observation. The virtual eye tracer dot 718 can take many shapes, sizes, colors, motion velocities, and the like.

FIG. 8 shows a virtual knife that can be used by medical personnel to'separate tissue'from an existing 3D medical imaging volume to enable observation facilitation of internal structures. In this example, a virtual knife is used to examine the patient's heart. This task is performed in cooperation with a 3D cursor that wraps around the 3D medical volume of the heart. FIG. 8A shows the virtual knife 800, along with the virtual cut surface 802 and associated (one or more) registration points 804. The medical person viewing the medical image can pick up the virtual knife 800 and move it to the volume of interest shown as the heart 806, where the tissue outside the heart has been removed and wrapped in the 3D cursor 808. FIG. 8B shows passing a knife 800 with a cut surface 802 and a registration point 804 through a 3D volume of interest 806 so that a portion of tissue 810 (ie, aorta and pulmonary artery) is cut and displaced. .. FIG. 8C shows the removal of the aorta and pulmonary artery to allow medical personnel to look into the aortic valve 812 and the pulmonary valve 814. Further isolation may allow investigation of the tricuspid valve (not shown). Finally, you can observe 4D datasets in collaboration with a virtual toolkit that provides a more advanced observation of the heart.

FIG. 9 shows a virtual ride in a blood vessel using a visual transport tool. A virtual transport tool is a means / path by which healthcare personnel can move within a hollow structure and visualize the state within it. The virtual transport tool can be used alone or with a virtual catheter. Virtual transport tools can be used with virtual catheters to generate 3D digital images of the interior of vascular structures within a patient, or to train interveners to treat vascular conditions. Virtual transport tools provide a perspective of what is anterior within a blood vessel. In this example, blood vessels are shown. Blood in the vessel is digitally removed, virtual light can shine on the vessel wall anterior by a distance from the current field of view in the tunnel, and the X, Y, Z coordinates of the contractile are recorded. To. Blood is removed from the blood vessels during the segmentation process, but the vascular structure remains. This allows virtual transport tools to visualize the internal structure of blood vessels without obscuring them. Typically, the virtual transport tool will be in the center of the blood vessel and will look forward. The center of the viewpoint is within the center of the blood vessel, but the actual appearance is offset according to the appearance of the left eye and the right eye. A medical person viewing a medical image can visualize it as moving through a blood vessel when viewed from a 3D headset (eg, augmented reality). It is possible to increase the diameter of the virtual blood vessel so as to improve the display according to the voxel operation (US Patent Application No. 16 / 195,251). For example, it can be said that it is difficult to see inside a small pipe, and it is also difficult to see all parts of a large tunnel from within it at the same time, so it is a great deal to be able to adjust the size of the tunnel. Provides flexibility in observation. If the user identifies an abnormal condition, the user can take different positions and orientations to examine the condition. The distance of the displayed blood vessels is selected by the medical personnel, and the lighting intensity of the structure is also selected by the medical personnel. A common clinical application expected by the use of these techniques is the measurement of carotid atherosclerotic plaques (eg, measuring the lumen and length of the stenosis in the stenotic region, which is, for example, North American symptomatic). It may be known to be a metric that determines the type and placement of the stent, which is better than current methods such as carotid endarterectomy test (NASCET) measurement techniques). As an example, a small volume in the lumen over a particular length is a good indicator of pathology and intervention compared to current methods. Rolling calculations of the lumen of each vessel are performed using the metrics provided to medical professionals. For example, this ride can be used in assessing vascular structure and the potential need for stenting. At any time, the medical practitioner looking at the medical image can see the vascular structure as a whole by indicating the current ride position in the blood vessel with an icon. FIG. A shows a normal blood vessel inner surface 900 with no plaque and blood scavenged. This largest circle 900 represents the inner surface of the blood vessel at the current observation position. Tissue 901 on the surface of the inner mucosa is shown. The medium-sized dotted circle 902 represents the inner surface of the blood vessel at a position intermediate distance from the current observation position, for example 5 cm from the current observation position. The smallest dotted circle 904 represents the farthest distance the user can see from the current observation position. A virtual marker, for example a double-headed arrow 906, may indicate a length within a blood vessel that is actively visible, for example, 10 cm from the current position 900 of the blood vessel to the farthest visible position 904 in the blood vessel. The virtual road sign 908 has a distance to an important intersection, for example "30.0 cm to the brachiocephalic artery". FIG. B shows vascular lumen stenosis due to atherosclerotic plaques and road signs that provide a description of the measurement results. This largest circle 910 represents the inner surface of the blood vessel at the current observation position. Tissue 911 on the surface of the inner mucosa is shown. The medium-sized dotted circle 912 represents the inner surface of the blood vessel at an intermediate distance from the current observation position, for example 5 cm from the current observation position. The portion 916 of the intermediate circle 912 at the 2 o'clock position is shown to protrude inward. Both the round portion of the intermediate circle 912 and the portion 916 of the intermediate circle protruding inward are located 5 cm from the current observation position. Therefore, the entire dotted line, including 912 and 916, is located 5 cm from the current observation position, and thus represents an "equidistant line". The smallest dotted circle 914 represents the farthest distance the user can see from the current observation position, for example 10 cm from the current observation position. A virtual marker, such as the large double-headed arrow 918, may indicate the length within the actively visible blood vessel, for example, 10 cm from the current position 910 of the blood vessel to the farthest visible position 914 within the blood vessel. A small double-headed arrow 920 is shown from the expected dotted line position (which defines a boundary 5 cm away assuming no plaque / stenosis) to the actual position 916 located 5 cm further inward. It should be noted that when the radius of a particular "equidistant line" becomes smaller, it points to the area of stenosis. It should be noted that when the radius of a particular "isodense line" increases, it points to the area of swelling / dilatation / aneurysm. Also, to mention, another virtual road sign 920 states "5.0 cm away from 30% atherosclerotic stenosis centered at 2 o'clock". The clock system here is an example of how the location of a stenosis can be described. FIG. C shows a visual transport tool approaching the bifurcation and confluence of three blood vessels. In pre-planning, the medical professional can choose which of these vessels the catheter should be placed in, and that vessel may be highlighted in pseudo-color to verify the correct path of the catheter. The largest circle 922 represents the inner surface of the blood vessel at the current observation position. Tissue 923 of the inner mucosal surface is shown. The medium-sized semicircle 924 at the 3 o'clock position represents a branching vessel (eg, the internal carotid artery) that is the desired option to enter. The medium-sized semicircle 926 at 9 o'clock represents a further branched vessel (eg, the external carotid artery) that is a second option to enter but is not desired in this scenario example. .. The red dotted line 928 is shown as an example of a virtual tool displayed on the image as a visual clue to help medical personnel know which branch they want to enter. A virtual load sign 930 is shown, which states, "5.0 cm away from the carotid bifurcation. Proceed towards 3 o'clock to enter the internal carotid artery."

FIG. 10 shows a virtual catheter that can be used with a volume medical image of the vascular structure within a patient with the help of a virtual icon. For example, 3D virtual catheters can be used in preoperative planning of interventional surgery, such as obtaining measurements of significant distances or angles. In the intervention procedure shown in this figure, a 3D virtual catheter is used to treat an aneurysm. FIG. A shows the blue solid line 1000 as a catheter located in the right inguinal region, entering at the common femoral artery and extending into the right external iliac artery and into the aorta. The tip 1002 of the catheter is a small black circle. The crossed route is shown as a solid line, and the planned route is shown by a broken line 1004. The planning route can be accomplished through the placement of location markers within the particular blood vessel that the medical practitioner is following with the target. Such location markers can be located at intermediate points along the desired pathway, or at the final target lesion 1006 (eg, cerebral aneurysm). After these position markers have been placed, the path connecting the markers can be marked (eg, a blue dotted line). Then, the measurement can be performed along the blue dotted line. The measurement results can be used to display a load sign 1008 to inform healthcare professionals of the distance to the vascular bifurcation connection that should be used in the actual interventional procedure. Note that virtual icons 1010, such as 2D or 3D objects, are also shown. FIG. B shows a virtual catheter 1012 extending into the thoracic aorta. As shown, the dashed line 1016 represents the desired path of the catheter, which passes through the brachiocephalic artery, then the common carotid artery, then the internal carotid artery, then the middle cerebral artery, and finally to the aneurysm 1018. It's in. Road signs can be displayed to inform healthcare professionals of the distance to the vascular bifurcation connection that should be used in the actual interventional procedure. An augmented reality distance marker is added to the 3D virtual catheter for each intersection that the intervener needs to pay attention to, preparing to change from one vessel to another. Screen captures of all critical vascular bifurcation points can be annotated with angular changes from the current path in the coordinate systems XY, XY, and YY planes. FIG. C shows an enlarged view of the vascular bifurcation junction, where the vascular bifurcation junction has multiple pathway options, and medical personnel are careful in moving the catheter to the correct vessel. There must be. The descending thoracic artery 1022, the brachiocephalic artery 1024, the left common carotid artery 1026, the left subclavian artery 1028, and the ascending thoracic aorta 1030 are shown. A virtual catheter 1032 is shown. The tip 1034 of the virtual catheter is shown. The blue dotted line 1036 represents the desired catheter route.

FIG. 11 shows the general concept of 3D medical images and technical examples behind the decomposition of 3D medical images into multiple separate organs. Medical personnel viewing medical images (eg, using the segmentation techniques outlined in US Patent Application No. 15 / 904,092) have their common features (eg, equivalent Hansfield units, anatomy). The 3D digital volume of interest can be divided into multiple parts based on the target atlas, etc.). This figure shows the general process in which it is desirable to examine key organs individually. Such a process may be constructed, for example, according to the items on the image review checklist. FIG. A shows a generalized explanatory view of the organs in the abdomen. Shown are liver 1100, right adrenal gland 1102, right kidney 1104, inferior vena cava 1106, right iliac vein 1108, spleen 1110, aorta 1112, pancreas 1114, left adrenal gland 1116, left kidney 1118, gastrointestinal tract 1120 and left iliac artery 1122. Has been done. The process is to deploy these organs outward from a point at approximately the center of the torso in the X, Y, Z directions to facilitate individual examinations without visual interference from adjacent organs. .. FIG. B shows the organ after the segmentation has been applied and a dashed line is drawn around the organ to show the segmentation process. The liver 1124, right adrenal kidney 1126, right kidney 1128, inferior vena cava 1130, right iliac vein 1132, spleen 1134, aorta 1136, pancreas 1138, left adrenal kidney 1140, left kidney 1142, gastrointestinal tract 1144, and left iliac artery 1146. It is shown. Note that the dashed line is shown to better indicate segmentation. FIG. C shows an exploded view. The coordinates of the organ (X, Y, Z) are changed to the new positions indicated by the dashed lines. The software for implementing this concept is, but is not limited to, the following procedure. A point within a 3D digital volume (ideally near the center of the 3D digital volume and between segmented subvolumes) where the medical personnel viewing the medical image will act as the origin of the decomposition. Can be selected. The liver 1148, right adrenal gland 1150, right kidney 1152, inferior vena cava and iliac vein 1154, pancreas 1156, gastrointestinal tract 1158, spleen 1160, left adrenal gland 1162, left adrenal gland 1164, aorta and iliac artery 1166 are shown. FIG. D shows one of several ways in which 3D digital subvolumes can be separated as if an explosion had occurred. One of those methods is, but is not limited to,: 8 large cubes 1168, each touching the center point and each parallel to the X, Y, Z axes ( For example, the first cube can be positive in X, positive in Y, positive in Z, the second cube can be positive in X, negative in Y, positive in Z, and so on). Then, the medical person who sees the medical image establishes a distance coefficient for the subvolume near the center point, and establishes a large distance coefficient for the subvolume farther away. Then, based on which cube the central voxel of the subvolume is in, these coefficients are applied to all voxels in each particular subvolume of the 3D digital image (note that the first cube described above). Now, for all subvolumes whose central voxel enters this cube, the X, Y, Z coordinates of the voxels in that subvolume increase by the specified coefficient in the positive X, positive Y, positive Z directions. In the subvolume in the second cube, the increase is in the positive X, negative Y, positive Z directions). Medical personnel viewing medical images change coefficients to change the gap between subvolumes during the course of the examination. For example, a medium gap of 1170 is shown. As an alternative, a larger gap of 1172 is shown.

FIG. 12 illustrates the use of a virtual transport viewer for more accurate virtual colonography reviews. The general public tends to avoid undergoing colonoscopy due to unpleasant preparation (eg, drinking large amounts of fluid) and anxious periods during the procedure. One alternative is to undergo virtual colonography, in which a CT scan is performed and the inner mucosal surface of the colon is reviewed. If no polyps are found, no treatment step is required. However, if a polyp is found, the preparatory stage is repeated at some later date, and a therapeutic stage (ie, colonoscopy) is performed to remove the polyp. In this figure, a virtual transport viewer is used to look inside the colon to determine if polyps are present. If nothing is present, everything is fine and no preparation for colonoscopy is required. If polyps are present, they can be detected by a virtual transport viewer, after which the necessary preparation and subsequent treatment can proceed. Under the virtual transport viewer process, the patient will follow the process of receiving the following: First, a CT scan of the colon is taken and a 3D volume of the colon is created from the CT 2D slice (incorporated US Pat. No. 8, 384,771), segmentation (incorporated US Patent Application No. 15 / 904,092) identifies the colon, and subtraction extracts the contents of the colon (eg, air, excreta). In doing so, the colon remains in its original shape, and a virtual transport is inserted to allow anterior-posterior-left-right examination within the colon. This inspection technique prevents the problem of polyps being blocked by folds that can occur when inserting a forward-view camera. If no polyps are found, the patient can continue to be confident that he is in good health and return home, and the patient will experience discomfort during the preparatory phase and colonoscopy or preparatory + air insertion phase. You have escaped the feeling of anxiety. FIG. 12A shows a plaque-free representation of the inner surface of the colon with air and excrement eliminated. This largest circle 1200 represents the inner surface of the colon at the current observation position. Tissue 1201 on the surface of the inner mucosa is shown. The medium-sized dotted circle 1202 represents the inner surface of the colon at an intermediate distance from the current observation position, for example 5 cm from the current observation position. The smallest dotted circle 1204 represents the farthest distance the user can see from the current observation position. A virtual marker, such as the double-headed arrow 1206, may indicate the length in the colon that is actively visible, for example, 10 cm from the current position 1200 of the colon to the farthest visible position 1204 in the colon. The virtual road sign 1208 has a distance to an important intersection, for example "20 cm to the ileocecal region". FIG. 12B shows an indication of the inner surface of the colon with three polyps erasing air and excrement. This largest circle, 1210, represents the inner surface of the colon at the current observation position. Tissue 1211 on the surface of the inner mucosa is shown. The medium-sized dotted circle 1212 represents the inner surface of the colon at an intermediate distance from the current observation position, for example 5 cm from the current observation position. The smallest dotted circle 1214 represents the farthest distance the user can see from the current observation position. A virtual marker, such as the double-headed arrow 1216, may point to a length in the colon that is actively visible, for example, 10 cm from the current position of the colon 1210 to the farthest visible position 1214 in the colon. Villous polyp 1218 is shown. A virtual road sign 1220 with a distance to an important landmark is shown, for example "3 cm to villous polyp at 10 o'clock". Stemless polyp 1222 is shown. A virtual road sign 1224 with a distance to an important landmark is shown, for example "7 cm to a stalkless polyp at 4 o'clock".

FIG. 13 shows a portion of a virtual 3D volume medical image containing a colon portion of the large intestine stretched into one long linear tube through voxel manipulation. The contents in the tube are then segmented and removed from the tube. Finally, the tube is split along the length axis and opened to allow observation of the internal structure. There are ways to physically examine the internal structure of the colon, such as preparation, insertion of air to fill and dilate the colon, insertion of a camera with lights and to observe and record the internal structure of this system. With movement along the length of the colon. The rendered TV recording can then be presented to medical personnel and patients. The limitation of rendered TV recordings is that polyps can be blocked from the TV view by folds along the colon. In addition, if a polyp is found, the patient must return for colonoscopy at a later date, which requires additional preparation and subsequent removal of polyp tissue. The process called in this virtual process does not require an unpleasant preparatory step in the preliminary inspection. In this process, CT imaging of the colon with / without contrast is performed. A 3D virtual image is then constructed from the CT 2D slice (incorporated US Pat. No. 8,384,771). Segmentation (US Patent Application No. 15 / 904,092) is performed and the tissue outside the colon is removed. Also, non-tissue contents in the colon are removed. The folds that can obscure the polyp are then stretched, thereby'stretching'the colon so that the fold-like colon tissue does not obscure the polyp. This stretching process involves voxel manipulation as described in US Patent Application No. 16 / 195,251. This stretched linear virtual colon is divided in two along the length axis so that the internal structure can be seen through the head display unit as shown in this figure. The hollow organ colon 1300 is straightened. After straightening, the colon opens like a book 1302, and the inside of the mucosal surface can be seen from above. After opening, the first polyp is shown cut in half of the first half portion 1304 and the second half portion 1305. The second polyp is shown as complete 1306. Alternatively, the colon may be opened and pulled apart like a book, flattened to 1308, and the inside of the mucosal surface may be viewed from above. After opening, the first polyp is shown cut in half of the first half portion 1309 and the second half portion 1310. The second polyp is shown as complete 1312. When the colon is flattened, the 3D display on the headset will cause the polyp to pop out even more.

FIG. 14 shows the insertion of a virtual contrast agent and its flow in the vascular system. First, the blood in the affected blood vessels is removed. In the upper row, the blood vessels are normal and non-pathological, and placement of a virtual contrast agent indicates normal blood flow. The proximal portion 1400 of the blood vessel, the intermediate portions 401a and 1401b of the blood vessel 1, and the distal portions 1402a, 1402b and 1402c of the blood vessel are shown. Therefore, when a virtual contrast agent is inserted, it mimics the normal blood flow that will be imaged. Three time points are shown, including the initial time point 1404, the subsequent time point 1406, and the final time point 1408. At the initial time point 1404, all of the blood voxels were removed in the first place, and no virtual contrast medium was inserted. Subsequent time points 1406, the virtual contrast agent 1410 shown in gray was inserted into the proximal portion 1400 of the blood vessel and the intermediate portions 1401a and 1401b of the blood vessel, whereas the virtual contrast agent was inserted into the distal portions 1402a, 1402b and 1402c of the blood vessel. Is not inserted (lack of virtual contrast is shown in white). At the final time point 1408, the hypothetical contrast agent 1412, shown in gray, is inserted into the proximal portion 1400 of the blood vessel, the intermediate portions 1401a and 1401b of the blood vessel, and the distal portions 1402a, 1402b and 1402c of the blood vessel. In the lower row, the blood vessel is in a pathological state (ie, one of the distal arterial branches is clogged with thrombus 1413). Again, the proximal portion 1400 of the blood vessel, the intermediate portions 1401a and 1401b of the blood vessel, and the distal portions 1402a, 1402b and 1402c of the blood vessel are shown. Therefore, because of the presence of thrombus 1413, when the virtual contrast agent is inserted, the virtual contrast agent will mimic the altered blood flow pattern. Three time points are shown, including the initial time point 1414, the subsequent time point 1416, and the final time point 1418. At the initial time 1414, all of the blood voxels were removed in the first place and no virtual contrast agent was inserted. At subsequent time points 1416, the gray virtual contrast agent 1410 was inserted into the proximal portion 1400 of the blood vessel and the intermediate portions 1401a and 1401b of the blood vessel, whereas the virtual contrast agent was inserted into the distal portions 1402a, 1402b and 1402c of the blood vessel. Is not inserted (lack of virtual contrast is shown in white). At the final time point 1418, the hypothetical contrast agent 1412, shown in gray, was inserted into the proximal portion 1400 of the blood vessel, the middle portions 1401a and 1401b of the blood vessel, and two of the distal branches of the blood vessel 1402b and 1402c. However, one of the distal portions of the blood vessel, 1402a, is not filled with the virtual contrast agent 1412 because it is clogged by the thrombosis 1413. Therefore, if a thrombus is present, it is assigned an obstruction-type interactive voxel parameter. In this figure, the insertion of a virtual contrast agent is shown with a normal setting of the blood vessel and a modified setting of the blood vessel (ie, having a thrombus). It should be noted that the virtual contrast agent can travel from proximal to distal to the point of the thrombus, but cannot cross the thrombus. The remaining branches experience the insertion of a virtual contrast agent. Therefore, assigning obstruction-type interactive voxel parameters stops the flow of virtual contrast agents. Alternatively, surgical clip occlusion type interactive voxel parameters may be used.

FIG. 15 shows, among other things, ablation techniques that can be used with 3D digital structures within 3D cursors. This allows a careful examination of the inside of the organ. The method underlying the ablation technique is not limited to: It shows the first step of the process that is to be determined (using the segmentation technique outlined in 15 / 904,092) (Note: in this figure, 3D cursor 1502 (US Pat. No. 9,980, U.S. Pat. No. 9,980). The liver 1500 wrapped in 691) is shown. To determine the outer shell of the liver, proceed from the central boxel 1504 in the 3D cursor in the outward direction 1506 to the segmented surface of the liver tissue. Alternatively, the 3D cursor plane may travel inward 1508 to reach the segmented surface of the liver tissue. This process allows the tissue outside the liver to be removed). .. FIG. 15B shows the next step in the process of removing one voxel depth in order from all voxels on the outer surface. For example, the outer shell of voxels is shown in black 1510. Then repeat this step several times on the remaining outer layer of tissue in the direction of the medical personnel viewing the medical image. Alternatively, select one layer in the X, Y, Z coordinate system (eg, select the XY layer with the largest Z coordinate, remove that layer, and the rest in the direction of the medical personnel viewing the medical image. Repeat this step several times on your 3D digital volume). FIG. 15C shows the next step in the process where the internal texture type becomes apparent as the layer is removed. The original outer shell 1512 is shown. The shell 1514 after N-step ablation is shown. It should be noted that the method of displaying voxels by this may include displaying faint or semi-transparent voxels for non-priority voxels, while not displaying faint voxels for preferred voxels. In FIG. 15D, the normal liver tissue is resected and the abnormal liver tissue remains. Note that the original outer shell 1516 is shown. In this example, after the ablation process, a benign lesion called focal nodular hyperplasia 1518 has been revealed. Focal nodular hyperplasia 1518 is shown with the characteristic imaging features of the low decay central scar 1520. Such a process is a more useful method for observing tumors inside solid organs compared to the process of searching on each slice. As a result of repeating the ablation step, the volume of liver tissue is reduced from the original volume.

FIG. 16 shows a virtual focus pen that guides voxel operations. The movement of the virtual focus pen is controlled by the medical personnel viewing the medical image. This figure shows the process of increasing the distance between closely spaced, overlapping, and indistinguishable blood vessels, and increasing the distance between blood vessels. When arteriovenous malformations occur in the brain at sites where multiple vessels are close together, it is difficult to determine which of these vessels should be injected with therapeutic material. Digitally increasing the distance can help identify the appropriate blood vessel for infusion. We will discuss two states. First, non-vascular tissue separates blood vessels from each other. Second, there is little or no non-vascular tissue separating the blood vessels, and some blood vessels are clustered (clustered). When different types of tissue separate blood vessels, a) segmentation to determine the type of tissue present within the volume of interest, b) volume is a multiple factor for all non-blood and non-vascular type tissues. It is expanded by a minute or an additive factor, and c) the coordinates of blood and blood vessels are adjusted according to the expansion of non-blood type and non-vascular type tissues. Next, to illustrate the clusters, d) segmentation is performed to determine which voxels are predominantly blood, which voxels are tissues (ie, blood vessels), and e) the tissue voxels temporarily. Remove, c) then use a multiple factor (or additive factor) for all coordinates of the blood voxel, f) apply a smoothing routine to the blood vessel (optional), g) put the blood voxel into the tissue tube. Wrap in. A display for medical personnel viewing medical images shows an enlarged vascular structure, thereby facilitating treatment. One of the problems encountered in radiology is the difficulty in understanding the relationships between multiple complex anatomical structures. One example is arteriovenous malformation (AVM) in the brain. A complex brain AVM can consist of multiple winding feeding arteries, entanglement of lesions with aneurysms, and multiple outflow veins. It is extremely difficult to understand the exact anatomy of this complex structure. The process is described as follows. In the first step 1600, the structure of interest that the user wants to separate is specified. In this case, the structures desired to be separated are two pink blood vessels that are very close to each other. The user moves the tip of the virtual focus pen into the space between the two blood vessels. Note that this process installs virtual red dots that indicate the location of voxels that will later be duplicated and inserted as the manipulated tissue type. In the second step 1602, the tissue characteristics between the two structures of interest (eg, cerebrospinal fluid) are characterized. In the third step 1604, a voxel operation (eg, insertion of an additional cerebrospinal fluid type voxel) is performed so that the distance between the two blood vessels is increased, and the positions of the two blood vessels are changed at the same time. .. The first blood vessel 1606 and the second blood vessel 1608 are closely spaced and intervene only with a slice of cerebrospinal fluid type voxel 1610. A virtual pointer 1612 is shown. The tip 1614 of the virtual pointer is also shown. A virtual symbol (eg, red dot 1616) marking a position within the imaging volume to be manipulated is also shown. Specific tissue characteristics can then be assigned to the tissue characteristics between the two structures of interest (eg, cerebrospinal fluid). To illustrate this, the boundaries of each of these voxels have been changed to light blue 1618. At this point, the first blood vessel 1606 and the second blood vessel 1608 are still arranged close to each other. An additional three rows of cerebrospinal fluid voxels 1620, 1622, and 1624 are then inserted to separate the first blood vessel 1606 from the second blood vessel 1608. It should be noted that the distance between the first blood vessel 1606 and the second blood vessel 1608 is increased. This is useful in that 3D observations can no longer be improved by being able to better see and understand the relationships between less spaced structures.

FIG. 17 shows the total imaging volume and a plurality of types of subvolumes. In contrast to the slice-by-slice approach traditionally used in radiology, we present here a subvolume-by-subvolume approach. It should be noted that, as described in US Pat. No. 8,384,771, a plurality of slices are arranged to construct a volume. First, the medical professional (or computer program) may select a particular spot 1702 that marks the boundary between the first set of 2D slices 1700 and the second set of 2D slices 1704. Second, the medical professional (or computer program) may select a particular spot 1706 that marks the boundary between the second set of 2D slices 1704 and the third set of 2D slices 1708. Once a slice for a particular subvolume has been determined, that subvolume can be pulled from the stack of 2D slices for observation and analysis. In this figure, the first subvolume 1710 created from the first set of 2D images 1700 is pulled upward toward the first set of 2D images 1700. A second subvolume 1712 created from the second set of 2D images 1704 is pulled to the side of the second set of 2D images 1704. A third subvolume 1714 generated from the third set of 2D images 1708 is pulled downwards to the side of the third set of 2D images. It should be noted that assigning stack boundaries can be done in multiple ways. For example, select a convenient subvolume size (eg, 20 2D slices) and divide the subvolume accordingly (eg, slices 1-20 are assigned to subvolume # 1 and slices 21-40 are subs. Assigned to volume # 2 etc.). The number of slices in each stack can vary. Alternatively, the user may use a virtual pen to touch and select boundaries such as points 1702 and 1706. The total imaging volume can be divided into a number of different combinations of subvolumes of various sizes, which can be observed from a number of different angles (US Pat. No. 8,384,771). In this example, subvolumes can be created by constructing a set of slices.

FIG. 18 shows the sequence of movement of the 3D cursor during the volume of interest in a random pattern. Random patterns are used based on possible items of interest, and the reviewer sees the entire virtual medical image volume at once, eg changing the transparency and a density different from the nominal density of the organ being examined. Techniques such as applying pseudocolors to the structure of can be applied. The reviewer can then move and resize the 3D cursor for various objects of interest for further inspection. This figure shows the total volume of 1800 scanned. The first subvolume displayed within the 3D cursor is indicated at the first time point 1802. After that, the 3D cursor is moved in the direction 1804. Subsequent subvolumes displayed within the second resized 3D cursor are indicated at subsequent time points 1806. The 3D cursor is then moved in direction 1808. Subsequent subvolumes displayed within the second resized 3D cursor are indicated at subsequent time points 1810. After that, the 3D cursor is moved in the direction 1812. Subsequent subvolumes displayed within the second resized 3D cursor are indicated at subsequent time points 1814. This figure shows moving and resizing the 3D cursor many times to observe an example of tissue of interest. This type of search pattern can facilitate the review process. This search pattern uses a 3D cursor (US Pat. No. 9,980,691 and US Patent Application No. 15 / 878,463). Note that when reviewing medical images in the original 2D format, the eye may jump from one spot to another according to the reviewer's jumping motion path, and a significant portion of the slice may not be observed. As a result, small chunks may be missed. With the use of 3D cursors, small findings extend to larger parts of the presented image, increasing the probability of detection proportionally. When such a random search pattern is displayed, the computer program tracks the displayed portion of the total volume and the undisplayed portion of the total volume. If any part of the total volume is not visible by the 3D cursor, the program reminds the user to observe those areas. In some implementations, subvolumes are displayed to healthcare professionals in an automated pattern, including, but not limited to, windshield wiper patterns or layer-by-layer patterns. At any time, the 3D cursor and / or the subvolume in the 3D cursor can be copied and pasted into a virtual movable table for later review. For example, a radiologist may first want to clear up all possible or apparently abnormal findings. The radiologist may then wish to examine each of the abnormal findings in great detail at a later period. Each time an abnormal imaging finding is identified, the radiologist places the abnormal finding within the 3D cursor and the entire abnormal finding is included in the subvolume (eg, the entire liver weight defines the boundaries of the subvolume. It can be confirmed that it is included in the 3D cursor). The subvolume is then set aside in a virtual bucket or virtual 3D clipboard. Then, the rest of the total imaging volume is reviewed. Once the entire total imaging volume has been removed and all anomalous subvolumes have been placed in the virtual bucket or virtual 3D clipboard, the radiologist initiates a close examination of the volumes in the virtual bucket or virtual 3D clipboard.

FIG. 19 shows an example of an ordered pattern of observation of a medical image (eg, a continuous virtual windshield wiper type pattern). 1900 in the x direction, 1902 in the y direction and 1904 in the z direction are shown. The total imaging volume 1906 is shown. A virtual windshield wiper can have multiple implementations in which the cursor moves in multiple orderly ways. As shown, the first subvolume 1908 to be inspected within the 3D cursor at the first point in time is shown, where one of the coordinates of the 3D cursor has one corner at position (0,0,0) 1910. is doing. The 3D cursor first increases the X direction of the 3D cursor as indicated by the dashed arrow 1912, and the y and z coordinates are invariant so that the subvolume along this moving direction is optimized. Moving. Therefore, this pattern continuously increases the X coordinate while keeping the Y and Z coordinates constant. When the corner of the 3D cursor reaches the maximum x value of the total imaging volume 1906, the 3D cursor has the y direction of the 3D cursor increased and the x and z coordinates are invariant as indicated by the dashed arrow 1916. , The subvolume along this direction of movement is moved so that it can be optimized. Therefore, when the maximum value of the X coordinate is reached, the y coordinate is incremented by a certain increment, the X coordinate is continuously decreased until the minimum X coordinate is reached, and then the Y coordinate is incremented again. Then, this process of moving the 3D cursor in the x-direction 1900 and the y-direction 1902 is repeated until the point where the bottom layer of the total imaging volume 1906 is completely inspected by the 3D cursor 1918. When this plane is complete, the Z coordinate is incremented. The 3D cursor is shifted upwards in the z direction 1904 1920. It should be noted that during this ordered search pattern, anomalies 1922 can be found at a particular 3D cursor position 1924. Such anomalies can be placed in a virtual bucket or virtual 3D movable table for further analysis. As shown in 1926, all subvolumes in the total imaging volume are inspected and the 3D cursor is repeatedly ordered in the total imaging volume 1906 until the 3D cursor reaches its final spot 1928. The movement can be made. A variation of the windshield pattern is'flyback'where the pattern resumes the X-coordinate increment with the Y-coordinate increment after the first row is completed. This type of search pattern helps ensure that a thorough inspection has been performed. This search pattern uses a 3D cursor (US Pat. No. 9,980,691 and US Patent Application No. 15 / 878,463). Note that when reviewing medical images in the original 2D format, the eye may jump from one spot to another according to the reviewer's jumping motion path, and a significant portion of the slice may not be observed. As a result, small chunks may be missed. With the use of 3D cursors, those small chunks extend to a larger portion of the presented image, increasing the probability of detection proportionally. In some implementations, subvolumes are displayed to healthcare professionals in an automated pattern, including, but not limited to, windshield wiper patterns or layer-by-layer patterns. It can be seen that the automated search pattern within the volume of interest increases the probability of detection. In this figure, an automated search pattern is shown so that the 3D cursor moves through the volume of interest. The mass is specified in a later subvolume. Future automated search patterns within the volume of interest are back and forth within each layer (similar to windshield wipers) and back and forth within the next layer.

FIG. 20 shows the volume of interest to be reviewed and the process by which missed review intent areas can be highlighted to the reviewing healthcare professional. These identified subvolumes are subsequently reviewed, thereby ensuring the integrity of the review. This process uses 3D cursors (US Pat. No. 9,980,691 and US Patent Application No. 15 / 878,463) to sequentially step into the volume to be examined (eg, according to a medical institution checklist). Invokes the sequential selection of subvolumes of the volume of interest by. In addition, questions may arise as to whether the entire volume was inspected after this step-by-step process was completed. In this implementation, the volumes contained in each of the inspected 3D cursors are summed and removed from the original total volume. This can result in missing some of the original volume that the review was intended for. In this implementation, those missed parts may be highlighted to the reviewing healthcare professional and warned to continue reviewing and examining the missed parts. Note that when reviewing medical images in the original 2D format, the eye may jump from one spot to another according to the reviewer's jumping motion path, and a significant portion of the slice may not be observed. As a result, small chunks may be missed. With the use of 3D cursors, those small chunks extend to a larger portion of the presented image, increasing the probability of detection proportionally. This figure illustrates the sequencing of 3D cursor movement within an organ of interest. The volume of interest (ie, liver) 2000 is shown. The subvolume 2002 displayed at time point # 1 is shown. The 3D cursor moves 2004 in an orderly manner in the volume of interest 2000, as described, for example, in FIG. At time point #N, the last subvolume 2006 is displayed. Control of the display of medical images (eg, incremental changes from one to another) is controlled by the medical practitioner. Alternatively, the user may control the movement of the 3D cursor by means of a joystick or other georegistrated tool, as described in US Patent Application No. 16 / 524,275. Eventually, the volume displayed within the 3D cursor can be tracked and reviewed at a later time (ie, before the inspection is completed). The size of the subvolume varies based on routine screening indications for cancer. Also, recording the position of the 3D cursor over time and comparing the displayed subvolume with the total volume makes it possible to deal with the subvolume 2008 that has not yet been displayed to the medical professional. Alternatively, it is possible to determine which subvolume has not yet been displayed by pulling the displayed subvolume from the entire volume. It should be noted that even if a small area of the structure 2008 is missed (ie, not included in the 3D cursor volume), they can be tracked and the radiologist will be able to track these areas prior to the completion of the examination. Have the option to review. These missed subvolumes 2008 can be moved to a new position and inspected. In another embodiment, the user can select the size of the 3D cursor, the speed at which the cursor moves, and the computer performs automated movements of the checklist items in the volume of interest. If the organ is inconspicuous, the subvolumes in the cube can be altered to make the imaged structure inconspicuous (eg, erased, altered by the Hansfield unit, etc.).

FIG. 21 shows a human icon that includes the position of the 3D virtual cursor at an approximate position within the body. This icon can be used in conjunction with observing the display of 3D medical images. In the process of volume testing by viewing medical images, it may be useful for healthcare professionals to quickly refer to the icon to correctly re-determine exactly where in the body is the tissue of interest / concern. .. This icon is also useful for discussions among healthcare professionals. FIG. 21A shows a body icon 2100 in a vertical position facing forward. This icon outlines (marks up) the subvolume being inspected. Such markups include, but are not limited to: markups of areas of concern pointed to by the ordering physician; markups of volume segments in which the radiologist is actively working (eg,). The radiologist is actively working on the liver item on the checklist, where the segmented liver is marked up on the icon); the markup of the subvolume being examined by the radiologist (eg,) , Radiologists are actively working on subvolumes in the liver within the boundaries of volume-containing 3D cursors); Markup of perspective on icons. For example, the ordering physician points to an area of concern (eg, as described in US Provisional Patent Application No. 62 / 843,612, METHOD OF CREATING A COMPUTER-GENERATED PATIENT SPECIFIC IMAGE). Can be sent to the doctor) and the area can be marked up on the virtual icon 2102. The segmented volume 2104 (eg, liver) in which the radiologist is actively working can then be marked up. The subvolume in the 3D cursor 2106 can then be marked up. Additional symbols may be shown outside the icon. For example, the initial viewpoint symbol 2108 is shown to represent the initial viewpoint. The movement symbol 2110 represents a change in position from the initial viewpoint represented by the initial viewpoint symbol 2108 to the subsequent viewpoint represented by the subsequent viewpoint symbol 2112. FIG. 21B shows a user augmented reality headset 2114 with a left eye display 2116 with a left eye view of the 3D cursor 2118 and a right eye display 2120 with a right eye view of the 3D cursor 2122. The left-eye view of the marked-up 3D icon 2124 is shown on the left-eye display 2116, and the right-eye view of the marked-up 3D icon 2126 is shown on the right-eye display. Therefore, the contour of the subvolume being inspected can be one of the icon markups. The approximate position of the (one or more) 3D cursors within the human body icon is another example of markup for the icon. Body orientation is under the control of the medical personnel viewing the medical image, as well as whether or not to display the icon. For example, the icon may be rotated, translated, bent (with corresponding voxel manipulation if desired), or subject to other changes as directed by the radiologist. Also, the markup icon of the 3D cursor may be added to the diagnostic 2D radiation monitor. When a medical person viewing a medical image rotates, tilts, and zooms the tissue contained in a 3D cursor, it may be useful to see where the current viewpoint is relative to the original viewpoint (eg, voxels). Position has been changed from the original orientation to the new orientation through roll, pitch, and / or yaw commands). This figure shows a curved arrow starting from the original point of view and ending at the current point of view. Whether or not to display the icon is under the control of the medical personnel who view the medical image. It is useful that the 3D cursor icon displays the contents of the 3D cursor, which is rotated and observed from different viewpoints, and at the same time sees the current position and where the viewpoint is with respect to the original position. be.

FIG. 22 shows a virtual movable table for storing virtual images of suspicious tissues stored by the checklist category. In this figure, in addition to the virtual storage box 2202 corresponding to the items on the medical institution checklist, the box 2204 for emergency items and the general / miscellaneous miscellaneous box 2206 (eg, image artifacts, teaching examples, quality). (Improvements, etc.) are shown above the virtual movable table 2200. Emergency Box 2204 can be used to place findings on items of critical, time-sensitive information. The virtual movable table is mobile in the sense that the user can view the virtual movable table on the augmented reality headset beside the imaging volume that the radiologist is currently working on. The radiologist may then move or resize it so that it is convenient for the workspace. Items that medical personnel consider important are'drag and place'to each virtual box according to the items in the checklist to be reviewed. For boxes that do not have a significant item, there is a description'not noteworthy'on the checklist item in the report that disappears when the item is added, and the radiologist replaces that item in the checklist with the appropriate description. .. In addition to reviewers, medical procedure personnel are alerted to and given access to'emergency boxes' that contain critical items. These items may be co-reviewed by both the therapist and the reviewer on the basis of rapid treatment. This table with storage boxes facilitates the creation of reports and enhances the quality and completeness of the reports. Current reports are nominally limited to word descriptions only. Under this process, annotated diagrams containing the organization in question may be added.

FIG. 23 shows a sample radiology report containing images processed by a virtual tool. It should be noted that the Radiology Report Sample 2300 contains 3D cursors and anomalous imaging findings.

Claims (62)

For the selected 3D image volume loaded in the image processing system, select a set of virtual tools from a set of available virtual tools in response to user input.
Georegistrate each virtual tool in the selected set with the 3D image volume.
Manipulating the 3D image volume in response to the operation of a virtual tool in the set of virtual tools.
How to have that. The set of available virtuals including virtual focus pens, virtual 3D cursors, virtual transport viewers, virtual pedestals, virtual knives, virtual catheters, virtual load signs, virtual ablation tools, virtual tables, virtual contrast agent tools, and virtual icons. The method of claim 1, wherein the set of virtual tools is selected from the tools. The set of virtual tools includes a virtual focus pen, the method of which highlights and annotates a portion of the 3D image volume to make the 3D image volume in response to the virtual focus pen. The method of claim 1, comprising operating. The method according to claim 3, wherein the three-dimensional image volume voxel adjacent to the tip of the virtual focus pen is changed. The set of virtual tools includes a virtual knife, the method responding to the virtual knife at least in one of a group consisting of separating voxels from the three-dimensional image volume and changing the position of the voxels. The method according to claim 1, wherein the three-dimensional image volume is operated. The set of virtual tools includes a virtual transport viewer, the method of moving the virtual transport viewer within the hollow structure of the three-dimensional image volume and presenting an image from the viewpoint of the virtual transport viewer. The method of claim 1, wherein the three-dimensional image volume is manipulated in response to the virtual transport viewer. The method of claim 6, wherein the virtual transport viewer is used to perform a virtual colonoscopy. The set of virtual tools comprises a virtual contrast agent, the method of which manipulates the three-dimensional image volume in response to the virtual contrast agent by inserting a visible moving voxel into the three-dimensional image volume. The method according to claim 1. 8. The method of claim 8, wherein different of the moving voxels are assigned a data unit (eg, Hansfield unit) value. Claiming that manipulating the 3D image volume in response to an operation of a virtual tool in the set of virtual tools comprises removing voxels in the outer shell of an organ in a repeating manner on a shell-by-shell basis. The method according to 1. Manipulating the 3D image volume in response to the operation of a virtual tool in the set of virtual tools means that adjacent tissues of interest are separated by adjusting the coordinates of multiple voxels of the tissue of interest. The method according to claim 1. The set of virtual tools includes a virtual table, the method of which is to place a plurality of parts of the 3D image volume in a virtual storage box of the virtual table in response to the virtual table. The method of claim 1, wherein the method comprises the operation of. The set of virtual tools includes a virtual catheter, the method of which is to limit the movement of the virtual catheter to a columnar blood voxel in a selected blood vessel, thereby responding to the virtual catheter with the three-dimensional image volume. The method of claim 1, wherein the method comprises the operation of. The method according to claim 1, wherein the information related to the selected subvolume of the three-dimensional image volume is automatically displayed. 14. The method of claim 14, comprising displaying the patient's current state, the patient's medical history, test results, and pathological results prompted to obtain patient metadata and medical image volume. 14. The method of claim 14, wherein the information is displayed using a virtual windshield. The method of claim 1, wherein the distance to a metric is displayed using a virtual load sign. The method of claim 1, wherein a visual auxiliary icon indicating a viewpoint is displayed. The method of claim 1, wherein a visual auxiliary icon indicating a finding detected by an artificial intelligence algorithm is displayed. The method of claim 1, wherein the visual auxiliary icon modified to be associated with the imaging volume (eg, indicating the position of the displayed subvolume with respect to the three-dimensional image volume) is displayed. .. The method of claim 1, wherein the volume-containing three-dimensional cursor is used to select at least one subvolume. 21. The method of claim 21, wherein the subvolume is selected from a plurality of subvolumes in a predetermined list of subvolumes. 22. The method of claim 22, wherein each of the plurality of subvolumes in the list is sequentially displayed. 21. The method of claim 21, wherein the subvolume is selected from a plurality of subvolumes defined by sequential search pattern coordinates. 21. The method of claim 21, wherein the subvolume is selected from a plurality of subvolumes defined by random search pattern coordinates. Manipulating the 3D image volume can change the voxel size, change the voxel shape, change the voxel position, change the voxel direction, change the voxel internal parameters, and create a voxel. The method of claim 1, comprising at least one of the following and the removal of voxels. The method of claim 1, wherein manipulating the three-dimensional image volume comprises dividing the subvolume of interest into a plurality of portions based on common characteristics. Manipulating the 3D image volume has the effect of producing an exploded view, in which the segmented structures of the 3D image are moved away from points in the 3D image volume. , The method according to claim 1. The method of claim 1, wherein the virtual eye tracker symbol is used to assist the human eye in observing. Claimed to have the virtual eye tracker symbol appear and disappear at spatially separated positions so that the human eye can perform jumping movements to jump from one position to another. 29. 29. The method of claim 29, wherein the virtual eye tracker symbol is smoothly moved along a path so that the human eye can perform smooth tracking. Has an image processing system,
The image processing system presents a set of virtual tools for selection from a set of available virtual tools in response to user input for the selected three-dimensional image volume loaded into the image processing system. An interface that geo-registers each virtual tool in the selected set with the three-dimensional image volume, and an image processor that operates the three-dimensional image volume in response to an operation of the virtual tool in the set of virtual tools. , Have,
Device. The set of virtual tools includes a virtual focus pen, a virtual 3D cursor, a virtual transport viewer, a virtual pedestal, a virtual knife, a virtual catheter, a virtual load sign, a virtual ablation tool, a virtual table, a virtual contrast agent tool, and a virtual icon. 32. The device of claim 32, selected from the group of available virtual tools. The set of virtual tools includes a virtual focus pen, the image processor responding to the virtual focus pen by highlighting and annotating a portion of the 3D image volume. 32. The device of claim 32. 34. The device of claim 34, wherein the image processor modifies a three-dimensional image volume voxel adjacent to the tip of the virtual focus pen. The set of virtual tools includes a virtual knife, and the image processor responds to the virtual knife at least in one of a group consisting of separating voxels from the three-dimensional image volume and changing the position of the voxels. 32. The apparatus according to claim 32, wherein the three-dimensional image volume is operated. The set of virtual tools includes a virtual transport viewer, the image processor moving the virtual transport viewer within the hollow structure of the three-dimensional image volume and presenting an image from the viewpoint of the virtual transport viewer. 32. The device of claim 32, which operates the three-dimensional image volume in response to the virtual transport viewer. 37. The device of claim 37, wherein the virtual transport viewer is used to perform a virtual colonoscopy through the interface. The set of virtual tools includes a virtual contrast agent, and the image processor operates the three-dimensional image volume in response to the virtual contrast agent by inserting a visible moving voxel into the three-dimensional image volume. 32. The apparatus according to claim 32. 39. The apparatus of claim 39, wherein the image processor assigns different data units (eg, Hansfield units) to different of the moving voxels. The image processor manipulates the three-dimensional image volume in response to the operation of the virtual tools in the set of virtual tools by removing the voxels of the outer shell of the organ in a repeating manner on a shell-by-shell basis. The device according to claim 32. The image processor responds to the operation of a virtual tool in the set of virtual tools by separating adjacent tissues of interest by adjusting the coordinates of multiple voxels of the tissue of interest. 32. The device of claim 32. The set of virtual tools includes a virtual table, and the image processor responds to the virtual table by arranging a plurality of parts of the three-dimensional image volume in a virtual storage box of the virtual table, so that the three-dimensional image is displayed. 32. The device of claim 32, which operates a volume. The set of virtual tools includes a virtual catheter, the image processor responding to the virtual catheter by limiting the movement of the virtual catheter to a columnar blood voxel in a selected blood vessel. 32. The device of claim 32, which operates a volume. 32. The apparatus of claim 32, wherein the interface automatically displays information related to a selected subvolume of the three-dimensional image volume. 45. The device of claim 45, wherein the interface displays the current state, the patient's medical history, the test results, and the pathological results from which the patient's metadata and medical image volume were prompted to be acquired. 45. The device of claim 45, wherein the interface uses a virtual windshield to display the information. 45. The device of claim 45, wherein the interface displays a distance to a metric using a virtual load sign. 32. The device of claim 32, wherein the interface displays a visual auxiliary icon indicating a viewpoint. 32. The device of claim 32, wherein the interface displays a visual auxiliary icon indicating a finding detected by an artificial intelligence algorithm. 32. The interface comprises displaying a visual auxiliary icon modified to be associated with the imaging volume (eg, indicating the position of the displayed subvolume with respect to the 3D image volume). The device described in. 32. The apparatus of claim 32, wherein the interface receives a selection of at least one subvolume using a volume containing three-dimensional cursor. 52. The device of claim 52, wherein the selected subvolume is one of a plurality of subvolumes in a given list. 53. The device of claim 53, wherein the interface sequentially displays each of the plurality of subvolumes in the list. 52. The apparatus of claim 52, wherein the selected subvolume is one of a plurality of subvolumes defined by sequential search pattern coordinates. 52. The apparatus of claim 52, wherein the selected subvolume is one of a plurality of subvolumes defined by random search pattern coordinates. The image processor operates the three-dimensional image volume by changing the voxel size, changing the voxel shape, changing the voxel position, changing the voxel direction, and changing the voxel internal parameters. 32. The device of claim 32, comprising at least one of creating voxels and removing voxels. 32. The apparatus of claim 32, wherein the image processor operates the three-dimensional image volume by dividing the subvolume of interest into a plurality of portions based on common characteristics. The image processor manipulates the 3D image volume by generating an exploded view, and the segmented structure of the 3D image is moved away from a point in the 3D image volume. 32. The apparatus according to claim 32. 32. The device of claim 32, wherein the interface has a virtual eye tracker symbol. 60. The virtual eye tracker symbol appears and disappears at spatially separated positions so that the human eye can make jumping movements to jump from one position to another. The device described. 60. The device of claim 60, wherein the virtual eye tracker symbol moves smoothly along a path.

Images