JP2000172836A - Imaging system for generating image spread sheet and method for implementing image spread sheet - Google Patents

Abstract

PROBLEM TO BE SOLVED: To match images from a variety of modalities by allowing an image spread sheet generator to execute a given set of expressions in a 1st set of cells in response to user input and converting a 1st image to cells of the 1st set. SOLUTION: The image spread sheet generator 10 is connected to a graphic generator 12 of a graphic system 14 which has a processor 16. Then the image spread generator 10 displays images in cells, which are so arrayed and aligned to match registration markers of the respective cells, in response to image data having relative registration data. The image spread sheet generator 10 executes the given set of expressions in the cells of the 1st set and converts the images to the cells of the 1st set in response to user input.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an imaging system for generating an image spreadsheet. The present invention relates to a method for executing an image spreadsheet.

[0002]

BACKGROUND OF THE INVENTION Spreadsheet application programs and spreadsheet metaphors that provide an intuitive and proven method for manipulating numerical data are known. Such data is typically text or ASCII values, and competes with other data in different formats, such as graphic data.

[0003] In the medical imaging arts, there are several very specific problems and the need to simultaneously view multiple, diverse and diverse types of images. Imaging modalities for three-dimensional images, such as CT, MR, PET, SPECT,
In 3D ultrasound, rather than an image, a volume is formed, which is usually the volume of the patient being imaged.
It is displayed as a cut slice in the form of a stack.
As a result, multiple images from different modalities need to be displayed simultaneously.

[0004] It is common to image a patient multiple times with a single or multi-modality. In the case of multi-modality imaging, the same target area is observed, but different quantities are measured. Additionally, spatial resolution may vary between image sets. Multiple inspections with a single modality are often used to perform differential evaluation. For example, contrast-enhanced magnetic resonance angiography, which is imaged before and after a contrast-enhancing agent (agent) is injected into the bloodstream, and is a treatment / diagnosis evaluation using pathological images before and after an intervention. Functional imaging in which the biological process is monitored over time.

In both single and multi-modality cases, imaging problems arise due to patient motion and scanner repeatability. In some cases, images of the patient may be taken individually over days, weeks, or months. Even if the patient remains in the scanner for the high-speed data acquisition set, the patient may be misaligned due to breathing, discomfort, excitement, or a digestive process. The scanner itself may shift due to mechanical inaccuracies.

Therefore, there is a need to match images from a variety of modalities for use in medical diagnosis and treatment.

[0007]

SUMMARY OF THE INVENTION It is an object of the present invention to provide an imaging system for generating an image spreadsheet and a method of executing the image spreadsheet, for use in medical diagnostics and therapy, for imaging from a variety of modalities. Is to match.

[0008]

SUMMARY OF THE INVENTION In accordance with the present invention, a spreadsheet having a plurality of cells with a processor for displaying a graphic user interface (GUI) on a display and an associated registration marker is provided. An image spreadsheet generator for displaying the grid in a graphic user interface (GUI), wherein each cell can be associated with a formula, the image spreadsheet generator can convert a plurality of images into a plurality of images; Responsive to image data having associated registration data for display in a portion of the cells, the plurality of cells being aligned and oriented to match a registration marker of each respective cell of interest. With images Red sheet generator, in response to user input, the expression of a given set running in the cell of the first set is solved by converting the first image into cells of the first set.

According to the present invention, a graphic user interface (GUI) is provided on a display.
, And an image spreadsheet generator for displaying a spreadsheet grid having a plurality of cells with associated registration markers on a graphic user interface (GUI), each cell comprising: , Associated with the formula, the multi-dimensional image can be displayed, and the image spreadsheet generator responsive to the set of formulas in the first set of cells converts the first image to the first set of cells. 1
And the image spreadsheet generator responds to the image data corresponding to the multi-dimensional image by applying the first set of cells to the first set of cells before applying the set formula to the first set of cells. This is solved by automatically aligning the images via a transformation.

According to the present invention, there is provided, in accordance with the invention, displaying a spreadsheet grid having a plurality of cells on a graphic user interface (GUI), associating at least one of the plurality of cells with a formula. Processing the image data having the registration data of, displaying a plurality of images in a plurality of cell portions, and aligning and orienting the plurality of images in each cell, Matching registration marks, at least one
Converting the first image into a first set of cells according to two formulas.

[0011]

BRIEF DESCRIPTION OF THE DRAWINGS Particularly advantageous embodiments of the invention are set out in the dependent claims.

In an image spreadsheet, image data spreadsheet metaphors are two-dimensional (2D) and three-dimensional (3D) using cells in the image spreadsheet assigned to image data and mathematical operations to perform image operands. 3D) Extended to images and image sequences. The image spreadsheet has a graphic user interface (GUI) on the display.
Executed on an imaging system having a processor for displaying. With the image spreadsheet generator, the graphic user interface (GUI) displays a spreadsheet grid with multiple cells with associated registration markers,
Each cell allows it to be associated with a formula. The image spreadsheet generator is responsive to the image data having the associated registration data, the plurality of images having the plurality of images aligned and oriented to match the registration markers of each cell. Display inside the cell part. The image spreadsheet generator executes a predetermined set of formulas in the first set of cells in response to user input,
Convert the image to a first set of cells.

The predetermined image conversion functions so that the user can customize each cell forming the image to be converted to a cell. For example, using image subtraction (image subtraction), magnetic resonance angiography (MRA) images may be used to remove background vasculature, including images shown with contrast enhancement agents injected into the vasculature. And the flow of the contrast enhancing agent is displayed. Two-dimensional images, three-dimensional images, and even text may be displayed in cells. A dog ear icon, pop-up window, or pop-up user interface (UI) in the cell may be actuated to selectively display one of the plurality of images associated with the first cell.

[0014]

BRIEF DESCRIPTION OF THE DRAWINGS The invention will now be described in more detail with reference to preferred embodiments shown in the drawings.

FIG. 1 is a diagram showing an entire system for executing an image spreadsheet.

As can be seen from FIG. 1, the image spreadsheet generator 10 and method have been used to manipulate two-dimensional (2D) and / or three-dimensional (3D) images in a graphic display system, and therefore, A familiar workspace is provided for processing many types of image data, a significant paradigm shift from the pipeline model used in common image processing systems. The image spreadsheet generator 10 is connected to a graphic generator 12 of a graphic system 14 having a processor 16. The graphics generator 12 interacts with the processor 16 via, for example, a task card 18 or other functions or subroutines, and displays 2D and / or 3D images on a screen or other output device.
Form and display an image. The processor 16 operates using an image symbol generator 20 that converts the imaging symbols into a collection of software objects for display on the output screen of the graphics system 14 for the user to understand. Good.

The processor 16 has an interface 2
2. It has a graphic user interface (GUI) manager 24 and an application program interface (API) 26. Such components 22-26 are connected to an interface server 28, each of which interfaces with the interface server 28, and via the interface server 28 with other systems 30, such as the Internet for remote communication. Interact with each other. Such components 22-26
Is connected to a hard copy server (HcS) 32 and transmits an output signal to an imaging device or a printer 34 for hard copy output. Such components 22-26 may include, for example, a database server (Db) for storing data in online storage 38.
S) 36. Such components 22-26 include a processing server (PrS) 40 for storing data in memory 42 of processor 16 or other central processing unit (s).
Is connected to Such a component 22-2
6 is connected to a user interface (UI) server 44, the user interface (UI) server 44 using at least one input and / or using the X WINDOWS protocol (eg, version 11 (X11)). Or an output (I / O) device 46,
For example, a display screen, keyboard, and / or
Or form an interface with a mouse.

In an advantageous embodiment, graphic generator 12 comprises a SIEMENS MEDICAL SYS
TEMS, INC. (Iselin, NewJerse
y) commercially available FLY THROUGH software, which is available from SIEMENS MEDICAL SYSTEM.
S, INC. Run on a workstation of the graphics system 14, such as PROMINENCE distributed by (Iselin, New Jersey). PROMINENCE is a Canadian ISG TE
An advanced medical viewing and post-processing workstation manufactured by CHNOLOGIES. Alternatively, the graphics system 14
Is a SIEMENS MEDICAL SYSTEM
S, INC. 3DVIRTUSOSO post-processing workstation available from WINDOWS NT platform, optionally, on a WINDOWS NT platform.
OM INFORMATION SYSTEMS, IN
C. Available LABORATORY INFOR
Use MATION SYSTEM. FLY TH
The GUI for using the ROUGH software is S
Available S made by IENS CORPORATION
It may be executed on the IENET MAGIC VIEW CT / MR satellite console (I / O device 46
Works as). Alternatively, U.S. Pat.
No. 762,909 (Vinging) and / or U.S. Pat. No. 5,825,909.
The issue specifications (Jang), which are referred to herein by reference, may be used as the graphics system 14.

In the case of 2D / 3D medical graphic imaging, the image symbol generator 20
The META4 collection of software objects may be used, and according to the META4 collection, imaging symbols are displayed in a metaphorical manner on the user interface UI of the I / O device 46 in a manner that is easy for the user to understand. The device interface 22 is D
ICOM (Digital Imaging and
Communications in Medicin
e) a set of communication interface objects, the interface server 28
It may be a DICOM-based server that interfaces with a medical system 30, such as a computer assisted tomography (CAT) scanner and X-ray equipment. In addition, the GUI manager 24 uses the M for XWINDOWS.
The OTIF manager may be used, and the API 26
enGL (Open Graphics Library)
y) The standard graphics API may be used.

The graphic generator 12 has a segmentation module, a model builder module, and an interactive display module. The segmentation module segments show different organs of the region of interest during the simulation of the endoscopy and generate a segmented volume data set. Model Builder Module Converter
Transform the segmented volume dataset into a segmented polyhedral model set,
Real-time display is performed by the interactive display module. GUI of I / O device 46
Provides the user with a screen having a viewing window, such as the main viewing window 48 shown in FIGS. 2-3. The main viewing window 48 may be subdivided into spreadsheet cells 50 for specific display purposes as described below. A user input window, such as window 52, may be dependent on a command from the user, or on a condition, e.g.,
It may be opened or closed depending on the graphics generator 12 responding to a QUERY USER FOR INPUT condition or an error condition. Window 52 is used for mathematical operations or cell 50
Pop-up window used to initiate other operations for the individual or group of users.

The portion 54 of the screen includes portions showing selectable tasks and a control panel of a scrollable task bar. For example, depending on available task cards, 2D, MPR, 3D, Ang, Re
Commands or icons such as g, U1, U2 and FLY may be provided. Other commands, such as task cards or tabs, may be displayed so that FILM, S
Commands such as AVE, ZPAN, and W / 1 can be selected. COPY SCREEN, COPY
SERIES, NEW PAGE, CLEAR IM
Icons labeled AGE, OPTIONS,
An icon such as a scrollable arrow command may be provided.

To execute the image spreadsheet in the main view window 48, a portion of the screen 5
4 is a cell edit command or CELL EDITOR,
ADD ROW, ADD COLUMN, EVALUA
It may have icons such as TE ALL, LOAD, and SAVE. Thus, the user can select a cell by clicking on the cell, ie, by moving the mouse accordingly, the GU
The user can select by moving the I cursor to the cell and operating the mouse button. Using such editing commands, the user can edit the cell, add rows and columns to cell 50, evaluate all the cells, and change the entire cell spreadsheet to reflect the changed image or formula. To load an image into an individual cell and save the current image, current formula, and / or current cell configuration throughout the spreadsheet.

As shown in FIG. 1, the I / O device 46 may optionally be connected to an image spreadsheet generator 10 for receiving user input, which allows the user to access the window 4.
Looking at the screen with 8,54, the view selection and orientation can be controlled to navigate through the image shown in cell 50.

Image spreadsheet system 10
Is hardware and / or software, incorporated as a subroutine as the graphics generator 12, or located remotely with respect to the graphics system 14, ie, for example, the image spreadsheet system 10 is connected to the graphics generator 12. And a personal computer that communicates with the graphic generator 12 via a network such as the Internet.

As shown in FIG. 2, the cells 50 of the image spreadsheet are labeled in a typical spreadsheet fashion, with numbers being labeled in cells 50 forming rows and forming columns. The cell is labeled with the alphabet, and in that way,
Cells A1 to A3, B1 to B3, C1 to C3, etc. are formed. Each cell 50 is associated with a formula or formula, which formulas or formulas can be viewed on an alternative display screen toggled by a user command, or a portion 54 of the display screen.
Are shown in a predetermined formula display area. In an alternative embodiment, the image spreadsheet shows a 2D image as shown in FIG. 2, a 3D image as shown in FIG. 3, and an image spreadsheet as shown in FIG. Showing text or, as shown in FIGS. 2-3, text,
3 shows a combination of a 2D image and a 3D image.

Referring to FIG. 2, each cell 50 has a registration marker 56, which is visible or invisible, ie, the cell 50 has such a registration marker 56.
The image spreadsheet generator 10 has associated data in memory that locates the registration marker 5 in response to a user command.
6 is toggled whether it is displayed in the selected cell 50 or in all cells 50. The registration marker 56 is used to position and align the image, and in the default setting, the registration marker 56 is used during the loading and viewing sessions of the image unless the user toggles the display of the registration marker 56. , Invisible to the user. For example,
Cells A1, B1 and C1 display a progressive frame during a magnetic resonance angiography (MRA) session, where a set 58 of blood vessels surrounding the tissue is displayed in cell A1. Cells B1 and C
1 shows a state in which the contrast enhancing agent is injected into the blood vessel, and the images 60-62 show the blood vessel 58 together with the contrast enhancing agent flowing through the blood vessel. The registration marker 56 allows the common subject, or blood vessel, to be imaged differently so that it is aligned and oriented with very high accuracy. The registration markers are stored by the image spreadsheet generator 10 in at least one of the image spreadsheets.
A visualization of the underlying spatial mapping between at least two images or volumes generated in one cell. In such a mapping, any given point in one image is mapped to a corresponding point in the image or volume. Within the image spreadsheet, the mapping is calculated to maintain anatomical structure coherency, as described below. Therefore, such a registration marker 5 shown in FIGS.
6 is shown for illustrative purposes only and does not need to be provided much in a cell and is neither visible nor invisible in the image spreadsheet, but instead such a registration marker 56 comprises a bit, pointer , Memory locations, or other data, a registration technique between the images can be formed by the registration technique performed by the image spreadsheet generator 10.

Cells A1B1 and C1 store a file F in a memory (such as the memory of the graphics system 14) previously received from a source of image data of the MRA session.
1, F2, and F3. Therefore,
The LOAD operation is performed in the image spreadsheet by the formula A1 = LOAD (F1), which loads cell A1 to display image 58. that time,
This formula is associated with cell A1. Alternatively, cells A1, B1, C1 have images corresponding to the times T1, T2, T3 of the MRA session. Therefore, the LOAD operation is represented by the equation A1 = LOAD (T
1) is executed in the image spreadsheet which loads cell A1. Alternatively, cells A1B1,
C1 has images corresponding to the different frames FR1, FR2, FR3 from the magnetic resonance imaging system, that is, cell A1 has the associated formula A1 = LOAD.
(FR1) = LOAD (FR1).

For such an MRA session, the images in cells A1B1, C1 obtained from the frames are associated with the respective times. An image spreadsheet is a supporting image function, such as FRAME (), which returns a frame from memory at time t in a time sequence of frames and, by the formula for cell 50, performs a cell operation. The function used is called. Therefore, A1 = LOAD
(FRAME (T1)) restores the image within the frame at time T1 and loads the image into cell A1.

During cell loading, the spreadsheet generator 10 automatically determines the registration markers for each image, then formats the images, places the image registration markers, and As shown, the information is displayed at the registration marker 56.

Other cell operations are well-defined in the form of imaging and support known mathematical operations, including, for example, addition, subtraction, multiplication, and division. For example, in the example of the MRA shown in FIG. 2, cells A1, B1 and C1 are shown temporarily introduced, and cell A1 has an image 58 of a blood vessel without a contrast enhancing agent 66. The cell B1 has an image of a certain contrast enhancing agent in the blood vessel, and the cell C1 shows the blood vessel after a lapse of a predetermined time after the introduction of the contrast enhancing agent 66.

After injection of the contrast enhancer 66, the blood vessels glow brightly. Surrounding anatomy remains relatively unchanged and can be removed by subtracting the pre-contrast image set from the post-contrast image set. Cell A2 is then instructed by the user to cause image spreadsheet generator 10 to use the associated formula A2 = C1-A1, which causes image spreadsheet generator 10 to subtract image 58 from image 62. In this way, the image of the base blood vessel is removed, so that only the contrast enhancing agent 66 and the path of the contrast enhancing agent through the blood vessel are shown. For example, such a subtraction,
Therefore, filtering the image of the blood vessel from image 62 removes the image of the background tissue from image 62, and the path of the contrast enhancer shown in image 64 may reduce any obstructions or blood clotting in the blood vessel. Indicate the blood flow aspect, such as the blood flow endpoint 68 shown. Similarly, blood and blood flow around tumors and other cancers are indicated by subtraction operations.

[0032] Text is entered into a cell (eg, cell A3), and the formula adds or subtracts such text from other cells. For example,
Cell B2 is a composite of cell A1 and cell A3 by operation B2 = A1, so that the text can be inserted as a title or label into image 58, and then image 58 can be Printed or output as presentation material, such as a hardcopy document or slide. Additionally,
Each cell 50 is associated with a plurality of other cells and / or
Indicating a plurality of images, and thus the image spread generator 10 may provide a dog ear icon 70 or other graphical icon in each cell, or
Alternatively, only such cells can have such multiple images. Alternatively, a pop-up window or pop-up interface (UI), such as input window 52, is created and displayed for the user to selectably view the different images associated therewith. Dock ear icon 70, pop-up window or pop-up interface (U
Using I), the user "pages", i.e., loads or scrolls through multiple images, and with a minus sign, backwards paging to the previous image,
With the plus sign, it is possible to forward page to the next image.

In an alternative embodiment, an arrow or other browser-actuable icon may be displayed simultaneously with such a dog-ear icon 70 or pop-up window or UI, or such a dog-ear icon 70 Alternatively, a pop-up window or UI can be provided and the image spreadsheet shown on screen 48 of FIG.
An Internet browser, such as NTERNET EXPLORER, can be used with cell 50 to be executed, for example, to execute in a browser frame and / or web page. In other embodiments, the JAVA applet and / or MICROSOFTActiveX
An ActiveX control that uses an image spreadsheet and / or an image spreadsheet generator 10 or a portion of the image spreadsheet generator 10 can be used. At that time, such a JAVA applet or Act
The IVEX control relies on a web browser or other Internet-based, World Wide Web-based, or network-based application, so that the user can image from any browser or network platform compatible with JAVA or ActiveX. Indirect access to spreadsheets.

The unary operation is:
A cell formula is available, such as an inverse function that replaces the image with its negative, as well as a root function, such as the exp () function as a power of the base e of the natural logarithm, a square root function sqrt () An expression of a root function such as is available. Filter operations are supported, such as color filters, grayscale filters, and intensity enhancement filters, which generate false color enhancements to distinguish specific features, such as cancer or contrast enhancers. be able to.

Other operations can be performed on the images in the cell, such as thresholding using a threshold function. For example, as shown in cells A1 to C1 in FIG.
An image 72 is transformed in a magnetic resonance imaging (MRI) sequence from an underlying blood vessel image 76 as shown in cell 78, resulting in a brain image 8
A zero is shown in cell 82. For example, B1 = SEGME
A SEGMENT function is called, such as NT (A1) and C1 = SEGMENT (B1). Additionally, images 72-76 are added and thus overlap in three dimensions with the combined images displayed in other cells, such as cell B2.

The maximum intensity projection, ie, MIP (maximum intensity), implemented as a cell function
y projection to MaxIP and minimum intensity projection, that is, MiniIP (minimum in
Additional operations may be performed, such as a strength project. For example, FIG.
States that a magnetic resonance (MR) image 84 of vascular tissue
R Stored in a file named VASCULAR. Function SEGMENT Using a vessel segmentation named VESSEL (), image 88
Is generated in cell 90 labeled A3 by the formula: A2 = LOAD IMAGE ('MR VASCULA
R ') A3 = MIP (SEGMENT VESSEL (A
1)) The above equation is implemented in cells A2 and A3. In this example,
The column dataset corresponding to the 3D image 84 is loaded into cell A2, and cell A3 contains the maximum intensity projection (MIP: m) of the dataset and the results of the vessel segmentation technique.
maximum intensity projecti
on) is set to display the composite.

Accordingly, other functions and imaging operations include edge detection, low / high pass bandpass filters, and noise reduction; as well as threshold enhancement; gray level windowing; image enhancement such as edge detection. Low / high-pass filtering, blurring, and color modification; segmentation such as static and motion-based operations; morphing; stereo correspondence; Comparison operations; cropping / masking; object recognition using, for example, neural networks and classifiers; image archive manipulation; video editing; Beauty, interactive mouse control rotation, such as the conversion of scaling and image, it is preferable to perform the inter other image computation, and the like. Additionally, the video data may be multi-dimensional and have at least one
One time axis and at least one spatial axis,
For example, a one-dimensional (1D) graph of test results, 2D-X
Includes line images and animations of 3D MRI images. Thus, the opened image spreadsheet provides a platform for generating, processing, and editing video sequences for, for example, medical and non-medical applications.

In addition, functions and operations may be embedded, such as functions and operations that invoke other functions or operations as described above when generating cell A3 of FIG. Further, as defined below, the term “=” indicates the assignment and / or reassignment of cells and variables indicative of cells. Other known formulas, programming, and / or operational conventions may be used. For example, the assignment operation is alternatively invoked by the symbol ": =", and the Boolean operation is invoked by the "=" symbol without a colon. For example, according to FIG. 2, the image 64 in cell A2 is generated by the following Boolean expression: IF COUNT = 0 THEN A2: = (C1-A
1) ELSE A2: = B1-A1 This formula is used in a program that calls the image spreadsheet and uses the control parameter COUNT. Thus, such a logical programming method implemented with a known numerical spreadsheet is implemented with an open image spreadsheet.
Additionally, the image spreadsheet is MICROS
It may be implemented in other applications, such as a word processor such as OFT WORD and a database such as MICROSOFTTACESS using OLE, DCOM and / or Active X technology.

Registration If a patient motion occurs during the acquisition of two images, the subtraction of the images as described for cells A1, C1 and A2 in FIG. 2 results in the subtraction of a non-corresponding area. So
Inaccuracies arise in terms of anatomy. The image spreadsheet generator 10 corrects this problem by using a registration technique to correct the image, for example,
By translation or morphing, the solution is made to correspond exactly in terms of anatomy before performing any mathematical operations between the images. That is, the registration process evaluates the movements that the subject is undergoing and corrects for such movements when operations between images are performed.

The registration achieves coherency of the anatomical structure of the multiple data sets. To achieve anatomical coherency, the point PA in the volume A is the point PB in the volume PB, which corresponds to the same region of the patient, and the point A maintained by the image spread generator 10. The mapping registered with point B (position correspondence) has anatomical coherency between each data set.

In the predetermined MR data acquisition, since the head is fixed during the data acquisition, registration (correlation of positions) is sufficiently performed as in the examination of neurons.
Is done. However, during data collection of other parts of the human body, movement of the patient cannot be avoided; for example, between different data collections, peripheral parts including movement of the patient table with the patient resting on the table. Inspection. One technique used is automatic rigid body registration, in which automatic rigid body conversion is performed by the image spreadsheet generator 10 and / or the graphics system 14, Multiple tests are registered. For example,
The FUSION task card (ie, a useful function used by the graphics generator 12 having the graphics system 14) contained within the task card 18 can be used for this purpose. By using an interactive selection of the corresponding points between the two volumetric tests, a full automatic registration can be performed to match the two tests based on pixel / voxel values, and so on. Can be properly aligned using the registration marker 56, which is invisible on the displayed image spreadsheet and the cells of the image spreadsheet are default set. Such automatic registration is
It is useful in MRA applications because of the very similar nature of multiple examinations and pixel / voxel statistics of ordinary body structures, as shown in cells A1-C1 of FIG. Thus, the user and / or automatic registration matching techniques (e.g., a neural network or classifier) align the images during pre-validation with another image manipulation, as described above.

Another related technique is volume rendering based registration, where the usefulness of interactive and real time volume rendering provides rigid body registration based on visualization. Using such a registration method, the volume image 7 of FIG.
Two studies are volume rendered at the same time, such as 2, 76, 80. User and / or automatic registration matching techniques, including neural networks or classifiers, interactively attempt to minimize overlap between each image. Real-time volume rendering is used to perform such interactive registration. In an alternative embodiment, a transform that transforms one volumetric body to match the other volumetric body can be used to perform automatic warping registration, thereby providing a non-rigid It can compensate for the movement of the human body.

Other known medical image registration techniques may be used, such as mutual information maximization (MMI) techniques, interactive closest point (ICP) techniques to match and manipulate images in such image spreadsheets. Used to do.

Method of Operation The image spreadsheet system 10 works with a graphic generator 12 that uses the method shown in FIG. The graphic generator 12 receives the image data at step 92 and generates a grid or spreadsheet at step 94 (eg, FIG. 2).
-3, cells A1 to A3, B1 to B
3, C1-C3, etc.). Along with the cells that are displayed,
It will be appreciated that more cells may be created and the remaining cells may be scrollable and displayed as in known spreadsheet systems. The image data, which is an image sequence having an appropriate image delimiter such as a header indicating which data corresponds to which image, is stored in step 9
After receiving at 2, the image spreadsheet generator 10 divides the image display area 48 at least equal to or approximately equal to the number of screen portions or segments as cells. For example, two to four image columns are displayed in at least a 2 × 2 format, five or six image columns are displayed in at least a 2 × 3 cell format, and seven to nine images are displayed in at least a 3 × 2 format. It is displayed in a three-cell format, and so on. Reformatting, such as row and column addition,
Selectable by the user, with empty rows and columns,
Extra cells are automatically generated and supplied to replace and display the results of cell manipulation.

The method then proceeds to step 96.
The images are loaded into cells so that the registration of each image is matched with the registration marker 56. Then, using the image loaded into the cell, the method receives a user command at step 98 and manipulates the image, the image including the formula entry at step 100 and the display control at step 102. This display control includes
Includes windowing, zooming and panning functions, as well as options, including linking a set of cells simultaneously, and including an image spreadsheet control at step 104 that includes print and save commands.

For the expression entry in step 100,
For example, by moving a highlight window that highlights the edge of the currently selected cell, or otherwise by clicking on the desired cell
The user selects a cell. Then, the user
Select the INE OPERATION command and DEF
The INE OPERATION command is displayed on screen 4
8 or a specific keystroke, mouse button, or pull-down window command. A pop-up entry window 52 appears, in which the user enters the desired operation or formula / symbol.

After defining some operation for one or more cells, the user may perform an operation entered by an appropriate command, such as a predetermined keystroke, or via a pull-down menu. , You can run an image spreadsheet. The status of the execution of the operation is GUI4
6, through, for example, changing colors on the screen or by outputting the word "EXECUTED".

During or after the formula entry in step 100 or the display control in step 102, all image spreadsheet controls are executed in step 104, for example, by the SAVE button,
Save the selected cells in the image spreadsheet and / or current settings, images and / or formulas. A new image spreadsheet for the same patient and / or exam can be generated, or a new image spreadsheet for a different patient and / or exam can be executed by step 104, , The method can loop back to step 92 to obtain image data. The user proceeds to step 10
At 6, the current image data can be retained or discarded before new image data is obtained.

If the currently loaded data can be manipulated, the method loops back to steps 102-106.

In an alternative embodiment, step 104
Can include commands to transfer the results of manipulating the current image in the image spreadsheet to another task card, such as a virtual endoscope application. By clicking or other means, the user selects the desired set of cells and then
The desired task card is activated to process the selected cell.

[0051] Additionally, step 104 may include:
It has commands to match image data to be imported in the future, for example, an image spreadsheet template or script (not necessarily identifying the patient). Such an image spreadsheet script can then be loaded and applied to new image data or inspection. For example, FIG.
According to the above, by performing a process of removing the background tissue from the input MRA data by the image subtraction formula in the cell A2, some compatible input image data is substituted for the image data for the images 58-66. Can be used.

Alternative Embodiment In an alternative embodiment, the size of the grid of cells shown in FIGS. 2-3 can be varied. A cell can contain original scanner data, a single mathematical formula, or a multi-instruction program.
Every cell in the image spreadsheet is used as an operand for such a calculation. The problem of resolution mismatch is handled automatically by automatic interpolation. In addition, each of the cells can have different visualization options, such as interactive cell-by-slice viewing of cells, maximum intensity projection (MIP).
projection or simulated X-ray rendering, volume rendering, etc. Interactive parameters vary depending on the type of visualization. For example, during viewing
The user can select parameters for slice, pan, zoom, window change, and leveling, and in the case of 3D viewing, can add controls for camera position, parameters, and the like.

In another alternative embodiment, visualization yoking is a method in which visualization in different cells is performed by viewing parameters in one cell of a group of yoking (cooperatively associated). The change is performed or the change is performed so that the joking can be performed. Viewing parameters include window, leveling, zoom, and pan for 2D viewing; and viewpoints for 3D viewing, camera parameters, clipboard, transfer function,
And so on, as well as filtering and other image conversion techniques. For example, according to FIG. 2, cells A1, A2, B2, C
Each of the one image 58-66 can be yoked to vary within the one cell visualization parameter that changes the visualization parameters of the remaining cells that have been yoked. In this example, cell C
The change in resolution to zoom in on the portion of the image 62 in the image spreadsheet generator 1
0 changes the resolution in cells A1, A2, B1,
A corresponding portion of each image 58-64 can be zoomed in.

In another alternative embodiment, interactive measurement and yoking of the region of interest (ROI) are performed. If interactive measurements such as distance, area, volume, histogram / intensity attributes, etc. are made on one cell of a yawked group, the same measurement will be reflected on other members of the group. For measurements based on histogram / intensity, the results will be different for each volume. The region of interest (ROI) selection and pointer location, or other graphical indicators for features in the image, are similarly yoked.

Image spreadsheet generator 1
0 runs a complete programming language interpreter for symbols and formulas in the cell. Such symbols and expressions are single-line arithmetic, binary arithmetic, or
Not limited to unary functions, such symbols are temporary variables, loops, definitions of functions and other types of high-level flow control and computational constructs. Symbols can include calling external functions and entire programs supplied by the user. Individual symbols and the entire processing sheet can be stored and recalled at the user's discretion.

The data in a cell is not limited to having a single scalar quantity at all spatial locations. The data set may be a vector-evaluated data set in which the information stored at the spatial location comprises an arbitrary length list of data elements. For example, the application of a gradient operator to a volume results in a volume where each spatial location is assigned three floating point values.

Operations are overloaded between the modality and type of the function, ie, the image spreadsheet generator 10 automatically determines the appropriate operation from the symbol and underlying data. For example, for different operations, such as scalar and vector operations, when a user adds two scalar-evaluated volumes, the sum of scalar quantities is used to produce a scalar-evaluated result.
However, when two vector-rated volumes are added, another type of operation that performs vector addition is used, and the result is a vector-rated volume. The type compatibility of the operands in the symbol is automatically checked, and if the operands are compatible, the symbol is converted to the image spreadsheet generator 1
Evaluated by 0. On the other hand, the image spread generator 10 notifies the user of the error and / or type mismatch via the image system 14,
Do not allow operations to be performed.

According to the foregoing, a novel image spreadsheet generator 10 and method has been disclosed using an advantageous embodiment. However, many modifications and alternative embodiments are possible without departing from the spirit of the invention. For example, while the preferred embodiment describes an image spreadsheet, configuring an image database operating with the image spreadsheet generator 10 as described above is entirely within the spirit of the invention. It is. Accordingly, the present invention has been described by way of examples rather than limitations.

In short, an image spreadsheet is
The spreadsheet metaphor is extended to two-dimensional (2D) and three-dimensional (3D) images and image sequences, with the mathematical operations performing image operands and cells in the image spreadsheet to which the image data is assigned Is used. The image spreadsheet runs on an imaging system that has a processor for displaying a graphic user interface (GUI) on a display. With the image spreadsheet generator, the GUI displays a spreadsheet grid with a plurality of cells (each cell can be associated with an expression) with an associated registration marker.
The image spreadsheet generator responds to the image data with the associated registration data,
Displaying a plurality of images (aligned and oriented to match the registration markers of each cell) within a portion of the plurality of cells. An image spreadsheet generator converts an image by executing a formula in a cell in response to user input. The predetermined image conversion function is customized by the user for each cell,
A transformed image can be generated in the cell. For example, using image subtraction, a magnetic resonance angiography (MRA) image (with an image indicating that a contrast enhancer has been injected into the vasculature) is transformed, thereby removing the background vasculature. , The flow of the contrast enhancing agent is displayed. Two-dimensional images, three-dimensional images, and text can also be displayed in cells.
A dog ear icon, pop-up window, or pop-up user interface in the cell can be actuated to selectively display one of the plurality of images associated with the first cell.

[Brief description of the drawings]

FIG. 1 is a diagram showing an entire system for executing an image spreadsheet.

FIG. 2 shows an image spreadsheet for performing magnetic resonance angiography.

FIG. 3 shows an image spreadsheet that performs 3D segmentation.

FIG. 4 is a flowchart showing a method of operating an image spreadsheet.

[Explanation of symbols]

10 Image Spreadsheet Generator 12 Graphic Generator 14 Graphic System 16 Processor 18 Task Card 20 Image Symbol Generator 22 Interface 24 Graphic User Interface (GUI)
Manager 26 Application program interface (API) 28 Interface server 30 Other system 32 Hard copy server (HcS) 34 Printer 36 Database server (DbS) 38 Online storage 40 Processing server (PrS) 42 Memory 44 User interface (UI) server 46 Input And / or output (I / O) device 48 main viewing window 50 spreadsheet cell 52 window 56 registration marker 58 set of blood vessels A1, B1 and C1 cells 60-62 images 66 contrast enhancer

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Alok Gupta United States of America New Jersey East Brunswick Thornton Coat 1 (72) Inventor Ali Bani-Hashemi United States of America New Jersey Bell Mead Catskill Court 119 (72) Inventor Bernhard Geiger United States New Jersey Pre Innsboro Tamarron Drive 8202 (72) Inventor Arun Krishnan United States New Jersey Pre Innsboro Ravens Crest de Live 42 (72) Inventor Mariapan Nader United States New Jersey Pre Innsboro Fox Run Drive 2113 (72) Inventor Rama Ramaraji United States Country New Jersey Day ton Jeremy DRIVE 23

Claims (21)

[Claims] 1. An imaging system for generating an image spreadsheet, comprising: a processor for displaying a graphic user interface (GUI) on a display; and a spreadsheet grid having a plurality of cells with associated registration markers. The graphic user interface (GU
An image spreadsheet generator for displaying in I), wherein each cell can be associated with a formula, wherein the image spreadsheet generator stores the plurality of images in a plurality of cell parts. Responsive to image data having associated registration data for display, the plurality of cells comprises the plurality of images aligned and oriented to match the registration marker of each respective cell. The image spreadsheet generator,
An imaging system, responsive to user input, executing a predetermined set of equations in a first set of cells and converting a first image to the first set of cells. 2. The image spreadsheet generator according to claim 1, wherein the image spreadsheet generator performs a predetermined image conversion function customized by a user to generate an image converted from the first image to a second set of cells. Imaging system. 3. The imaging system according to claim 2, wherein the predetermined image conversion function is a mathematical function. 4. The predetermined image conversion function includes thresholding, gray level windowing, image enhancement,
3. The imaging system of claim 2, wherein the imaging system is selected from the group comprising: segmentation, morphing, stereo correspondence, visualization, comparison operations, cropping / masking, object recognition, image archive manipulation, and video editing. 5. The imaging system according to claim 1, wherein the image data includes a plurality of magnetic resonance imaging (MRI) frames. 6. An imaging system according to claim 5, wherein the MRI frame corresponding to the magnetic resonance angiography (MRA) image has an image showing a contrast enhancing agent injected into the vasculature. 7. The imaging system of claim 6, wherein the MRA image shows a temporally continuous portion of the contrast enhancer injection. 8. The image spreadsheet generator executes a subtraction formula that filters a first image of the vasculature without the contrast enhancer from a second image of the vasculature with the contrast enhancer. The imaging system of claim 7, wherein a formula indicates a flow of the contrast enhancing agent in the vasculature. 9. The imaging system of claim 1, wherein the image spreadsheet generator generates a two-dimensional image in the plurality of cells. 10. The imaging system according to claim 1, wherein the image spreadsheet generator generates a three-dimensional image in the plurality of cells. 11. The imaging system of claim 1, wherein the image spreadsheet generator generates text in a plurality of cells. 12. The imaging system of claim 1, wherein the image spreadsheet generator generates a pop-up window via a graphic user interface (GUI) to facilitate entry of user input for each cell. 13. The image spreadsheet generator generates a dog ear icon in a first cell and selectively displays one of a plurality of images associated with the first cell using the dog ear icon. The imaging system of claim 1, wherein the imaging system is actuatable. 14. The image spreadsheet generator generates a pop-up user interface (UI), which is actuated to selectively display one of the plurality of images associated with the first cell. The imaging system according to claim 1, which is capable. 15. The imaging is selected from a group comprising magnetic resonance (MR), computed tomography (CT), positron emission tomography (PET), single photon emission CT (SPECT), and 3D ultrasound. The imaging system of claim 1, wherein the imaging system is generated from a modality. 16. The image spreadsheet generator performs a yoked visualization of a first image, such that the image spreadsheet generator converts the first image into a first set of cells that are to be yoked. 2. The imaging system of claim 1, wherein the conversion is performed on each image displayed in at least the second cell according to the imaged visualization parameters. . 17. An imaging system for generating an image spreadsheet, comprising: a processor for displaying a graphic user interface (GUI) on a display; and wherein the graphic user interface (GUI) has an associated registration marker. An image spreadsheet generator for displaying a spreadsheet grid having a plurality of cells, wherein each cell is associated with a formula and can display a multi-dimensional image, wherein the image The spreadsheet generator converts the first image into the first set of cells in response to a set of formulas in the first set of cells, the image spreadsheet generator corresponding to the multidimensional image. To respond to the image data An imaging system for automatically aligning the images via transformation of the first set of cells before applying the set of equations to the first set of cells. 18. Aligning transformation (aligning tra)
nsformations) is a rigid transformation (rig)
18. An imaging system according to claim 17, wherein the imaging system is an id transformation. 19. The alignment transformation may include warping (warpi).
18. The imaging system of claim 17, wherein the system is an ng) transform. 20. The imaging system of claim 17, wherein the image spreadsheet generator provides anatomical coherence between different images displayed in different cells of the image spreadsheet. 21. A method for executing an image spreadsheet, comprising: displaying a spreadsheet grid having a plurality of cells on a graphic user interface (GUI); and associating at least one of the plurality of cells with a formula. Processing image data having associated registration data to display a plurality of images in portions of a plurality of cells; aligning and orienting a plurality of images within each of the cells; A method comprising: matching a registration mark of a cell; and converting the first image to a first set of cells according to at least one equation.