Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/0002—Inspection of images, e.g. flaw detection
  • G06T7/0012—Biomedical image inspection
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B21/00—Microscopes
  • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
  • G02B21/365—Control or image processing arrangements for digital or video microscopes
  • G02B21/367—Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T3/00—Geometric image transformations in the plane of the image
  • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
  • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10056—Microscopic image
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20021—Dividing image into blocks, subimages or windows
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
  • G06T2207/30024—Cell structures in vitro; Tissue sections in vitro

Definitions

• the present invention relates to systems for mapping surgically excised tissue specimens to correspond anatomically to the orientation of the excised tissue with respect to the patient from whom the tissue was removed, and particularly to a system for obtaining images by optical sectioning of a surgically excised tissue specimen and anatomically mapping these images.
• a system for obtaining images by optical sectioning of a surgically excised tissue specimen and anatomically mapping these images In certain surgical procedures, particularly those where specimens are excised for the determination of cancerous tissue, such as in Moh's micrographic surgery, considerable surgical time is occupied by histopathology in order to provide numerous frozen sections through the specimen which are assembled to produce maps of the excision showing the cancerous regions.
• the system (which includes the method which carries out optical sectioning and mapping of surgically excised tissue specimens) may utilize a macroscopic imaging 5 means for capturing a multi-spectral full field macroscopic image of the excised tissue which has been placed in a cassette marked with respect to the orientation of the excised tissue with respect to the patient.
• the imaging means captures images either by white light reflectance or fluorescopic imaging techniques or with confocal imaging means (see U.S. Patent No. 5,788,639 or 5,880,880).
• a translation stage may be coupled to the cassette for moving it in o orthogonal directions such that different parts of the specimen may be presented to the objective of the imaging means.
• the objective of the imaging means is translatable in a direction orthogonal to the directions of translation of the stage so as to image different slices on or within the specimen.
• a two-dimensional image block may be obtained by two- dimensional scanning of a beam on, or inside, the specimen.
• a map of the image blocks may be formed by scanning in a serpentine stepwise motion of the cassette with respect to the objective or vice versa.
• a single image block represents a map element. Each block is identified by a row and column in the bit map and the image of each block is digitally stored in memory and may have a pointer associated with each block.
• compression such as with a median filter, a compressed composite image is obtained and stored in memory.
• a mosaic of the map elements constitutes a full resolution image.
• a frame may have a group of these elements of the mosaic and thereby displays the image in high resolution sufficient for the physician to locate regions of interest, for example, containing cancerous cells.
• the composite image is a macroscopic image of compressed or reduced resolution, for example, one pixel per map element obtained by pixel elimination, mean-value-substitution or median filtering, the latter being preferred.
• a frame containing the composite, compressed image or selected blocks of the full resolution mosaic can be displayed either alternately or in different windows of a screen to the physician. The display shows those blocks in the region which are marked, as by a user interface or mouse, on the composite macroscopic image. At any time different slices may be displayed.
• the regions of the high resolution image may be marked by dots of different colors or patterns as may be defined in a look-up table, such as red for cancerous, green for non-cancerous and yellow for uncertain, on the display.
• the information appears on the macroscopic image as to the location of the potentially cancerous regions and may be used by the surgeon in making further excisions for removal of the cancerous cells.
• a complete two-dimensional image with the markings and the map all oriented with respect to the patient (the anatomic map) may be displayed, if desired.
• the images in any frame either of the composite or of the map elements from the mosaic may be printed out on a color printer, or the entire two-dimensional image may be displayed.
• Fig. 1 is a block diagram of a system for optically sectioning and mapping surgically o excised tissue, which embodies the invention
• Fig. 2 is a plan view of the cassette which contains the excised tissue specimen and which is shown on a stage in Fig. 1 ;
• Fig. 3 is a view similar to Fig. 2, but with the specimen mounted in the cassette with the deep margin up, the deep margin being oriented to face the objective of the confocal 5 imaging system;
• Fig. 4 is a perspective view of the cassette showing the lid of the cassette down with the cassette cover being closed, the cassette cover being the portion of the cassette adjacent to the objective lens in Fig. 1, and the specimen being placed on the lid in the center thereof, and the cassette cover, which contains a plate or window which compresses the specimen, in the o open position;
• Fig. 5 is a showing of the serpentine path of the scan, the raster across the specimen and individual elements which make up the mosaic of elements of the high resolution or mosaic image;
• Fig. 6 is a view showing the clipping limits or vignetted part of an image elements 5 from which an image of the element is taken so as to remove blurring or reduce intensity at the image margins;
• Fig. 7 is a block diagram showing the relationship between the composite macroscopic image and the specimen, and the map of the image as shown on the composite, the figure also showing a block of map elements within an image frame, the map elements 0 being indicated on the diagram of the composite image, and the figure also showing how the block is marked by indicating the neoplasm shown therein as being cancerous, non-cancerous or uncertain or ambiguous as to its cancerous or non-cancerous nature;
• Fig. 8 is a view of the composite image showing blocks which have been marked as being cancerous but flipped to reflect the orientation in the patient which is deep margin down rather than up;
• Figs. 9A, 9B and 9C constitute a diagram of the flow chart of the method carried out 5 by the system of Fig. 1, the figures are assembled at the connectors (x) and (x) and (y) and
• Fig. 10 shows a complete dermal/epidermal margin and its location on the specimen
• Fig. 1 1 is a view similar to Fig. 10, but showing an incomplete dermal/epidermal o margin.
• a confocal imaging system similar to the system shown in Fig. 14 of the above-referenced International Publication WO 00/49392.
• the system uses as its imaging means a confocal microscope 10, preferably having the polarization and acetic acid image enhancement facilities of the above-referenced 5 International Publication WO 00/55669.
• the objective (objective lens 12) of the microscope is equipped with a focus control input 14, which may physically move the objective up and down to focus at a horizontal section (slice) within an excised tissue specimen in a tissue cassette 16.
• the cassette 16 is mounted for X and Y movement (that is in a horizontal plane), as shown by the coordinate system 18, on a translation stage 20. It may be operated by a 0 remote control 22 which provides inputs to a controller 24 connected to the microscope 10.
• the controller 24 receives the reflected signal from the specimen, provides scanning signals so as to scan the specimen and create the mosaic image. Signals are provided to the stage 20 via a translator line, marked “cassette position control", and to the objective 12 via a focus control line.
• the laser power control for controlling the power from the laser in the 5 microscope is indicated by the connection marked "laser power”.
• the controller 24 receives inputs from a computer 26 and the remote control 22.
• the imaging algorithms are carried out in the computer 26.
• the computer 26 provides serial communications over a bus for controlling the scan. It also receives a pixel clock, a line clock and a frame clock which marks the pixels individually and the lines and entire frame of o the mosaic as they are contained in the reflected signal.
• the computer 26 creates the composite macroscopic image as well as the mosaic of map elements and stores them in memory together with their associated pointers.
• the image may be outputted to a network for telepathology at a remote source, or even over the Internet, via a network interface indicated at 28.
• Control for identifying map elements, which are to be viewed in separate frames and to switch between windows or the same window for viewing the composite and high resolution frames is obtained by user input 30 which may be implemented as a "mouse" or with a touch screen.
• the image is printed out on a printer 32 or may be viewed on a terminal or display 34 with the high resolution image in the center frame window and the composite image as a map in a corner window.
• Soft controls 36 on the display 34 are also provided for marking or for viewing different sections or slices of the specimen.
• Fig. 2 shows the bottom or lid of the cassette 16. It shows a mark 54 at the top which indicates the head or top center of the head (superior margin) of the patient. There are marks 56 or blocks indicating five, ten, fifteen and twenty millimeter regions all concentric with each other, as shown in Fig. 3.
• the cassette 16, as shown in Fig. 4, has a lid 38 and a cover plate 40 which when closed compresses the specimen 42.
• Fig. 5 shows the mosaic of individual image fields or map elements which are created by scanning in X and Y step wise across a region which contains the specimen 42.
• the scan is called the "mosaic trajectory". Only the individual map elements or image fields across the first or top traverse of the scan are shown to simplify the illustration.
• the signals may be limited at boundaries Y and X so as to vignette the image thereby removing areas at the boundary which may not be unambiguously digitized in the computer, as shown in Fig. 6.
• the controller 24 which receives the reflected signals (either gray scale or in color of the tissue) determines a continuous outer boundary of the tissue shown in the image by first thresholding each of the pixels of the image.
• the pixels above the threshold are stored as "Is" in a bit map, while all other bits are stored as "0".
• the area of the bit map is divided into the blocks shown as the individual image fields in Fig. 5. The area of these blocks may constitute 80% to 100% of the field of view of the confocal imaging system, which may be from 0.4 to 1.0 millimeters.
• An "exclusive-or" operation may be performed in the controller on the bits of each block of the bit map, which if the operation results in a "1" for any block, identifies that block as being contained inside the boundary line of the tissue against the window.
• the bit map forms a map of the tissue in contact with the window or other horizontal slices further away from the window (in the z direction).
• the mosaic may be 15 by 15 individual image blocks or frames, where each frame is stored as 640 by 640 pixels.
• a mosaic is shown at 50 in Fig. 7.
• the region containing four image blocks or map elements is selected, via the controls by viewing the macroscopic image.
• the region is enlarged and shown at 58, displays in high resolution four blocks of the image mosaic. From this high resolution image the cancerous cells can be detected by the physician (the pathologist) who observes the confocal image on the display 34 or on a display which is coupled via the network 28. Then the neoplastic condition can be identified by using the controls 36 and marked on the display or map in different patterns or colors; red for cancerous, green for non-cancerous and yellow for uncertain.
• Image maps shown at 55, 52, and 57 in Fig. 7 are marked cancerous, non- cancerous and uncertain, respectively. It will therefore be apparent that there is a complete anatomical map of the specimen, as well as images of sufficient resolution to detect cells which are neoplastic.
• Fig. 8 This image serves as a tumor map and can be used to guide subsequent excisions.
• Fig. 9A illustrates the tissue loading and imaging phase
• Figs. 9B and 9C illustrate the image formation and display phase, respectively.
• the tissue is marked to indicate the tissue orientation with respect to the patient by the physician. That is, the superior margin toward the head of the patient is important as well as the lateral or sides of the excise specimen.
• This step is shown at 62.
• the tissue specimen is then placed on the lid 38 of the cassette 16 with the deep margin facing up, as shown in Figs. 3 and 4.
• the superior margin faces the indicia 54. This step is shown at 64.
• the specimen is placed on the lid 38 inside the marked grid of the lid.
• the cassette cover then is closed and imaging fluid, for example the acetic acid, is inserted into the cassette around the specimen. This step is shown at 66.
• the user inputs the size of the largest grid which circumscribes the specimen. This is the grid 56 shown in Fig. 3. This provides a tissue size input which is used in creating the mosaic of map elements as shown in Fig. 5.
• the size of marking step is indicated at 68.
• the specimen in its cassette 16 is then mounted on the stage by placing it on the receptacle indicated at 55 in Fig. 1.
• the loading step is shown at 70.
• the remaining steps of algorithm and the operations to obtain the image are apparent from Figs. 9B and 9C.
• the X, Y and Z parameters of the map are inputted automatically from the controller at 80.
• the translator (translation stage) position which corresponds to the center of the cassette is programmed into the controller, initially.
• the focus is then adjusted 5 by the user to obtain the slice or depth (in z) of interest for the particular image at 82, such as with focus control input 14.
• the stage is then sent home to a start location at 84.
• the serpentine scan is then carried out at 86.
• Each map element is then stored in memory. Since there are many bits, disk memory may be used. See 88.
• the clipping or vignetting of the image step is next carried out at 90.
• the composite or macro image is obtained either by i o pixel elimination, mean value substitution or median filtering and saved in bit map, DICOM format or similar format at 92.
• the composite is displayed, this may be in the main window or in the side window on display 34.
• the user then clicks with the mouse on the composite to a desired X, Y location and the four map elements at the X, Y location are displayed in the main window. This is step 96. In this manner, the four map elements
• the user may select to view live images of the specimen in a window on the display with the confocal microscope imaging of specimen at a desired X-Y location, and the user may manipulate imaging, such as by changing focus (depth) or laser power of the confocal microscope, or may move the translation stage with respect to the objective lens of
• the pathologist marks the image as a mapped section of the specimen which has cancerous cells.
• the margins of the specimen are thoroughly investigated and all cancerous regions marked for further excision and removal during the latter stages of the surgery.
• the marking step is shown at 96 A. If a map button is clicked on the display 34, the composite map is again displayed at step 94, and steps 96 and 5 96A may be repeated at the same or different X-Y location on the composite map.
• the mapping described above may be reactivated (repeated) for the specimen if any setting of the confocal microscope, such as focus, laser power, or other imaging parameter, is changed at step 98A. After the specimen is examined, a complete two-dimensional image with the markings and the composite map all oriented with respect to the patient may be outputted to o the display 34 or color printer 32.
• Fig. 10 shows a complete dermal/epidermal margin.
• the corrugated line in Fig. 10 represents the interface between the epidermal layer and the dermal layer.
• Fig. 11 shows an incomplete dermal/epidermal junction. The cause for an incomplete junction is due to tissue lifting from the imaging window 40. If the physician were to see image blocks as shown in the expanded region of Fig. 1 1 , the physician should mark the region as "uncertain" and investigate the area with live images focusing higher into the tissue until the entire boundary is imaged. This insures that a projection of cancer has not evaded detection.
• imaging modalities include, optical coherence tomography, multi-photon microscopy or high frequency ultrasound.
• In-vivo mapping of the lesion is also possible by using an in-vivo reflectance confocal microscope to image the surgical field rather than an excised specimen.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Multimedia (AREA)
• Optics & Photonics (AREA)
• Analytical Chemistry (AREA)
• Chemical & Material Sciences (AREA)
• Health & Medical Sciences (AREA)
• General Health & Medical Sciences (AREA)
• Medical Informatics (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Radiology & Medical Imaging (AREA)
• Quality & Reliability (AREA)
• Investigating Or Analysing Biological Materials (AREA)
• Microscoopes, Condenser (AREA)
• Image Processing (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Publications (2)

Family

ID=22594952

Family Applications (1)

Country Status (3)

Cited By (7)

Citations (5)

Family Cites Families (5)

• 2000
  • 2000-11-10 EP EP00977080A patent/EP1247246A4/de not_active Ceased
  • 2000-11-10 WO PCT/US2000/030729 patent/WO2001035325A1/en active Application Filing
  • 2000-11-10 AU AU14768/01A patent/AU1476801A/en not_active Abandoned

Patent Citations (5)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events