WO2000010451A1

WO2000010451A1 - System and method for spectral topography of mammalian matter using white light illumination

Info

Publication number: WO2000010451A1
Application number: PCT/US1999/019004
Authority: WO
Inventors: Jeremy M. Lerner; Sandor G. Vari
Original assignee: Cedars-Sinai Medical Center
Priority date: 1998-08-19
Filing date: 1999-08-18
Publication date: 2000-03-02
Also published as: AU5493499A

Abstract

A multispectral topography system (12) and method is provided whereby an image of mammalian matter (20), illuminated with white light (43), is transmitted to a multispectral imaging prism and mirror spectrograph (30) which substantially simultaneously spectrally disperses the transmitted light. The system then digitizes and manipulates the spectral image for further processing. The processor (70), typically resident in a computer system, processes the digital spectral data with a hybrid neural network to provide a diagnosis of the matter. The matter could be stained, pathological specimens to be magnified and analyzed by a standard, optical, transmission microscope (40), or, alternatively, in vivo (live) matter whose image is transmitted by another conventional white light image transmitter, such as an endoscope. The system is also capable of displaying the transmitted visual image (58) as well as the spectral image (38) in conjunction with the diagnostic output.

Description

SYSTEM AND METHOD FOR SPECTRAL TOPOGRAPHY OF MAMMA TAN

MATTER USING WHITE LIGHT ILLUMINATION

FIELD OF THE INVENTION

This invention relates to clinical pathology and more particularly to systems and methods that assist in the automatic analysis and diagnosis of diseased cells and tissue.

BACKGROUND OF THE INVENTION

Pathologists study samples of tissue and cells for the presence of malignancies and other diseases and abnormalities. As described below, recently, microscopy, spectroscopy and digital image processing, three traditionally distinct disciplines, have been coalescing to result in clinical pathology tools that more rapidly, automatically and accurately assist the pathologist to analyze and diagnose these conditions than was possible with conventional microscopy alone. 1. Microscopy

In one method of practicing conventional histopathology and cytopathology, after a biopsy of tissue or a smear of cells is removed from the suspect area of the patient and prepared on one or more slides, the pathologist studies the specimen under a light, or transmission, microscope. Tissue biopsies are often sliced into very thin sections and stained using one of, or a combination of, well-known staining techniques, such as H and E (hematoxylin & eosin) staining. Cell smears are similarly stained. Areas of abnormality are visually compared with representations of known disorders to determine whether and what disorders are presented. The fluorescent microscope is another diagnostic tool often used by pathologists. Instead of passing incandescent, "white" light through the specimen, the illuminating "high energy excitation" light, from a source such as a Xenon or mercury lamp, is passed through one or more filter sets that passes only certain wavelengths of light onto (not through) the specimen. In response to the light incident upon it, the specimen, usually, but not necessarily, stained, then variously fluoresces, or produces lower-energy emissions of measurable wavelengths (colors) and intensities, thus providing diagnostically valuable information about the tissue or cells.

Fluorescent microscopy techniques can be classified into two categories, namely, those that involve endogenous (or natural) fluorophores, or reactions, and those that are exogenous (originating from outside the specimen). The former includes two types of reactions, namely, autofluorescence, whereby tissue compounds, such as elastin and collagen, naturally fluoresce without any augmentation from outside the specimen, and fluorescent tagging, or labeling, of a cell. In this latter technique, a molecule, such as a peptide, protein or antigen, that attaches to a highly specific target is "tagged" with a fluorophore such as fluorescein in order to detect whether there is a positive interaction. In one method, for example, a fluorescein conjugated monoclonal antibody is used to identify a specific antigen. This technique is often used to assist in the identification and diagnosis of nuclear-based diseases, such as viruses, but may also be used to identify bacterium and malignant tissue, for example. Another technique in this category, fluorescent in situ hybridization ("FISH"), takes advantage of either a) a paired-nucleotide interaction between a labeled probe (the "antisense strand") and endogenous mRNA (the "sense strand"), or b) a protein-protein interaction, whereby proteins are labeled and incubated with tissues that contain target binding proteins or receptors. Exogenous reactions also includes general fluorescent staining, such as the application of a auramine/rhodamine preparation to smears suspected for disease, such as tuberculosis. Such staining can highlight a specific feature such as a nuclear membrane from cytoplasm. 2. Image Processing The increased processing power, speed and miniaturization of digital systems and computers have revolutionized the field of pathology. While the pathologist previously relied exclusively on the microscope and his or her own eyesight and experience to diagnose pathological disorders, current imaging and processing technologies have greatly enhanced the accuracy and speed with which today^'s pathologist may diagnose. For example, the charge-coupled device (CCD) array has been coupled to the microscope to enable the high resolution capturing of data representing microscopic images. These video images may be digitized by an analog-to-digital (A-D) converter, then displayed on a display, magnified or otherwise manipulated, and stored on other digital media. The video image data may also be further processed by a variety of image processing softw.are. Neural networking, which is a non-linear, system that sorts out patterns from the data with which it is presented and learns from discerning and extracting the mathematical relationships that underlie the data, is one such processing structure. Applied to pathology, neural network systems compare specimen data collected by a CCD array to learned patterns of data representative of healthy and diseased tissue and cells in order to automatically characterize and assess the specimen for particular disorders. 3. Spectroscopy Spectroscopy is the study of the spectral characteristics of objects, and more particularly, the study of the component parts (individual wavelengths) of the light of objects and the intensity of those wavelengths. It has long been used as a tool in the field of chemistry for identifying elements since each element possesses it own unique spectra. Since it was first realized that it can provide detailed information about both the chemical and physical nature of matter, spectroscopy has shown great promise for the field of medical diagnostics as well. For example, it has been demonstrated that certain spectral characteristics, such as fluorescence, indicate the presence of a malignancy or the metabolic condition of tissue. This information can be correlated to the object's location in the target field of view. It is often possible to determine the boundaries of indistinct edges by correlating spectra with "clustering" of spectral objects. Thus, this field has recently become a valuable partner with microscopy in order to analyze the spectral characteristics of a given suspect pathological sample. Unfortunately, the potential for spectroscopy to enhance, and even revolutionize, the field of medical pathology has not yet been fully exploited. First, as discussed, the conventional fluorescent microscope uses one filter to block all but a single wavelength of light to excite the specimen, and another filter that permits only the reemited light (and blocks the higher energy excitation light) to pass to the optical output. Since the specimen is usually stained with a fluorophore that fluoresces at a single wavelength in the spectrum, this method provides useful diagnostic information about the specimen at this single wavelength only.

In order to gather more, and more meaningful, information about the specimen, the entire image must be captured again at a second wavelength through a second set of filters. This process is repeated many times until the desired number of spectral frames collectively obtain the "spectral envelope" of the object. Each frame is stored in a computer and the composite image can analyzed by the processing software described above and/or displayed on a display. This is known as multispectral imaging ("MSI").

However, this particular technique for MSI is impractical for numerous reasons. First, it requires the availability of numerous, costly filters. Second, exchanging filter sets in and out of a fluorescent microscope is time consuming. Third, the specimen must normally be stained with numerous dyes that cause the specimen to fluoresce at each excitation wavelength, thus risking the occurrence of specimen "bleaching," a phenomenon that can ruin or degrade the diagnostic value of the specimen.

Several new automated filter systems are available, such as rotatable filter wheels, acousto-optic tunable filters (AOTF), liquid crystal tunable filters (LCTF), and the interferometer, all of which capture an entire image at each wavelength sequentially until the entire spectrum has been acquired. The interferometer has been particularly useful in the field of karyotyping. However, these systems are very costly, and the data processing of these images is an enormous task considering that a typical 512 by 512 pixel array, capturing, for example, 80 wavelengths, results in 21 MB of data per scene. Due mainly to complex and time consuming data processing, it is not uncommon for investigators to feel compelled to reduce the number of wavelengths acquired. Although 80 wavelengths may appear to be a large number, it is not uncommon for analytical chemists to acquire up to 1024 wavelengths simultaneously with off-the-shelf CCD detectors in a laboratory setting. Thus, acquiring and storing so many consecutive frames is impractical for analysis of specimens that are prepared with many fluorophores (which also risks photo-bleaching the specimens), or when computational speed is required or desired. In addition, the physical cost of these instruments is extremely high and shows little evidence of decreasing. Further, the above methods require the target object to be stationary and suffer no chemical change due to the environment during spectral acquisition.

Point spectroscopy is one known method of obtaining spectral data from an object. This technique captures the entire spectrum of, as its name suggests, only a single small point of an object at a time. In order to be practical and meaningful for the field of medical pathology, however, a spectroscopy system must be capable of spectrally dispersing, displaying and analyzing an entire specimen, or at least substantial sections of a specimen. Thus, point spectroscopy is not an ideal solution for obtaining diagnostically useful information about cells and tissue.

In sum. there is a need for a low cost system capable of rapidly and efficiently obtaining multispectral imaging data of mammalian matter, including stained pathology specimens, that can be manipulated in order to diagnose such matter for the presence of disease and disorder. It would be particularly desirable to obtain this data with the application of simple white light illumination to the mammalian matter. It would also be desirable to obtain this diagnostic data from mammalian matter presented in other forms, such as unstained and in vivo biological matter.

SUMMARY OF THE INVENTION The present invention addresses these needs by providing a system and method for automatically assessing mammalian matter, notably one or more cells and tissue, for evidence of disease. The system includes an image transmitter having an optical output and a source of white light that illuminates the matter, the transmitter adapted to transmit a transilluminated or reflected image of a section of the matter to the optical output, a multispectral imaging spectroscopy subsystem connected to the optical output, wherein the subsystem substantially simultaneously spectrally disperses the transmitted image into multiple component wavelengths to create a spectral image, and a processor that processes the spectral image to provide diagnostic data representative of the image. It should be understood that the light of the transmitted image of the specimen may be formed in numerous ways. In one way, the white light from the source transmits through the specimen (transillumination), as would typically be the case for thin specimens, such as cellular smears. For thicker specimens, however, such as a thick and dense tissue section, where the light transmitted through specimens is either minimal or nil, the light may be reflected off of the specimens to the optical output. However, in either method of image transmission, the use of a white light source eliminates the need for light filters to collect spectral data and other costly and specialized equipment needed to obtain multispectral imaging of mammalian matter. Further, the image transmitter can be either a lens based image magnification device, or a non-lens based, non-magnifying system, such as a fiber optic bundle device (i.e. an endoscope), thus providing an extremely versatile tool and method.

In a more detailed embodiment, the multispectral imaging spectroscopy subsystem includes an imaging spectrograph having an entrance slit that permits the passage of light from a slice of the transmitted image of the section of the matter and a spectrum dispersing prism and mirror arrangement that disperses the light passed through the entrance slit into multiple component wavelengths of a predetermined spectral range to create a spectral image, and a first charge-coupled- device (CCD) camera coupled to the spectrograph that acquires and prepares the spectral image. The preparation of the acquired image may entail several steps, namely digitizing the image and pre- processing the digitized image with appropriate algorithms, as is well known in the art of image processing. A computer, including a data processor, processes the prepared image and provides diagnostic data representative of the slice of the image. The system assesses matter that is removed from the mammal and prepared as a specimen on a slide. However, it also has the potential for assessing in vivo matter. It is believed that in some situations, the diagnostic system of the present invention may be capable of obviating the need for a biopsy or smear procedure, by enabling direct spectral analysis of the suspect matter without removing it from the patient.

This system provides several important advantages. The spectrograph of this invention eliminates the need for sequential capture of multiple wavelengths of a given portion of a specimen in favor of capturing the entire spectrum simultaneously. This enables rapid data processing which contributes to the system real-time diagnosis capability. The system also provides a low cost, efficient system capable of automatically assessing the pathology of cytopathology and histopathology specimens via multispectral acquisition of images of those specimens whose images are illuminated and magnified with white light and transmitted to the optical output via a standard transmission, or white light, microscope. Tlie optical output of the microscope may also include a standard camera interface, such as a "C-mount" connection for rapidly connecting and disconnecting the imaging spectrograph thereto.

In a more detailed embodiment, the transmission microscope includes an automatic x-y stage capable of controllably translating the slide for sequential MSI acquisition of the entire specimen. After one slice is acquired by the system, the computer-controlled x-y stage automatically sequences, or moves, the specimen so that the image of an adjacent slice of the specimen can pass the image- acquiring, entrance slit for spectral dispersion and acquisition of that slice. This process repeats until the entire sample, or as much as is desired, has been acquired and investigated.

In yet a further embodiment, a second CCD camera is provided in order to capture video images of the magnified sections of the specimen from the transmission microscope and to provide the image for display and/or further processing. With this further embodiment, a beam directing assembly may be disposed between the microscope and the spectrograph in order to alternatively direct the magnified optical output to either the first CCD camera or the second CCD camera. In this way, the system is capable of providing the pathologist with two diagnostic tools in one, one being traditional video imaging and the other being spectral topography data, the latter provided in the form of graphical display, called spectral graphs, in tabular form, text (on screen or as a printout) that contains diagnostic, prognostic or suggestive information (after such data is operated on by the processor) and/or a combination of all of the above.

In an alternative embodiment, a beam splitter cube may be disposed between the microscope and the spectrograph in order to simultaneously direct the optical output from the microscope to both the spectrograph and the second CCD camera. In one particular embodiment of the invention, when the transmission microscope is set at 40x, and the image acquiring slit of the spectroscopy subsystem is approximately 5 mm long, the area of the sample submitted to the entrance slit is 1.25 μm wide by 125 μm long. In this particular embodiment the spectrograph is capable of acquiring spectral data at approximately each 0.5 wm along the slice. Each such 0.5 μm by 1.25 μm wide section is called an "object," and in the present system, a maximum of 240 objects can be captured simultaneously from each slice of the target sample. Further, each spectrum for each object contains up to 740 wavelength data points in the 380 to 800 nm range, with a spectral resolution of 1 nm at the 400 nm wavelength to approximately 15 nm at 700 nm.

In the preferred embodiment, the data processor comprises a neural network. The neural network algorithm may alternatively comprise an unsupervised neural network (USNN), a supervised neural network (SNN), or a combination of the two. The USNN operates to automatically recognize and map (i.e. to train for) the presence of "fingeφrint" spectra. The SNN may be implemented to automate the system and perform routine autocalibration.

One preferred method of spectrally analyzing mammalian matter for the presence of disease based upon a spectral analysis of the moφhologic and physiologic deviation of the matter from the norm includes illuminating the matter with a white light source, transmitting one of a transilluminated and reflected image of the matter to a multispectral imaging spectrograph, spectrally dispersing the transmitted image through a prism and mirror arrangement into multiple component wavelengths of a predetermined spectral range, acquiring the spectrally dispersed image of the multiple component wavelengths, preparing the acquired image for processing, and processing the prepared image to provide a diagnosis. Typically, a single slice of the matter, comprised of multiple objects, will be transmitted to the optical output for spectral dispersion, acquisition, preparation and processing. In such case, the method provides that after the first slice is processed, an adjacent slice of the matter may be moved into position for analysis of that slice. This process may be repeated until all slices, or a desired portion, of the matter have been processed to obtain a complete analysis.

Other features and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. la is basic schematic showing the automated multispectral topography system of the present invention wherein white light transilluminates through mammalian matter: FIG. lb is a variant of FIG. la. wherein the white light incident on the matter results in a reflected image at the optical output.

FIG. 2 is a schematic of a more detailed embodiment of the present invention; FIG. 3 is a flow chart describing one preferred method of the present invention; FIG. 4 is an exemplary table, called a "chromagram." of data presenting the spectral moφhology of small slices of three typical cells shown with a color coding table. FIG. 5 is a chromagram representing the spectral moφhology of sections of cells from Pap smear slides containing cells preclassified as "normal";

FIG. 6 is a a chromagram representing the spectral moφhology of sections of cells from Pap smear slides containing cells preclassified as "koilocytes"; FIG. 7 is a chromagram representing the spectral moφhology of sections of cells from Pap smear slides containing cells preclassified as "ASCUS"; and

FIG. 8 is a chromagram representing the spectral moφhology of sections of cells from Pap smear slides containing cells preclassified as "malignant".

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT The invention summarized above and defined by the enumerated claims may be better understood by referring to the following detailed description, which should be read in conjunction with the accompanying drawings. This detailed description of particular preferred embodiments, set out below to enable one to build, use and practice particular implementations of the invention, is not intended to limit the enumerated claims, but to serve as particular examples thereof. The particular example set out below, which decribes the experimental results obtained from analysis of stained Pap smear slides, details a preferred specific implementation of a system that provides multispectral topography of stained mammalian matter, namely, one that provides automated spectral data acquisition, analysis and diagnosis of images that are suspected to be diseased with simple white light illumination of the matter. The invention, however, may also be applied to other types of mammalian matter, such as histopathology and cytopathology specimens and in vivo matter.

Before describing the invention in greater detail, namely, the method of multispectral analysis employed by the present invention, its various components and experimental results, we now explain the theory behind the application of spectral topography as a tool for revealing the "hidden moφhologies^" and physiologies of tissue and cells and its implications for medical pathology.

I. Morphologic Effects On Normal And Abnormal Cells and Tissue

It has long been recognized that the study of the form, such as size and shape, of cells and their environments (cytomoφhology) and tissue biopsies can reveal diagnostically valuable information regarding the state of those structures. General Biologic activity is reflected best in the cellular structures of the nucleus. Functional activity is reflected mainly in the moφhology of the cytoplasm. The "healthy,^" baseline moφhology of cells may be considered a reference level, called euplasia. which is the form and structure of normal cells absent stresses from pathologic processes. In euplasia, key nuclear structures may appear under a light microscope as round or rounded, uniform and having regular patterns of nuclear components such as chromatin, and possess a degree of predictability from one nucleus to another. In tissue analysis, predictability from one cell to another cell of the same type would be expected. Numerous moφhological effects of processes associated with carcinomas and precursors to the same have been identified. For example, significant irregularities in the shape of nuclear membranes of cells, disarray in the structural orderliness and shape of chromatin and parachromatin (the pale areas of the nucleus), the enlargement and irregularity of the shape of nucleolus, and importantly, an increased ratio of nuclear area to cytoplasmic area (N/C ratio) are some of the recognized moφhological factors which favor a finding of cancer.

Unfortunately, to date, such moφhological analysis under a light microscope has, by itself, been of limited value in diagnosing diseased cells and tissues for several reasons. First, moφhological changes in cells and tissue occur in varying degrees. Many changes cannot be recognized (either clearly or at all) through the eyepiece of a light microscope even by an experienced pathologist, or even with the manipulation of video images of slides of specimens. These hidden moφhologies are due to either the nature of the changes or the stage of moφhologic activity. Second, to date, there has been no observable absolute moφhologic feature of cancer - or a malignant criteria - that when present, unequivocally reveals that the particular cell or tissue under observation is cancerous or, when absent, means that there is no cancer.

The present invention is employed to reveal the hidden moφhologies of suspect cell and tissues that are typically stained, by illuminating them with white light, and analyzing their spectral content with the aid of a unique multispectral imaging spectrograph and sophisticated processing algorithms.

II. The Method and Components

1. Push Broom Multispectral Imaging

There are several traditional ways of acquiring multispectral information from a remote field of view. As discussed above, one method acquires an entire field through a series of wavelength filters. The number of filters will be equivalent to the number of wavelength data points needed to identify the spectral signatures of each component in the field. The field of view is fixed so data cannot be acquired if the object moves or changes in any way.

In another method developed for remote earth monitoring and implemented by the present invention, the system acquires a small slice of the field and passes it through a specialized wavelength dispersive spectrograph that acquires the entire spectrum of the slice simultaneously. To cover the entire field it is necessary to move on to the next, adjacent slice. Concatenating each acquisition enables the entire field to be covered. This method is often referred to as "push broom^" spectral topography because the sample is "pushed" across the spectrograph entrance aperture, or slit, and is, in numerous ways, more versatile and efficient than the prior described method of multispectral imaging acquisition. The present invention implements the push broom multispectral image acquisition methodology to analyze suspect mammalian tissue and cells, to reveal their hidden moφhologies and physiologies, and to ultimately provide clinical diagnosis of the tissue and cells. 2. The Components FIG. la shows the basic components of the system. An image transmitter 1 includes a white light source 2 and an optical output 3. The light source 2 illuminates suspect mammalian matter 4, which is typically stained, whose image is transmitted to the optical output 3. An imaging spectroscopy subsystem 6 substantially simultaneously spectrally disperses the transmitted image into multiple component wavelengths of a given range. This method of spectral dispersion uses a spectrograph originally designed for remote, telescopic earth monitoring and astronomy and is currently in use for both applications in military and civilian environments. The original spectrograph was patented to Warren et al. (patent no. 5,127,728) and was designed for use in the infrared wavelength range of 3 to 15 μm and is incoφorated herein by reference. For the life sciences applications of the present invention, the optics were redesigned for use in the primarily visible 360 to 800 nm wavelength range. A processor 8. such as a PC computer loaded with the appropriate software, operates on data representative of the spectrally dispersed image to ultimately provide a diagnostic output 10 relating to the condition of the matter 4. This diagnostic output 10 could be provided at a computer screen, at a printout, or at any conventional output device. As seen, FIG. la depicts an image transmitter arrangement whereby the white light source 2 transilluminates through the matter 4. This method is typically used when, for example, a standard transmission microscope is the image transmitter for a relatively thin specimen, such as a cell smear, or a very thin tissue biopsy. FIG. 1 b shows an alternative arrangement for the image transmitter 1, whereby the white light from the source 2 is reflected off of the matter 4 and to the optical output 3. The reflection method may be used in several situations, one being where the suspect tissue sample is too thick for the white light to meaningfully transilluminate therethrough, and another being where the matter to be analyzed is not removed from the patient, but is rather in vivo.

FIG. 2 is a schematic of a more detailed preferred embodiment of the present invention. The primary components of this multispectral topography system 12 include a conventional transmission microscope 40, a prism and mirror imaging spectrograph 30, a first CCD camera 34, a processor 70 and a diagnostic output device 80.

The conventional transmission microscope 40, found in many laboratories and research facilities, has an eyepiece 41 and an optical output 42, which has a standard camera interface, such as a video port with a "C-type" mount. The white light source 43 of the microscope illuminates and magnifies a section of a specimen 18 which has been prepared on a slide 16. A spectroscopy subsystem connected to the microscope 40 includes the prism and mirror, wavelength dispersive imaging spectrograph 30 having an entrance slit 32 for permitting the image of a slice 20 of the specimen 18, the slice itself comprising many "objects" 21, to pass therethrough and into the spectrograph 30. and a first CCD camera 34. The novel and modified spectrograph 30 provides good image quality over a broad range of operating wavelengths simultaneously, allowing large spectral intervals to be surveyed without moving any of the elements of the system.

A beam directing assembly 60, also called a "beam splitter," constructed with a "flip" mirror 62, provides both a video image and spectral acquisition. Thus, when the mirror is flipped in one direction, the spectrally dispersed light is focused on, and acquired by. the first CCD matrix array 36 of the camera 34. In the present embodiment, the system, the analog image stored on the array is manipulated, or pre-processed, by the spectral image pre-processor 37, with digital signal processing. The processing may be implemented in software or DSP hardware. It should also be understood that some CCD cameras digitize the analog image with an analog to digital (A to D) converter prior to the pre-processing step. The manipulated, pre-processed, spectral image can then be fed into the processor 70, as discussed in detail below, and/or displayed on a display 38, such as a CRT or LCD screen or any other conventional graphical display that is well known in the art.

When the flip mirror 62 is rotated to a second orientation, the visual, white light, microscopic image, i.e. a video image, is captured by the matrix array 56 of the second CCD camera 54. This image is also typically processed with advanced image processing algorithms 57 to manipulate the image for high quality display on a conventional display 58. Image pre-processing algorithms may include smoothing, normalization, background subtraction, principal component analysis (PCA) and partial least squares (PLS), for example. In this way, both spectral imaging data and visual imaging data of the specimen can be displayed for the pathologist, stored, analyzed and/or further manipulated. In an alternative embodiment not shown, a beam splitter cube replaces the beam directing assembly 60 to send light simultaneously to the spectrograph 30 and the white light CCD camera 54.

In the preferred embodiment, the spectrograph 30 uses a prism made of inexpensive flint glass. The support and body of the unit may be formed in cast aluminum, metal plate, or any other material that provides for lightweight and for enhanced rigidity. The optical system is fully ray-traced. The system, with or without the beam director 62 is easily assembled to the optical output of a conventional microscope, typically a video port, with the use of a standard "C-type" mounting or any other acceptable connecting means.

In one embodiment, when the transmission microscope is set at 40x magnification, each image slice 20 captured through the slit 32 and by the spectrograph 30 is approximately 1.25 μm wide by 125 μm long. In this configuration, the system is capable of acquiring spectral data at approximately each 0.5 μm along the slice 20. Each 0.5 μm by 1.25 μm wide section is called an "object" 21. Based upon the resolution of image acquisition of the CCD camera used, up to 240 objects can be captured simultaneously from each slice 20 of target tissue. Each spectrum for each object contains up to 740 wavelength data points from 380 to 750 nm. Sequentially scanning the entire image of the specimen 18 is achieved by scanning the microscope stage under computer control in the histology or cytopathology setting. The first CCD matrix array detector 36 collects individual spectra along rows of pixels from objects located in the entrance slit 32. This format is sometimes referred to as an "open image" because there are no restrictions on the area of the object to be examined and is certainly the least expensive and most flexible method for pathology samples subject to microscopic examination. All spectra of all objects in a slice are acquired simultaneously in milliseconds, depending on signal strength.

The processor 70. which will typically be part of a computer system, such as a PC, contains a powerful neural network 72 that provides near instantaneous recognition of the spectral fingeφrints of the objects 21 of the specimen 18. In one preferred embodiment, there are up to 240 objects, each as small as 0.5 μm, that the neural network can substiuitially simultaneously process. The relatively small file size for each acquisition greatly enhances computation speed and simplifies memory management.

FIG. 3 shows how the push-broom methodology is applied to the present invention for multispectral analysis of a tissue sample or a cell smear on the standard white light transmission microscope 40. In step 100, a prepared slide of either a slice of a tissue sample or smear, i.e., the specimen, is placed onto the transmission microscope 40 for illumination, magnification, and transmission at an optical output, such as the video port found on many of today's research and pathology-grade transmission microscopes. The microscope presents a particular field, or section, of the magnified specimen to the entrance slit 32 of the spectrograph 30. In step 102, a single slice 20 of the section, containing up to 240 objects 21 , passes through the slit 32 and, in step 104, strikes the first curved surface of the prism, is refracted, strikes the second surface, and exits to strike the spherical mirror as is described in the Warren et al. patent. The wavelength dispersed light then returns through the prism to be focused and stored onto a first CCD matrix array 36, step 106. Over 80% to 90% of all light, over the entire wavelength range, is transmitted through the system. The analog image is then pre-processed, or manipulated, as discussed above in order to improve its appearance. The processor then processes the spectral data representing the spectral envelopes of each object in the slice, step 108, using the customized hybrid neural network that sorts out the spectral moφhological patterns in the objects of the cell(s). The system then inquires into the status of the acquisitions, step 1 10. If only a single acquisition is needed or desired, the process halts in step 112 and the diagnostic data may be reviewed by the pathologist. However, if as is usually the case, an additional slice of the specimen is to be spectrally dispersed and analyzed, i.e. the answer to inquiry 1 10 is "no." and the specimen slide is moved by the x-y stage of the microscope 40 to permit the passage of light representing a slice adjacent to the previously dispersed slice, step 1 14. Then, the process reverts to step 104 for spectral dispersal, acquisition and analysis of the objects in that second slice. This process is repeated until the entire or enough of the specimen is analyzed in this fashion. Once this sequential acquisition process is complete and the spectral data for each object is stored in the appropriate memory "bins." as described below, the processor may then synthesize this data to produce a complete diagnosis for the specimen. The advantages of utilizing push broom spectral data acquisition for cellular analysis are numerous. First, the diagnosis is practically instantaneous. To acquire enough spectral data using alternative techniques would require the creation of huge digital files prior to data processing. A full acquisition according one preferred embodiment of the present invention includes 240 spectra, each with 740 data points, yet is only 185KB. A file of this size is easily handled, resulting in very rapid data processing and decision making. Further, the entire system costs much less than other methods of analysis, such as the interferometer.

This method also enables fast low-resolution assessments to determine gross parameters prior to high-resolution scans. In addition to acquiring multispectral topographical maps, the system can "paint" the video image of the specimen with false color "fingeφrint" spectra indicative of normal and diseased tissue.

3. The Neural Network

Traditional mathematical algorithms perform calculations sequentially, delivering results based on linear transformations. A neural network (NN) 72 performs calculations in parallel, in an analogous way to the human brain, to perform non-linear transformations. To date the processing of the manipulated image has been implemented with an unsupervised neural network (USNN). In one preferred embodiment, however, the system would be powered by two neural network components: a USNN component and a supervised neural network (SNN) component. The USNN collects each spectrum from each object, characterizes it, sorts it, and places it into a "bin" of unique spectral signatures. It can thus be employed to automatically recognize and map (i.e. to train for) the presence of "fingeφrint" spectra, i.e., to identify bins containing identical spectral signatures. The SNN may then determine the special features that differentiate the spectra in each bin from those in other bins. Thus, the system can then compare the presenting spectra of the objects of the matter under analysis with the map to identify the presence of the disorder. The USNN can be thought of as a "filter" or "sieve" that characterizes and sorts each spectrum as it is collected and places it into its moφhological bin or "class" of similar spectral signatures. The process is analogous to alphabetizing the words in a book by combining those that are the same or have a common root. By the end of this procedure, the number of bins represents the number of spectral objects defined by their spectral-moφhological characteristics. The process is referred to as "digital chromatography" because the function is almost identical to the process used in analytical chemistry for the separation of mixtures of chemical compounds. The big difference is that the USNN program adds the extra dimension of providing spatial information by mapping a particular class of spectrum back to the sample itself.

The operator can decide to combine some spectral classes, such as background features, or eliminate others. This is achieved by "thresholding" either by the user or the computer. This is a process that enables the USNN to delineate, identify and separate identical and non-identical spectra within user defined limits of the application. Tlie incoφoration of human "wet neurons" to set up the thresholds enables human experience to control the operating limits of the software. Tlie USNN becomes a "white box" rather than a "black box" because the sensitivity and the operating parameters of the network are available to operator influence at all times.

Once the USNN has self-trained, with initial well-qualified samples, the USNN compares each newly acquired spectrum against the spectral classes and categories it recognizes. The USNN identifies matches, near matches, and no match spectra in the new acquisition(s). Each class of spectrum is coded into a few bytes of data and stored in memory. Every future spectral acquisition is similarly coded and compared to the stored data. This process reduces a spectrum of 740 data points to a block of memory only a few bytes in size, consequently recognition of hundreds of spectra is performed in near real-time. Training can take up to two minutes for complex materials and a few seconds for simple spectra. After training, depending on signal strength, recognition may take less than one second.

As stated above, the processor 70 may incoφorate an SNN component to operate in conjunction with the USNN. The SNN can identify the special features of each bin. automate the system, and may even perform routine autocalibration. Those skilled in the art have begun to recognize that combining these two neural network architectures is very effective for controlling many variables, some or all of which can change with time. Most humans tend to overlook or "accommodate" certain changes. In a multi-parametric system, such as cellular pathology, this is very dangerous because a change in one variable can result in non-linear affects elsewhere in the process and compromise the accuracy of an assessment. The two component system is designed to form a transparent alliance to automatically ensure that conditions necessary for accurate repeatable diagnoses are not compromised by changes in temperature, mechanical maladjustment or operator error.

Ideally, one or, more likely, many systems of the invention will be trained and retrained by USNN's in order to spectrally characterize the moφhologies of the cells of all presenting pathological disorders and diseases, thus creating a "library" of spectral fingeφrints representative of those presenting conditions. It is expected that, eventually, the entire and finite universe of categories of pathological disorders will be spectrally characterized by USNN's. When this milestone is achieved, systems could then be equipped with SNN's which include databases containing the all pre-trained, spectral fingeφrints of the various possible conditions. When a specimen whose pathology is unknown is presented to the system of the present invention, the SNN will rapidly compare and map its spectral fingeφrint with that of all fingeφrints stored in the database or, in the appropriate category of the database. With the availability of powerful, fast and low cost personal computer systems with large amounts of memory capacity, a single personal computer system will be capable of storing the an entire "spectral library" thereby providing automatic diagnosis of the condition of any cellular or tissue specimen presented to it.

It is understood that those skilled in the art may develop protocols to enable specific histological assessments to be selected from a computer menu to simplify and minimize continuous human interaction with system, perform autocalibration. continuously monitor the status of the entire system, and record user operations.

HI. Example - Application to Cytopathology

The novel system and method of the present invention has been applied in the following experiment on the cells of prepared and previously-diagnosed Pap smear slides, in order to demonstrate the invention's capacity to assist in the identification and diagnosis of cervicovaginal disorders, including Human Papilloma Virus (HPV).

Effective diagnosis of this disorder has become a subject of increasing importance to pathologists because it is believed to be a precursor of cervical cancer. However, the diagnosis of this disorder is subject to numerous limitations in technology. Pap smears currently screen for the early detection of cervical cancer and other abnormalities by preparing slides of stained, exfoliated cervical cells for analysis on a light microscope. W ile a valuable screening tool. Pap smears detect only 50-80

% of the abnormalities subsequently found by histopatho logical examination of cervical tissue. Further, this disorder is not successfully tested for with the conventional PAP smear screening. The cells in the prepared slides used in the experiment fall into one of four previously diagnosed categories under the Papanicolaou classification system, namely, (1) normal or healthy; (2) koilocytes (a squamous epithelial cell and the classic manifestation HPV-infected cells) (3) ASCUS

(an acronym for "Atypical Squamous Cells of Undetermined Significance", and is a saclike spore case, consisting of a single terminal cell); and (4) malignant. It should be understood that while the Papanicolaou classification system is considered by some to be less precise than the newer classification system known in the art as the "Bethesda System," since the prior nomenclature was used during the experiment, it will be repeated throughout.

The experiment identified the degree of koilocyte moφhological characteristics, evidencing the presence of human papilloma virus (HPV), that are present in the other types of cells using the multispectral topography system of the present invention. It revealed that it is possible to identify these four categories of cells by evaluating their spectral content with the system of the present invention together with a standard light microscope.

In particular, nine prepared, stained, Pap smear slides containing normal, koilocyte, ASCUS and malignant cells were illuminated and magnified by a standard transmission microscope, and were subjected to the imaging spectrograph of the present invention using the method described above. Individual cells, comprising nuclei and cytoplasm, were passed through the spectrograph slit to simultaneously capture a large number of objects per acquisition, each object, as stated above, being approximately 0.5 micrometer in height by up to 1.25 micrometers in width. To keep computation time down, the bulk of the empty area surrounding each single cell was eliminated and only 100 out of the possible approximately 240 objects were analyzed. The unsupervised neural network was used to automatically characterize the spectra and code them into "objects". Each spectrum is associated with a series of digits such as "object 1 1000". Each digit in the series is analogous to a decimal place, so the five digits indicated would be referred to as a spectrum of the fifth order. If a particular spectrum is identified with an exact match of all attributes of a known cell characteristic it would be indicative of a very strong hit, i.e. that the portion of the cell having the spectrum matching the spectrum of, for example, normal or malignant cells, is, in fact normal or malignant. If the spectrum located starts with the same series of digits but misses one or more of the last digits, it is indicative of a similar but not complete match.

In this experiment, most, if not all. cytoplasm objects start with a "0" and all nucleus objects start with a "1". Target detail is contained in the roll-up through the cytoplasm into the nucleus. The neural network was trained on a "typical" koilocyte named "Koilo 1 ". This protocol was designed to determine whether spectral fingeφrints might identify objects associated with the presence of human papilloma virus (HPV) and are 1 ) not apparent to the naked eye; 2) not present in normal cells, and 3) occur in cells classified as ascus or malignant. FIGS. 5, 6, 7 and 8 are charts called "chromagrams" that spectrally characterize the objects in the cells analyzed. FIG. 4 is an examplary chromagram showing the spectral moφhology of small sections of three fictitious cells and a matching color coding table and is provided in order to assist in the reading and inteφretation of the chromagrams developed in the experiment. At the top of each table the name and identity of each cell is shown in a box outline. The slide from which the cell was found is given in parenthesis. Thus, "norm 12(1)" is the 12th normal cell analyzed and is found on slide 1. The original chromagrams (shown here uncolored) are color-coded (yellow, blue, green, red and white-no color) to symbolize five specific spectra. Tlie spectral objects were color coded according to their presence in the "Koilo 1(3)" cell, the cell on which the neural network was trained. In particular, as seen in FIG. 4, yellow spectral objects are those that the neural network classified as starting with the numeric string "1_0_0" and are commonly found in the nucleus of most, if not all, cells. Blue objects are those that the neural network classified as starting with the numeric string "1_1_0" and appear in or near the nucleus of koilocyte but are not evident in the majority of normal cells. Green objects are those that the neural network classified as starting with the numeric string "0_1_2" and red objects are those that the neural network classified as starting with the numeric string "0_1_0." Red and green objects originate in the cytoplasm immediately surrounding the nucleus. White objects ("0_0_0" objects) occur in the mass of the cytoplasm. It should be understood that the color coding in this experiment was arbitrarily chosen and is intended merely to be a visual aid in identifying the spectral characteristics of each object in a cell.

In characterizing the cells in the nine slides, three decision criteria were used. 1 ) If certain spectra occur in all cells, they are of no diagnostic value. 2) If a certain spectrum or spectra occur exclusively in koilocyte. or a ceil known to be infected with HPV, they are diagnostic of koilocyte or HPV. 3) If certain spectra or combination of spectra occur in koilocyte, ASCUS and malignant cells, but not in normal cells, then the abnormality is indicative of a common origin or condition.

One general observation from these charts is that normal cells can be visually seen to have small nuclei compared to koilocyte, ASCUS and malignant cells. Specific observations follow:

Recurring Spectral Patterns Found in "Normal" Cells (20 Specimens) - FIG. 5

Recurring Spectral Patterns Found in Koilocytes (15 samples) - FIG. 6

1. 15 of 15 cells present presented alternating green and red bands in areas surrounding the nucleus. Sometimes in extended areas of couplets, for example, (Koilo 10, 1 1(4) and Koilo 16(5)).

13 of 15 koilocytes presented blue bands mostly at the periphery of the nucleus but also embedded in yellow bands of the nucleus.

6 of 15 presented multiple alternating and yellow bands and couplets.

10 of 15 present blue and green couplets.

7 of 15 present 3 or more adjacent green bands.

Recurring Spectral Patterns Found in "ASCUS" cells (15 samples) - FIG. 7

13 of 15 presented one or more red/green couplets.

6 of 15 presented blue bands

ASCUS 13 (9) presented a nucleus the size of a "normal" cell and the red green alternating bands typical of a koilocyte.

ASCUS 15 (9) presented multiple green bands and a red band. No alternating bands were present.

ASCUS 1 (3) presented all the characteristics found in a koilocyte.

4. 10 of 15 malignant cells presented breaks in the nucleus with evidence of alteration of the nuclear spectra similar to cytoplasm spectra.

These charts provide considerable evidence that diagnostically valuable but hidden-to- the-eye moφhological patterns occur in each of the cells, and that these patterns can be rapidly and inexpensively observed with the use of standard transmission microscopy together with the multispectral topography system of the present invention. The present invention provides a low cost, rapid and efficient diagnostic tool and method that will assist in the identification of these "early risk" disorders together with (in addition to) the Pap smear screening, all in a single test. Having thus described the basic principles and exemplary embodiments of the invention, it will be apparent that further variations, alterations, modifications, and imDrovements will also occur to those skilled in the art. For example, it is understood that the system may be designed with modified optics to capture a wider or different range of wavelength spectra than that identified above. Further, it is understood that the present invention is not limited to the use of transmission microscopes. The spectrography subsystem may operate with any type of white light image transmitter capable of mammalian cell and tissue analysis, such as any lens based, telescopic, system, or a fiber optic based imaging system, such as an endoscope. Further, the tissue and cells are not limited to prepared slides. For example, the system could automatically analyze the spectral characteristics of cervicovaginal tissue during an actual gynecological examination by connecting the spectroscopy subsystem and computer to a colposcope, for example. Accordingly, the foregoing discussion is intended to be illustrative only; the invention is limited and defined only by the various following claims and equivalents thereto.

Claims

1. A multispectral topography system for automatically assessing mammalian matter for evidence of disease, the system comprising: an image transmitter having an optical output and a source of white light that illuminates the matter, the transmitter adapted to tr<_nsmit one of a transilluminated and reflected image of a section of the matter to the optical output; a multispectral imaging spectroscopy subsystem connected to the optical output, wherein the subsystem substantially simultaneously spectrally disperses the transmitted image into multiple component wavelengths to create a spectral image; and a processor that processes the spectral image to provide diagnostic data representative of the image.

2. A multispectral topograohy system for automatically assessing mammalian matter for evidence of disease, the system comprising: an image transmitter having an optical output and a source of white light that illuminates the matter, the transmitter adapted to transmit one of a transilluminated and reflected image of a section of the matter to the optical output; a multispectral imaging spectroscopy subsystem connected to the optical output, the subsystem including an imaging spectrograph having an entrance slit that permits the passage of light from a slice of the transmitted image of the section of the matter and a spectrum dispersing prism and mirror arrangement that disperses the light passed through the entrance slit into multiple component wavelengths of a predetermined spectral range to create a spectral image, and a first charge-coupled-device (CCD) camera coupled to the spectrograph that acquires and prepares the spectral image for processing; and a computer having a data processor that processes the prepared spectral image and provides diagnostic data representative of the slice of the image.

3. The system of claim 1 or 2, wherein the matter to be assessed is in vivo.

4. The system of claim 1 or 2, wherein the image transmitter is a lens-based image magnification system.

5. The system of claim 2, wherein the matter is a pathology specimen prepared on a slide and the image transmitter is a transmission microscope that transmits the image of a magnified section of the specimen to the optical output.

6. The system of claim 5, wherein the microscope includes an x-y stage capable of controllably and sequentially moving the specimen so that the entrance slit of the spectrograph permits the passage of light from adjacent slices of the specimen.

7. The system of claim 5, wherein the optical output of the microscope includes a standard camera interface and the imaging spectrograph connects to the interface.

8. The system of claim 2, further including an image director disposed between the image transmitter and the spectroscopy subsystem and having a first output in optical communication with the entrance slit of the spectrograph and a second output in optical communication with a second CCD camera that captures an observed image of the section of the matter.

9. The system of claim 8, wherein the image director is a beam splitter that alternatively directs the image of the matter to the spectrograph or the second CCD camera.

10. The system of claim 8, wherein the image director is a beam splitter cube that simultaneously directs the image of the matter to the spectrograph and the second CCD camera.

1 1. The system of claim 2, wherein the matter is a tissue biopsy.

12. The system of claim 2, wherein the matter is a cell smear.

13. The system of claim 2, wherein the matter is a stained Pap smear.

14. The system of claim 2, wherein the data processor comprises an unsupervised neural network.

15. The system of claim 2, wherein the data processor comprises a supervised neural network.

16. The system of claim 2. wherein the data processor comprises an unsupervised neural network component and a supervised neural network component.

17. A multispectral topography system for automatically assessing a pathology specimen prepared on a slide for evidence of disease, the system comprising: a transmission microscope having an optical output and a source of white light that illuminates the specimen, the transmitter adapted to transmit one of a transilluminated and reflected magnified image of a section of the specimen to the optical output; a multispectral imaging spectroscopy subsystem connected to the optical output, the subsystem including an imaging spectrograph having an entrance slit that permits the passage of light from a slice of the transmitted image of the section of the matter and a spectrum dispersing prism and mirror .arrangement that disperses the light passed through the entrance slit into multiple component wavelengths of a predetermined spectral range to create a spectral image, and a first charge-coupled-device (CCD) camera coupled to the spectrograph that acquires and prepares the spectral image for processing; a second CCD camera that captures an observed image of the section of the specimen, the second camera being connected to optical output of the microscope; an image director disposed between the microscope and the spectroscopy subsystem and having a first output in optical communication with the entrance slit of the spectrograph and a second output in optical communication with the second CCD camera; and a computer having a neural network data processor that processes the prepared spectral image and provides diagnostic data representative of the slice of the image.

18. A multispectral topography system for automatically assessing mammalian matter for evidence of disease, the system comprising: means for illuminating the matter with white light; means for transmitting one of a transilluminated and reflected image of the matter to an optical output; means for simultaneously spectrally dispersing the image into multiple component wavelengths of a predetermined spectral range; means for acquiring and preparing the dispersed image from the dispersing means; and means for processing the prepared, spectrally dispersed image with a neural network to provide a diagnosis of the specimen.

19. A method of spectrally analyzing mammalian matter for the presence of disease based upon a spectral analysis of the mo╧åhologic and physiologic deviation of the matter from the norm, the method including: illuminating the matter with a white light source; transmitting one of a transilluminated and reflected image of the matter to an optical output; spectrally dispersing the tr._nsmitted image into multiple component wavelengths of a predetermined spectral range; acquiring the spectrally dispersed image of the multiple component wavelengths and preparing the image for processing; and processing the prepared image to provide a diagnosis.

20. The method of claim 19, further including providing a visual display of the transmitted image obtained at the optical output.

21. The method of claim 20, further including providing a visual display of the prepared image.

22. The method of claim 19. wherein the preparing of the spectrally dispersed image includes digitizing the image and pre-processing the digitized image with digital signal processing.

23. The method of claim 19, wherein the processing implements a neural network.

24. A method of spectrally analyzing a pathology specimen for the presence of disease based upon a spectral analysis of the biological and functional deviation of the specimen from the norm, the method comprising: illuminating the specimen with a white light source; transmitting a magnified image of a section of the specimen to an optical output; permitting the light of a slice of the image to transmit through an entrance slit, the slice of the image comprising multiple objects; spectrally dispersing the transmitted image of the slice through a prism and mirror arrangement into multiple component wavelengths of a predetermined spectral range; acquiring and preparing the spectrally dispersed image of the multiple component wavelengths for processing; and processing the prepared image with a neural network to classify each object in the slice of the image into one of a preset number of categories indicative of the pathological condition of the object.

25. The method of claim 24. further including providing a first visual display containing the spectrally dispersed image and a second visual display of the observed image.

26. The method of claim 24. further including: (a) translating the specimen along an axis to permit the light of a slice of the image adjacent to the slice of the image previously transmitted to transmit through the entrance slit;

(b) spectrally dispersing the transmitted light of the adjacent slice of the image through a prism and mirror arrangement into multiple component wavelengths of a predetermined spectral range;

(c) acquiring and preparing the spectrally dispersed light of the adjacent slice of the image for processing; and

(d) processing the prepared light of the adjacent slice of the image with a neural network to classify each object in the adjacent slice of the image into one of a preset number of categories indicative of the condition of each object.

27. The method of claim 26, further including repeating (a), (b), (c), and (d) until the desired number of slices of the image of the specimen have been processed and providing a clinical diagnosis therefrom.