JP2016514869A - Method and system for analyzing biological samples by spectral images - Google Patents

Abstract

A method and system for detecting an abnormality in a biological sample with high accuracy and providing the analysis result to a doctor or the like. Receiving an image of a biological sample in the system; applying annotation information associated with a particular disease or condition to the image and spectral data to generate an actual image of the biological sample; and And applying an algorithm to the spectral data to generate a predicted image including a predicted classification of the identified disease or condition from the biological sample, and transmitting the predicted image including the predicted classification for display on a display A method for classifying a biological sample, comprising the step of:

Description

Related applications

  This application is a US patent application Ser. No. 13 / 645,970 filed Oct. 5, 2012 entitled “METHOD AND SYSTEM FOR ANALYZING BIOLOGICAL SPECIMENS BY SPECTRAL IMAGING”. “INFRARED AND RAMAN MICRO-SPECTRAL IMAGING OF HUMAN CELLS AND” filed on March 19, 2013, “Infrared and RAMAN MICRO-SPECTRAL IMAGING OF HUMAN CELLS AND” Claims priority based on US Provisional Patent Application No. 61 / 803,376 entitled TISSUE FOR MEDICAL DIAGNOSTICS. This application is a US patent application Ser. No. 13 / 507,386, filed Jun. 25, 2012 entitled “METHOD FOR ANALYZING BIOLOGICAL SPECIMENS BY SPECTRAL IMAGING”. US Provisional Patent Application No. 61 / 322,642 entitled “A TUNABLE LASER-BASED INFRARED IMAGING SYSTEM” filed on April 9, 2010, PCT (Patent Cooperation Treaty) Patent Application No. PCT / US2009 entitled “METHOD OF RECONSTITUTING CELLULAR SPECTRA USEFUL FOR DETECTING CELLUALR DISORDERS”, May 29, 2009 Filed on May 29, 2008, based on the Filed on Feb. 17, 2011 and filed Apr. 23, 2013, claiming priority based on US Patent Application No. 61 / 056,955 entitled “METHOD OF RECONSTITUTING CELLULAR SPECTRA FROM SPECTRAL MAPPING DATA”. US Patent Application No. 12 entitled “METHOD OF RECONSTITUTING CELLULAR SPECTRA USEFUL FOR DETECTING CELLUALR DISORDERS” issued US Pat. No. 8,428,320 No./994,647, US Provisional Patent Application No. 61/358 entitled “DIGITAL STAINING OF HISTOPATHOLOGICAL SPECIMENS VIA SPECTRAL HISTOPATHOLOGY” filed June 25, 2010 , 606, filed Apr. 11, 2011, "Wavelength-tunable laser-based infrared imaging system and US Patent Application No. 13 / 084,287 entitled “TUNABLE LASER-BASED INFRARED IMAGING SYSTEM AND METHOD OF USE THEREOF” and “Analyzing Specimens by Spectral Images” filed on June 24, 2011 US Patent Application No. 13 / 067,777 entitled “METHOD FOR ANALYZING SPECIMENS BY SPECTRAL IMAGING”. The contents of all of the above applications are hereby incorporated by reference.

  One problem of this technical field is that a method and system for detecting an abnormality in a biological sample with high accuracy and providing the analysis result to a doctor or the like does not yet exist.

  In the prior art, many diseases can be diagnosed by using classical cytopathological or histopathological methods including examination of nucleus and cell morphology and staining pattern. These diagnostic methods are generally performed by examining up to 10,000 cells in a biological sample and discovering about 10-50 cells or portions of tissue that may be abnormal. This discovery work is based on the subjective judgment of a doctor or the like by visual microscopic examination of cells in the sample.

  One example of classical cytology is a test method for detecting the onset of cervical disease, proposed by Papanicolaou in the middle of the 20th century and now commonly known as the “Pap test”. This test is performed by detaching the cells using a spatula or brush and depositing them on a microscope slide. Since this inspection was initially performed by rubbing the peeling brush on the microscope slide, it is also called “pap smear”. Subsequently, the cells are stained with hematoxylin-iocin (H & E) staining or Pup staining (consisting of H & E staining and several other counterstains) and visually inspected by cytologists and cell technicians using a low power microscope. .

  Micrographs of such samples often show cell aggregation, contamination with cell debris and blood-based cells (red blood cells and white blood cells / lymphocytes). Thus, the original “Pap test” had a very high rate of false positive and false negative diagnoses. Modern liquid-based methods (cell centrifugation, ThinPrep® or Surepath®) have improved cell samples by eliminating cell aggregation and eliminating confounding cell types.

  However, although the method for preparing a detached cell sample on a microscope slide has been substantially improved, the diagnostic steps in the art are generally still in the visual inspection and the results and cytologists. Rely on a comparison with a database in memory. Thus, diagnosis remains essentially subjective and causes low reproducibility between and within observers. In order to alleviate this aspect, an automatic visible light image analysis system, a new prior art, has been introduced to assist cytotechnologists during visual inspection of cells. However, the distinction between atypia and low-grade dysplasia is extremely difficult, and such conventional automated image-based methods have not yet substantially reduced the actual responsibility of cytologists.

  In classical histopathology, doctors and the like have stained tissue sections, not individual detached cells, with an appropriate method and observed them using a microscope. Doctors and the like pay attention to the overall tissue structure, cell morphology, nucleus morphology, nucleocytoplasm ratio, chromatin distribution, presence / absence of mitosis, etc., in order to detect abnormalities. The above criteria are based on morphology, and the resulting assessment is somewhat subjective. In immunohistochemistry and more recent analysis methods, tissue diagnosis based on subjective evaluation by doctors or the like is increasingly used.

  In the art, spectral methods are applied to histopathological diagnosis of tissue sections obtainable by biopsy. Data acquisition in the above technique called “spectral histopathology (SHP)” can be performed using the same spectral method as used for spectral cytopathology (SCP).

  In some methods in the art, broadband infrared (IF) or other light output is transmitted through a sample (eg, a tissue sample) using an instrument such as an interferometer to generate an interference pattern. The reflected and / or transmitted light is then typically detected as an interference pattern. Next, a fast Fourier transform (FFT) is performed on the proportionalized pattern to obtain spectral information associated with each sample pixel. Result information is referred to as a pixel spectrum.

  One limitation of the FFT-based technology process is that because it uses broadband infrared spectral radiation, the amount of radiant energy available per unit time in each band can be very low. As a result, the data available for this method process is usually inherently limited. Also, in order to discriminate between received data and background noise at such a low energy level available, a high sensitivity instrument such as a high sensitivity liquid nitrogen cooling detector is required (cooling reduces the background). The effect of infrared interference is mitigated). As a further disadvantage, such technical systems can lead to significant costs, installation space and energy usage.

  Transmits and detects infrared rays used for imaging tissue samples and other samples under atmospheric conditions for the purpose of classifying diseases for diagnosis, prognosis, treatment and / or prediction of diseases and / or medical conditions There is an unmet need in the technology for apparatus, methods and systems for In addition, there is an unmet need in the technology relating to the system and method for providing the analysis results to doctors and the like.

  Aspects of the invention relate to systems and methods for analyzing and evaluating samples, including tissue samples, with image data to classify biological samples into diagnostic, prognostic, predictive and therapeutic classes. More particularly, the present invention relates to a system and method for receiving a biological sample and providing analysis results of the biological sample data to assist in medical diagnosis.

  Aspects of the present invention provide coherent / tunable, such as broadband tunable quantum cascade lasers (QCL) or optical parametric oscillators (OPSs) designed to rapidly collect infrared diagnostic data for medical diagnostics over a wide range of discrete spectral increments. Included are methods, apparatus and systems for imaging tissue and other samples using infrared spectral information from non-coherent light sources. Processing infrared data with the SHP system provides analytical data, medical diagnosis, prognosis and / or predictive analysis.

  By using the method, apparatus and system according to the present invention, an abnormality in a biological sample can be detected earlier than when a known cytopathological or histopathological method is used.

  By using the method, apparatus and system according to the present invention, for example, a doctor or the like can easily obtain information on biological samples including analysis data and / or information on medical diagnosis.

  By using the method, apparatus and system according to the present invention, a machine learning method or algorithm for performing diagnosis, prognosis, treatment, subtype and / or predictive classification of biological samples can be trained. Furthermore, by using the methods, devices and systems described above, classification models used for medical diagnosis, prognosis, treatment, subtype and / or predictive analysis of biological samples can be created.

  By using the method, apparatus, and system according to the present invention, the confidence value of the prediction classification can be generated. The confidence value can be included in the confidence prediction image. The confidence value can also be included in the confidence prediction report.

  By using the method, apparatus and system according to the present invention, new classes and / or subtypes of cancer can be identified and assigned. Further, the predicted classification can be graded by using the above-described method, apparatus, and system. The grade represents, for example, the degree of progression of cancer. The predictive classification and grade can also be used for medical diagnosis and / or prognosis of biological samples. In addition, predictive classification can be used to associate patients with a collection of clinical data based on disease status (eg, cancer progression).

  Further, by using the method, apparatus and system according to the present invention, the collected material can be used for molecular gene sequence analysis for treatment. Furthermore, gene expression in biological samples can be annotated by using the method, apparatus and system according to the present invention.

  Additional advantages and novel features related to variations of the present invention are set forth in part in the description which follows, and by examining the following description or practicing the embodiments described therein, It will be partially clearer to the vendor.

  Aspects of the present invention will become more fully apparent from the detailed description provided herein below and the accompanying drawings, which are given by way of illustration and example only, and thus are not limited to that aspect.

FIG. 1 shows an example of a procedure for identifying a disease state using a confidence value and a differentiation value for assisting in identifying a cancer class in a biological sample in one embodiment of the present invention. This figure shows that a new class or subtype has been discovered (see well differentiated / low confidence region or poor differentiation / high confidence region). FIG. 2 shows a color photostat image as a result of performing an SHP analysis on an area where two different diagnostic areas (blue and magenta areas) in a biological sample are adjacent in one embodiment of the present invention. In this figure, the region where diagnosis is uncertain (low reliability) is shown in white. FIG. 3 illustrates, in one embodiment of the present invention, a procedure for analyzing a spectral data set from a biological sample to perform diagnosis, prognosis and / or predictive analysis of a disease or condition in addition to identifying a new class or subtype. An example is shown. FIG. 4 shows an example of a procedure for using an SHP image to identify and identify a region to be microdissected in a biological sample in one embodiment of the present invention. FIG. 5A illustrates an example of a preprocessing procedure of infrared image data in one embodiment of the present invention. FIG. 5B illustrates an example of a pre-processing procedure for infrared image data according to one embodiment of the present invention. (A) is a color photostat image showing an example of an actual image (actual annotation) in one embodiment of the present invention. (B) is a color photostat image showing an example of an SHP predicted image in one embodiment of the present invention. (C) is a color photostat image showing an example of a reliability prediction image in one embodiment of the present invention. (A) is a color photostat image showing an example of a reliability prediction image in one embodiment of the present invention. (B) is a color photostat image showing an example of a reliability prediction image in one embodiment of the present invention. (C) is a color photostat image in which a predicted image is superimposed on a clinical image in one embodiment of the present invention. (D) is a color photostat image showing an example of a reliability prediction image in one embodiment of the present invention. (A) is a color photostat image showing an example of a real image annotated based on pathology in one embodiment of the present invention. (B) is a color photostat image showing an example of a predicted image in one embodiment of the present invention. (C) is a color photostat image showing an example of a predicted image after applying true positive / true negative and false positive / false negative analysis in one embodiment of the present invention. (A) is a color photostat image showing an example of an image in which a region of interest is set (partially hardly differentiated) in one embodiment of the present invention. (B) is a color photostat image showing an example of an image in which a region of interest is set (partially hardly differentiated) in one embodiment of the present invention. FIG. 10 is a color photostat image illustrating an example of a reliability prediction image in one embodiment of the present invention. FIG. 11 is a color photostat image showing an example of an actual image in one embodiment of the present invention. FIG. 12 is a color photostat image showing an example of a predicted image in one embodiment of the present invention. FIG. 13 is a color photostat image illustrating an example of a legend of a confidence scale associated with a confidence prediction image (for example, FIGS. 10 to 13) according to an aspect of the present invention. FIG. 14A is a color photostat image showing an example of biological sample classification according to one embodiment of the present invention. FIG. 14B is a color photostat image showing an example of classification of benign and malignant tumors in one embodiment of the present invention. FIG. 14C shows the algorithm structure used in one aspect of the present invention, where A, B, C and D in the figure represent a certain organizational state, class or subtype. FIG. 15 is a color photostat image showing an example of a classification report in one embodiment of the present invention. FIG. 16 is a color photostat image showing an example of a validity report in one embodiment of the present invention. FIG. 17 is a color photostat image showing an example of a microdissection selection region in one embodiment of the present invention. FIG. 18A is a color photostat image showing an example of specifying and identifying a microdissection selection region in one embodiment of the present invention. FIG. 18B is a color photostat image showing an example of specifying and identifying a microdissection selection region in one embodiment of the present invention. FIG. 18C is a color photostat image showing an example of specifying and identifying a microdissection selection region in one embodiment of the present invention. FIG. 18D is a color photostat image showing an example of specifying and identifying a microdissection selection region in one embodiment of the present invention. FIG. 19 shows a configuration of a computer system used in the embodiment of the present invention. FIG. 20 shows an example of a computer system used in an embodiment of the present invention.

  Aspects of the invention include methods, systems and devices for classifying biological samples into diagnostic, prognostic and therapeutic classes and providing biological sample analytical data and medical diagnosis, prognosis, treatment and / or predictive analysis.

  In addition, by using the method, apparatus, and system according to the present invention, for example, a confidence value of a prediction classification can be generated. The confidence value represents the degree of confidence that the disease is present in the biological sample. For example, a confidence value indicates that a disease is present in a biological sample with 90% confidence. In another example, the confidence value indicates that the disease is present in the biological sample with 3% confidence. In one aspect of the present invention, the confidence value may be included in the confidence prediction image. For example, the confidence prediction image may include a visual representation of confidence values for all or a portion of the biological sample. The trust image may additionally be used to assist a doctor in making a diagnosis. The trust image may also be used to identify a region of interest for microdissection.

  In addition, by using the confidence image and the confidence value report, for example, it is possible to visually display the overlapping pathology in different types of diseases, the difference in the type of the disease, and the reliability regarding the overlapping pathology. This allows the physician to identify significant diseases in the biological sample and other diseases that may be present in the biological sample.

  By using the method, apparatus, and system according to the present invention, the cancer identified by the prediction classification can be graded (for example, the identified cancer class and / or subclass). The grade can indicate the degree of progression of cancer from the initial stage of progression to one that has progressed sufficiently. For example, the grade can be expressed by a number such as “grade 1”, “grade 2”, “grade 3”. Further, for example, “high grade”, “low grade”, and “medium grade” may be expressed as a range. The grade of disease may be determined by subjective interpretation of the biological sample by a pathologist. The system according to the present invention can apply grades to predictive classifications created for biological samples. The system according to one aspect of the present invention can receive a biochemical characteristic of a biological sample and use it to determine the grade of a predictive classification. The physician can receive a report that includes one or more classes and / or subclasses of the cancer identified in the biological sample, as well as the progression of the identified cancer class and / or subclass. In this way, medical classification and / or prognosis of biological samples can be performed using predictive classification and grade. In addition, treatment decisions regarding biological samples can be made based on predicted classification and grade.

  In one embodiment of the system according to the invention, the differentiation value of the sample can be defined to assist in determining the disease grade or progression. Differentiation values are a quantitative measure of disease grade or progression. Low differentiation values indicate, for example, that the disease identified in the sample has not progressed sufficiently and is in the early stages of progression and / or that there may be different types of disease. Show. When the differentiation value is medium, for example, it indicates that a specific disease is progressing in the sample. A high differentiation value indicates, for example, that a specific disease present in a biological sample is more advanced. Moreover, when a differentiation value is low, it shows that each area | region in a sample is hardly differentiated, and when a differentiation value is high, it shows that each area | region in a sample is fully differentiated.

  The poorly differentiated region may include a region of a real image where information (eg, morphological features) specified in a biological sample used to diagnose a disease is insufficient. The real image may include an annotated image by a physician indicating the type of disease that may be present in the biological sample, such as, for example, a class and / or subclass of cancer. For example, the poorly differentiated region may occur in a portion where information in an image is not clear. 9A and 9B show exemplary images that include poorly differentiated regions. The well-differentiated region may include a real image region rich in information specified in a biological sample used to diagnose a disease. For example, well-differentiated regions can occur in the part where information in the image represents a class or subclass of cancer.

  The system according to the present invention can determine whether a region in a sample is a poorly differentiated region or a highly differentiated region by analyzing annotations regarding a real image of a biological sample. For example, a physician can annotate an actual image of a sample by identifying regions of the image that are low quality, medium quality, or high quality. The system according to the present invention can also determine whether a sample region is a poorly differentiated region or a highly differentiated region by spectral analysis of a predicted image. For example, the system according to the present invention can determine a low performance spectral region. Also, the system according to the present invention can determine a high performance spectral region.

  As shown in FIG. 1, the method, apparatus and system according to the present invention is used to identify a normal region of a biological sample and the class and / or subclass of cancer present in the biological sample and / or biological sample. Can be assigned a new class or subclass of cancer. FIG. 1 illustrates an exemplary graph 100 for assisting in identifying a class of cancer in a biological sample using confidence and differentiation values in one aspect of the invention. The y-axis of the graph 100 represents level values in the range of 1 to 10, where 1 indicates a poorly differentiated sample and 10 indicates a highly differentiated sample. The x-axis of the graph 100 represents the reliability value in the range of 1 to 10, where 1 indicates that the reliability value is low and 10 indicates that the reliability value is high.

  The transition region includes the region where the disease is ongoing in the biological sample. In the early stages of progression, even different diseases are difficult to distinguish. The transition region can identify multiple diseases in the biological sample. A pure region can include a region of a biological sample in which the disease is highly advanced.

  FIG. 2 illustrates an analysis example of a transition region according to one embodiment of the present invention. FIG. 2 shows an example of a confidence prediction image (for example, the rightmost image in FIG. 2) generated by the system according to the present invention based on the analysis of the transition region of the biological sample specified in FIG. For example, the reliability prediction image represents the reliability of the system according to the present invention related to the presence of a specific class of cancer in the transition region.

  In one aspect of the invention, a new class can be identified in a biological sample if the sample is well differentiated but the confidence of the identified class or subclass type is low. Also, a new class can be identified in a biological sample when the sample is not fully differentiated but is highly reliable. High reliability is achieved by spectral analysis. For example, if the spectrum signal from the predicted image is pure (no other spectrum is mixed in the signal), the reliability is high. Conversely, if the spectrum signal from the predicted image is impure (the signal is mixed with another class of spectrum), the reliability is low.

  The new class is present in the actual image (eg, an annotated image by the physician indicating the type of cancer present in the sample) and the predicted image (eg, in the sample, as detailed in the description with respect to FIG. 3 below. It can be identified when wrinkles occur between spectral images based on spectral analysis indicating the type of cancer.

  In one embodiment of the present invention, a class or subclass of cancer that can exist in a biological sample can be identified by linking a confidence value with a differentiation value. The identification of a new cancer class will be described in detail in connection with FIG.

  FIG. 3 illustrates a biological sample analysis procedure 300 for diagnosis, prognosis and / or predictive classification of a disease or condition in one aspect of the invention. Procedure 300 can be used to identify a subclass of cancer. Procedure 300 can also be used to distinguish between normal tissue and cancerous lesions (eg, distinguishing normal tissue proximal to cancerous lesions and normal tissue distal to cancerous lesions).

  The method according to the invention includes a step 302 of receiving a biological sample. Biological samples include tissue or cellular material collected from individuals such as humans and animals. The biological sample is collected by a doctor or the like using any known method. The sample includes, for example, a microtome section collected by biopsy, exfoliated cell sample deposition, fine needle aspiration (FNA), and the like. However, the present disclosure is not limited to these biological samples and includes samples that require spatially resolved infrared spectroscopy information.

  With the method according to the present invention, various cells or tissues can be examined. These cells include exfoliated cells such as epithelial cells. Epithelial cells can be squamous epithelial cells (monolayer or stratified and keratinized or non-keratinized), columnar epithelial cells (monolayer, stratified or pseudostratified, and pili or nonpiliated) and cubic epithelial cells (mono Layer or layer, and cilia or non-cilia). These epithelial cells cover various organs throughout the body such as the intestines, ovaries, male reproductive tissues, respiratory system, cornea, nose and kidneys. Glandular epithelial cells are a type of epithelial cells that cover the throat, stomach, blood vessels, lymphatic system, and tongue. Mesothelial cells are a type of epithelial cell that covers a body cavity. Urinary epithelial cells are a type of epithelial cells that cover the bladder. Endothelial cells are cells that cover blood vessels.

  One aspect of the system according to the invention comprises a receiving module operable to receive a biological sample. In another aspect, data corresponding to a biological sample can be received. For example, the user may provide data corresponding to the biological sample.

  The method according to the invention includes a step 304 of generating a spectral image of the biological sample. The system according to the present invention collects spectral data of a biological sample and generates a spectral image. Spectral data is not limited to infrared spectral data; Raman and related technologies such as surface or chip-enhanced Raman and non-linear Raman technologies such as coherent anti-Stokes Raman, femtosecond stimulated Raman effect, visible, terahertz and fluorescence spectroscopy Appropriate data based on methods including Infrared spectroscopy includes, but is not limited to, infrared reflection measurements such as total reflection Fourier transform infrared spectroscopy (ATR-FTIR). In general, infrared spectroscopy is used because of its high fingerprint sensitivity, and this is also true for Raman spectroscopy. Infrared spectroscopy may be used, for example, on wider tissue sections and is also used to provide data that is easier to handle than Raman spectroscopy. In addition, infrared spectroscopy data is easier to obtain fully automatic and easier to analyze. In addition, infrared spectroscopy has the sensitivity and specificity necessary for the diagnosis of various tissue structures and diseases.

  In one aspect of the invention, spectral data is acquired by a physician or the like using a tunable laser-based infrared imaging system and method described in related US patent application Ser. No. 13 / 084,287. Spectral data is acquired by using an infrared wavelength tunable laser as a coherent light source. The wavelength of infrared transmission from the tunable laser is changed in discrete steps across the spectrum of interest, and transmission across the spectrum and / or reflected transmission is detected and used for image analysis. Spectral data can also be acquired by a conventional Fourier transform infrared spectroscopy (FTIR) system using a non-laser light source such as Glover, Synchrotron or other broadband light sources.

  One example of a laser that can be used in aspects of the present invention is a quantum cascade laser (QCL) capable of changing the infrared wavelength output, for example, between about 5 μm and 12 μm. The array detector is used to detect transmitted and / or reflected infrared wavelength image information.

  In one exemplary implementation according to an aspect of the present invention, the QCL beam is optically adjusted to illuminate a macroscopic spot (about 5-8 mm in diameter) on an infrared reflective or transmissive slide, on which infrared light is transmitted. The beam interacts with the sample. The reflected or transmitted infrared beam is projected to the infrared array detector via suitable imaging optics and samples the entire illuminated area with a pixel size that is generally less than or equal to the analysis limit.

  The infrared spectrum of a tissue or cell voxel represents a snapshot of the overall chemical or biochemical composition of a sample voxel. This infrared spectrum is spectral data used for generation (304) of a spectral image. The above description shows a method for obtaining spectral data and an outline example of the obtained spectral data. Further details of the process for obtaining spectral data are described in US Patent Application No. 13 / 084,287. Has been.

  In one embodiment of the present invention, after all spectrum data is acquired by a doctor or the like, the obtained spectrum data and biological sample are transmitted to the SHP system. For example, the SHP system has a receiving module operable to receive transmitted data. Data is automatically or manually input to an electronic device capable of transmitting data, such as a portable device such as a computer, a mobile phone, and a personal digital assistant (PDA). In one aspect of the invention, the SHP system may include a remotely located computer with a suitable algorithm for analyzing the data. In another aspect of the invention, the SHP system may include a computer located within the same local area network as the electronic device into which data is input, or is located on the electronic device into which data is input. (I.e., physicians etc. can enter data directly into the SHP system). If the SHP system is located separately from the electronic equipment, the data can be transmitted to the SHP system by suitable electronic transmission means to a local computer using a local area network or the Internet. An example network layout and system for transmitting data to the SHP system is described in more detail below in connection with FIGS.

  In another aspect of the present invention, the sample itself is delivered to the SHP system, rather than the physician acquiring data and transmitting it to the SHP system. For example, the SHP system has a receiving module operable to receive a sample. When the physical sample is sent to the SHP system, the spectrum data can be acquired without the doctor or the like operating the SHP system. In this case, rather than transmitting only the spectral data to the SHP system, for example, the biological sample may be physically delivered at a remote location. Nevertheless, doctors etc. can provide applicable clinical data.

  The method according to the invention further comprises a step 306 of preprocessing the spectral image. As detailed in related US patent application Ser. No. 13 / 067,777, preprocessing spectral data is also useful for obtaining the desired cellular material and for removing mixed spectra. .

  5A and 5B illustrate a pre-processing procedure 500 in one embodiment of the present invention. The method according to the present invention includes a step 502 of reading initial infrared image data. For example, a system according to the present invention reads infrared image data that has been received and / or previously stored in the system. In one aspect of the system according to the present invention, the received infrared image data is converted into an absorptance and a spectral parameter is calculated.

  The system according to the invention includes a step 504 for selecting binned data and a step 506 for storing the data set. One embodiment of the system according to the present invention can bin the image data to reduce the number of pixels. By reducing the number of pixels, for example, the signal-to-noise ratio can be improved. For example, the system according to the present invention reads an image file using 2 × 2 binning settings. The system according to the present invention stores a data set in a data storage device.

  The method according to the invention includes a step 508 of removing the offset and a step 510 of correcting the minimum value of the data in the selected range. For example, the system according to the present invention can remove baseline offset from spectral data by processing data from frequencies within a selected range.

  The method according to the invention includes the step 512 of generating a grayscale image by integrating over a selected range. In one aspect of the system according to the present invention, a data set of grayscale images is generated by integrating spectral intensities within a certain range. For example, a grayscale image makes it possible to observe pixels with significant infrared intensity before the filter is applied.

  The method according to the invention includes a step 516 of reading the water vapor correction. For example, the system according to the present invention can correct the water vapor effect in the spectral image data by reading the water vapor correction information and applying it to the spectrum.

  The method according to the invention includes a step 514 of applying water vapor correction and a step 518 of saving the data set. In one aspect of the system according to the invention, multiplicative signal correction (MSC) can be used to correct the vibrational rotation contribution of residual water.

  The method according to the invention includes a step 520 of providing a noise reference and a step 522 of saving the data set. In one aspect of the system according to the present invention, the image data set can be divided into noise and signal regions. For example, the system assigns a black area of a grayscale image to a noise area and assigns a grayish area to a signal area. In one aspect of the present invention, noise and signal domain separation is achieved by integrating the spectral structure between certain wavelengths. If the integrated value in this range exceeds the minimum value in the signal, it can be defined as a noise spectrum if the integrated intensity is within the above range.

  The method according to the present invention includes a step 524 of applying multivariate noise reduction. For example, the system according to the present invention applies principal component analysis (PCA) to the noise spectrum to reduce the noise contribution of the eigenvectors of the spectrum. While ordering, the signal spectrum of the data set can be reconstructed as a sum of eigenvectors.

  The method according to the invention includes a step 530 of determining the shape and power of the full range of signals. The method according to the present invention also includes a step 532 of excluding bad data (eg, data not included in the range between the minimum and maximum values) and a step 534 of saving the data set. In one aspect of the system according to the present invention, one or more quality tests are performed to determine the shape and power of the signal. Examples include, but are not limited to, peak frequency position, band shape, total signal intensity, band area ratio, and the like. In one aspect of the system, a quality test based on the peak frequency is performed. In another aspect of the system, a quality test based on the total spectral integrated intensity in a certain range may be performed. The integrated intensity needs to fall within a range between a certain minimum and maximum value. In one aspect of the system according to the present invention, a quality test may be performed based on the peak area ratio between the amide I and amide II bands by integration over a certain range. In one embodiment, the intensity ratio needs to fall within a range between a certain minimum and maximum value.

  The method according to the invention includes a step 536 of reporting the region signal to noise ratio and a step 538 of storing the signal to noise ratio. The system receives an input and calculates a signal to noise ratio. The input includes, for example, the left and right ends of the reference region of interest, the left and right ends of the signal region of interest, and the left and right ends of the noise region of interest.

  The method further includes enhancing 560 the signal for classification and saving 562 the data set. In one aspect of the system, for example, smoothing differentiation is performed to smooth the window width, order, and derivative. The system standardizes to enhance the classification signal.

  The method includes a step 564 of selecting a region of interest for phase correction and a step 566 of saving the data set. For example, the system can expand the spectral region of interest between certain values and select the spectral region of interest for phase correction.

  The method includes a step 568 of performing phase correction on the selected region of interest. In phase correction, a 512-data point second-order differential spectrum vector is subjected to finite Hilbert transform (censored FFT) and divided into a real part and an imaginary part. In one aspect of the system, coordinate transformation can be performed to generate a new spectral vector. The system selects a phase corrected trial spectrum having a maximum frequency reference peak in a range of values as a corrected spectrum. Phase correction is described in detail in US patent application Ser. No. 13 / 067,777.

  The method according to the invention includes a step 570 of selecting a region of interest for classification. For example, the system can extend the range of the region of interest.

  The method includes a step 574 of performing normalization. Normalization includes, but is not limited to, vector normalization, standard normal variables, and multiple regions. The system can apply normalization to the region of interest.

  The method includes a clustering step 576. For example, the system can perform clustering.

  The method includes a step 578 of generating a cluster image and a step 580 of saving the data set. The system can generate and store cluster images.

  The method includes generating 582 different metrics for associating with the cluster image. The above steps include association and distance calculation. The method also includes a step 589 of generating a validity report. For example, the system can generate a cluster metric and associate a cluster image with a known reference. Metric clustering includes, but is not limited to, k-means clustering and hierarchical cluster analysis (HCA). The system can also generate validation. FIG. 16 shows an example of a validity report.

  As shown in FIG. 3, the method includes a step 310 of receiving clinical information in an annotation process. In one aspect of the system, clinical information can be received from a data source such as a physician, patient electronic medical record, or a data repository storing clinical information. Clinical information includes, for example, information on diagnosis and / or prognosis such as the type of cells likely to be present in the sample, the body part from which the sample was collected, and the type of disease or condition likely to be present. Is included. The clinical information includes the “gold standard” of the current clinical technology. For example, the “gold standard” of clinical technology includes the use of immunohistochemical (IHC) staining and panels, staining of biological samples such as hematoxylin stain, eosin stain, and Papanicolaou stain . In addition, the “gold standard (optimum standard)” of clinical technology includes measuring and specifying characteristics of biological samples such as staining patterns using a microscope.

  The method according to the present invention also includes a step 308 of receiving annotation information of the infrared image. Annotation information includes appropriate clinical data regarding the selected annotation area, for example, biochemical properties related to cell and / or tissue type characteristics likely to be present in the sample, sample staining grade, intensity Include molecular data such as the state of molecular markers (eg, the state of molecular markers for IHC staining), the body part from which the sample was taken, and / or the type of disease or condition likely to be present It is. The annotation information may also relate to measurable marks on the visual image of the sample. Further, the annotation information includes, for example, a timestamp (eg, date and / or time when the annotation was created), parent file annotation identifier information (eg, whether the annotation is part of an annotation set, etc.), user Information related to the annotation such as information (for example, the name of the user who created the annotation), cluster information, cluster spectrum pixel information, cluster level information, the number of pixels in the selected region, and the like may be included. In this system, annotation information may be received from a user such as a doctor.

  In one aspect of the present invention, when a user selects an annotation region of a registered spectral image, the annotation information is provided. The user may use a system that selects regions associated with the biochemical characteristics of the disease and / or condition in the registered image. For example, the user can place a border around a region in the spectral image where the pixel spectrum is substantially uniform (eg, the colors in the region of the spectral image are substantially the same). The boundary allows identification of pixels associated with the biochemical characteristics of the disease or condition in the spectral image. In another aspect of the present invention, the user can select an annotation region based on features or characteristics of the visual image. That is, the annotation region can be associated with various visual features in addition to the biochemical state of the biological sample. The annotation region is described in more detail in US patent application Ser. No. 13 / 507,386. The user may also select annotation regions that are not related to the biochemical characteristics of the disease or condition in the registered spectral image.

  In other aspects of the system according to the present invention, as detailed in US patent application Ser. No. 13 / 645,970, automatically or otherwise (eg, with user assistance or input parameters). The annotation information of the selected annotation area can be provided. For example, the system can provide the date and time the annotation was created along with cluster information for the selected region. In addition, the system automatically or otherwise selects an annotation region of a registered spectral image and associated clinical data (eg, data related to diagnosis and / or prognosis or classification of a disease or condition). ) Can be provided.

  In one aspect of the system according to the present invention, some or all of the cluster levels of the spectral image are reviewed and the spectral clusters of the pixels are relatively uniform (eg, uniform pixels having a similar spectrum for each predetermined parameter). (Spectral Cluster)) The cluster level can be specified. In one aspect of the system, uniform spectral clusters can each be presented as a single color (eg, blue for one cluster and red for another cluster). The system compares the identified cluster level with the cluster level of the selected annotation region of the spectral image and if it determines that they are compatible, it should not select another level or cluster level. Is determined.

  The above is an overview of an example annotation process and further details are disclosed in US patent application Ser. No. 13 / 645,970.

  The method according to the invention includes a step 312 of generating a real image. The real image is, for example, a visual image including an annotation region of a biological sample. A visual image of the sample can be obtained with a conventional microscope of the kind commonly used in pathology laboratories. The microscope may be provided with a high-resolution digital camera that digitally captures the field of view. This real-time digital image is based on the normal microscopic field of the sample and shows the structure of the tissue, cell morphology, and staining pattern. The image may optionally be stained using hematoxylin-iocin (H & E) and / or other components, immunohistochemicals, in situ hybridization (ISH), and the like.

  6A, 8A and 11 show examples of real images. FIGS. 6A and 11 show images in which a cancer area of adenocarcinoma (ADC) is annotated in a biological sample. For example, the dark blue region of the image indicates an annotation region where the user has specified the ADC in the biological sample. FIG. 8A shows an actual image of the entire biological sample in which the ADC region is specified (blue region of the image).

  The method according to the invention also includes a step 314 of generating a classification model and training a classification algorithm. The system can train algorithms to provide diagnosis, prognosis and / or predictive classification of a disease or condition as detailed in US patent application Ser. No. 13 / 645,970. The system can also form one or more classification models for diagnosing disease, as detailed in US patent application Ser. No. 13 / 645,970. In one aspect of the invention, the data repository can include a set of listed tissue or cell classes. The classification can be created, for example, based on expert opinions, organizational decisions, and individual or organizational criteria. That is, algorithms used to provide diagnostic, prognostic and / or predictive analysis of biological samples can be trained for use by professionals, various organizational standards, and individuals.

  For example, the system can receive a query that includes one or more parameters for training and testing a function associated with a biological characteristic indicative of a particular disease, condition, functional state, and / or class. . Such parameters include the type of disease or condition (eg, lung or kidney cancer), cell or tissue class, tissue type, disease state, classification level, spectral type, and tissue location. In one aspect of the system, queries and parameters can be received from a user. In other embodiments, the system can determine parameters to use for a particular disease or condition, either automatically or in other ways. That is, the training and inspection functions can be customized based on the received parameters.

  The system can extract the pixels of the visual and spectral images stored in the data repository that correspond to the parameters of the training test function. For example, the system can access an annotation image stored in a data repository along with appropriate annotation information and / or metadata corresponding to the annotation image. The system can compare the query parameters with annotation information and / or metadata of the annotation image. If the parameters and annotation information and / or metadata match, for example, the visual and spectral image pixels associated with the parameters can be extracted to form a training data set. Pixels extracted for the training data set include pixels from different cell or tissue classifications and / or tissue types. Note that pixels extracted from different tissue types may be stored as part of different inspection functions. That is, for example, pixels from different tissue types may be assigned to different inspection functions, and pixels from the same tissue type may be assigned to a single inspection function. Further, the training data set may include spectral data associated with a particular disease, condition, cell or tissue type (hereinafter referred to as “classification”). Thus, the system can generate visual and spectral image pixels that effectively display a disease or condition based on parameters provided to the training function to perform diagnosis, prognosis and / or predictive analysis of the disease or condition. Can be extracted.

  The verification test includes a quality test of the training data set and a function selection test. In one aspect of the system, an algorithm generated from a training data set and a test data set to verify the accuracy of the algorithm can be used. The test data set includes biological samples that do not have a specific disease or condition and biological samples that have a specific disease or condition.

  The system determines the accuracy of the algorithm, for example, by determining whether the algorithm can accurately distinguish between a biological sample having a particular disease or condition and a biological sample that does not have a particular disease or condition. Can be verified. If the algorithm can accurately distinguish between a biological sample having a disease or condition and a biological sample having no disease or condition, the system determines that the accuracy of the algorithm is high. Conversely, if the algorithm cannot accurately distinguish between a biological sample having a disease or condition and a biological sample having no disease or condition, the system determines that the accuracy of the algorithm is low. In one aspect of the invention, the result of the algorithm may be compared to an index value that represents the probability that the algorithm correctly identifies the biological sample. When the index value exceeds the threshold level, the algorithm has a high probability of correctly identifying the biological sample. On the other hand, when the index value is below the threshold level, the algorithm has a low probability of accurately identifying the biological sample.

  For example, if it is determined that the accuracy of the algorithm is low, the system can improve the training data set. The system can increase and / or decrease the number of pixels to increase the feasibility of the statistically relevant reasonable performance of the algorithm. Note that the number of pixels required for the training data set may vary based on the type of disease or condition diagnosed by the algorithm and / or the classification of the selected cells or tissues.

  If it is determined that the algorithm has high accuracy, the system can generate an improved algorithm to provide diagnosis, prognosis and / or predictive analysis of a particular disease based on the testing function. It should be noted that a plurality of algorithms for providing disease diagnosis, prognosis and / or predictive analysis may be generated based on the received parameters. For example, training multiple algorithms, each trained to diagnose a specific type of lung cancer, based on various parameters associated with the biochemical characteristics that represent the disease or functional state and disease classification May be.

  For example, the system can save one or more trained algorithms in a data repository in which annotation spectral and visual images, annotation information and / or metadata are stored.

  The system can also form one or more classification models for diagnosing disease, as detailed in US patent application Ser. No. 13 / 645,970. For example, various forms of cancer (eg, lung cancer, breast cancer, kidney cancer, etc.) diagnostic algorithms can be combined to form a model for diagnosing cancer. Note that the classification model may include a sub model. That is, the classification model for diagnosing cancer may have submodels for diagnosing various forms of cancer (for example, lung cancer, breast cancer, kidney cancer). Moreover, the submodel may have a further submodel. As one example, a model for diagnosing lung cancer has multiple submodels for identifying types of lung cancer that may be present in a biological sample.

  In one aspect of the system, by identifying major cancer types and benign / malignant, for example, “benign”, “small cell lung cancer (SCLC)”, “adenocarcinoma (ADC)”, “squamous cell carcinoma (SQCC)” Lung cancer can be subtyped as “Large Cell Lung Cancer (LCLC)”. The system also identifies the subtypes of the identified main cancer types and further identifies the subtypes of subtypes. Subtypes include, but are not limited to, scales (Lepidic), acinar type (Acinar), papillary type (Papillary), micropapillary type (Micropapillary), and solid type (Solid). In one aspect of the system, one or more classification models for diagnosing disease using the identified subtypes and types can be generated. For example, the system can classify subtypes and types as cancer classes in a classification model. The cancer class is used for diagnosis of biological samples. The cancer class is associated with the medical data group. The medical data group includes, for example, an appropriate treatment method according to the disease state. For example, a cancer class is associated with a group of clinical data about a disease state and its treatment. The system can use the classification model to suggest an appropriate treatment (as a companion diagnostic or in conjunction with literature data mining) for a disease identified in a class or subclass.

  The system can also distinguish between disease types and subtypes in normal tissues (tissues presumed not to contain related diseases). The system can identify heterogeneity of biological samples using, for example, the above classes. In one aspect of the system, as shown in FIG. 14A, normal tissue proximal to the cancer lesion and normal tissue distal to the cancer lesion can be distinguished. FIG. 14 shows cancer tissue (CA) and proximal normal tissue (PN) located in the vicinity thereof. FIG. 14A also shows distal normal tissue (DN) located remotely from the cancer tissue and benign normal tissue (BN) located outside the cancer lesion. In one aspect of the system, proximal normal tissue, distal normal tissue and benign normal tissue are analyzed. Normal tissue inside the tumor has characteristics that are different from benign lesions. Proximal normal tissue also has different properties than distal normal tissue. For example, the properties of proximal normal tissue suggest that cancer is present in the proximal normal tissue, whereas the properties of distal normal tissue exhibit different pathologies. In one aspect of the system, proximal normal tissue located in the vicinity of cancer tissue is used to measure the strength of disease relevance, disease progression, and disease pattern. For example, the system can analyze appropriate cell and tissue morphology descriptors such as stroma, connective tissue and vessel walls.

  After identifying various types and subtypes of cancer, the system can further identify types and subtype variants. Variants include cancer types and histological subtypes such as mucinous adenocarcinoma, colloidal, fetal (low and high atypical) (fetal) and enteric (enteric) The resulting modifiers are included. In one aspect of the system, variants can be classified as classes in a classification model.

  FIG. 14B illustrates an example of benign and malignant tumor classification according to one embodiment of the present invention. FIG. 14B shows an example of sub-classification that is clustered into lung hamartoma (benign lesion), sarcoidosis (granulomatous), organizing pneumonia type (blue) and lung cancer tumor normal type (red). 1402 shows an example of a benign sub-classification classified by SHP. 1404 shows an example of necrosis, keratin pearls, and lepidic sub-classifications. 1406 shows an example of a grade of squamous cell carcinoma classified by SHP. 1408 shows an example of a subclass of adenocarcinoma.

  The system can establish a rule set for determining the application order of algorithms in the classification model. Furthermore, the present system can establish a rule set for limiting the use time of the algorithm. Note that the rule set varies depending on, for example, the disease and / or the number of algorithms combined to form a model. The system can generate one or more classification models for diagnosing a particular disease after establishing a rule set for the classification model. In addition to the above methods, various other methods can be used to create a classification model for a specific disease or condition.

  One example of a rule set for applying the above algorithm in a classification model is a variation reduction order determined using hierarchical cluster analysis (HCA) or other clustering / segmentation methods. One example of an HCA is described in detail in US patent application Ser. No. 13 / 067,777 (hereinafter referred to as the “'777 application”). As described in the '777 application, HCA identifies tissue or cell classes that are classified by various similarities. Based on the HCA, the most effective iteration order or variation reduction order can be determined. That is, the iteration hierarchy / variation reduction order can be established based on all data variations provided by the HCA. Using HCA, based on the similarity or variance of the data, it is possible to determine the tissues or cell classes that will be labeled and not included in subsequent data subsets in order to remove variance and improve identification accuracy .

  FIG. 14C is an example of a rule set for determining the classification of lung cancer in one embodiment of the present invention, and A, B, C, and D in the figure indicate a certain tissue state, class, or subtype. A user, such as a physician, can apply the method described above to determine whether the sample contains the listed tissue or cell class during work. That is, the iterative process can be repeated until the desired result is reached. For example, a physician or the like can examine a sample for cancer cells or certain types of cancer. Based on clinical data (for example, the most likely pathological condition), the pathological condition to be examined may be determined, or various pathological conditions may be comprehensively examined. The methods disclosed herein can improve the accuracy of diagnosis, especially when there is little or no information about a disease state that is likely to exist. Further, the methods disclosed herein can be used for prognosis and / or predictive classification of a disease or condition.

  The method further includes a step 316 of generating a predicted image. The system can apply one or more classification models and / or one or more classification algorithms trained with the classification models to the real image to generate a predicted image. The system can also apply one or more classification models and / or classification algorithms to the biological sample. 6B and 12 show examples of predicted images. That is, FIG. 6B shows that the presence of squamous cell carcinoma (SQCC) is predicted in the reddish purple region and the presence of adenocarcinoma (ADC) is predicted in the blue region. Therefore, it is predicted that both squamous cell carcinoma (SQCC) and adenocarcinoma (ADC) are present in the biological sample of FIG. 6B.

  FIG. 12 shows another example of the predicted image. FIG. 12 is an image of the entire sample containing five classes of predicted tissue. The above image shows squamous cell carcinoma (SQCC) in the blue region, adenocarcinoma (ADC) in the reddish purple region, necrosis in the green region, small cell lung cancer (SCLC) in the yellow region, and normal tissue in red. Each is shown in the area.

  The method includes a step 326 of generating a confidence prediction image. The confidence prediction image includes a confidence value indicating a degree of confidence that a specific cancer class or subclass exists in the prediction image. For example, a high confidence value indicates that there is a high probability that one or more diseases are present in the predicted image. Moreover, when the confidence value is high, it indicates that the specific disease is more advanced. For example, in this system, when a spectrum from a predicted image is analyzed and the spectrum signal is close to the center of the cancer class, the reliability is high. Also, the reliability is high when the spectrum signal from the predicted image is pure (no other spectrum is mixed).

  In one embodiment of the present invention, a low confidence value indicates that there is a high probability that one or more diseases are present in the predicted image. For example, the system analyzes the spectral signal of the predicted image to determine how close it is to the center of the cancer class. For example, if a spectral signal is included in a class of cancer but is far from the center of the class (located at or near the class spectrum boundary), it overlaps with other cancer classes. Therefore, the reliability with respect to the presence of a specific cancer class in a biological sample is low. A signal far from the center of the class also indicates that the sample contains a new cancer class, a different type of cancer, or a different subtype of cancer. A low confidence value indicates that the disease is not progressing and / or is a different type of disease.

  In one embodiment of the present invention, the reliability value is represented by a numerical value in the range of 1 to 10, for example, where 1 indicates low reliability and 10 indicates high reliability. In another example, the reliability value is represented by a numerical value between 0 and 1, with 0 indicating low reliability and 1 indicating high reliability. In one aspect of the system, the confidence value is calculated using one or more prediction calculations. Prediction calculations include, but are not limited to, for example, plat separation plane, random forest, Bayesian a priori estimates, artificial neural networks, and LDA. Various prediction calculations can be used for the calculation of the confidence value.

  In one aspect of the present system, the confidence value of each class or subclass indicated in the prediction image can be overwritten, and a confidence prediction image indicating the overwritten confidence value can be generated. For example, the confidence value can be expressed in a binary manner, such that a white point is added when the confidence value is low and nothing is added when the confidence value is high.

  In another example, a color scale can be used to represent the reliability. For example, it can represent that the confidence value is high in light color, and that the confidence value is low in lighter shade or white. FIG. 7A to FIG. 7C and FIG. 10 show examples of reliable prediction images. As shown in FIG. 7A, FIG. 7B, and FIG. 10, white spots in the image represent a low confidence in the presence of a particular cancer class or subclass in the biological sample. That is, the spectrum indicates that there is some abnormality in the above region, but the reliability for specifying the abnormality is low. In FIG. 7B, high density white pixels are densely packed in a part of the region. The above regions indicate that there are new cancer classes or subclasses. The colored areas of FIGS. 7A, 7B, 7D and FIG. 10 indicate a high degree of confidence for the presence of a particular class or subclass in the biological sample.

  FIG. 7C shows an example of a trust image in which clinical images are superimposed. For example, the system allows the user to observe the clinical image and the trust image simultaneously by superimposing the trust image on the clinical image. The system can transmit, for example, a trusted image for display in a field of view of a microscope (such as a pathologist's microscope), a display of a computing device, and / or a document or report. In one aspect of the system, the virtual image of the trust prediction image is projected onto the field of view of the microscope, so that the user can observe the trust image together with the biological sample. In this system, the virtual image of the trust prediction image is superimposed on the image of the biological sample of the microscope, so that the user can view the biological sample from the trust prediction image projected on the front surface of the microscope field of view while viewing the image of the biological sample. A biochemical analysis can be received. The confidence prediction image can highlight a region of the biological sample that the user desires to analyze in more detail. For example, the user can enlarge and observe different regions of the biological sample based on the region of interest specified in the reliability prediction image. Moreover, in this system, the user can display / disappear the reliability prediction image. In this system, by superimposing the predicted image on the clinical image, when the user observes the biological sample, the biochemical analysis for specifying the predicted classification of the biological sample can be made more efficient.

  FIG. 13 shows an example of a prediction legend having a confidence scale used when observing a confidence prediction image in one embodiment of the present invention. The prediction legend represents various cancer classes and degrees of confidence by color. For example, when the reliability is low, a light color or white is used, and when the reliability is high, a dark color is used.

  The predictive case includes various cancer classes displayed in the reliable prediction image. In one aspect of the invention, predictive precedents include, for example, adenocarcinoma (ADC), squamous cell carcinoma (SQCC), necrosis, small cell lung cancer (SCLC), and normal tissue. In addition, a color value can be assigned to each cancer class displayed in the prediction case. For example, squamous cell carcinoma (SQCC) is displayed in blue, adenocarcinoma (ADC) in magenta, necrosis in green, small cell lung cancer (SCLC) in yellow, and normal tissue in red. Predictive cases include several cancer classes and / or different types of diseases. In addition, cancer classes are differentiated by various methods, and a method using color is one example.

  Moreover, the prediction case includes a confidence scale indicating the reliability of the prediction. For example, the confidence scale is displayed in a range of 0 to 1, with 0 indicating low reliability and 1 indicating high reliability. The predictive case can change the color of the cancer class based on the reliability. For example, white or light color indicates low reliability, and dark or dark color indicates high reliability. Thus, the bright blue color represents a low confidence that the biological sample contains adenocarcinoma (ADC). On the other hand, the dark green color represents a high confidence that the biological sample contains necrosis cancer.

  As shown in FIG. 3, the method includes a step 328 of generating a prediction report that includes a confidence value. The prediction report can identify the class and subclass of cancer found in the biological sample and provide a confidence value that indicates the degree of confidence that a particular cancer class or subclass is present in the prediction image. As shown in FIGS. 6A to 6C, FIGS. 7A to 7D, FIGS. 10, 11, and 12, the prediction report includes, for example, a real image, a predicted image, and a reliable image. For example, FIG. 12 shows an example of a predictive image showing five cancer classes found in a biological sample. FIG. 10 shows an example of a confidence image showing the degree of confidence of five cancer classes found in a biological sample.

  As shown in FIGS. 15 and 16, the prediction report includes, for example, a chart and / or graph that displays a disease found in a biological sample, and a degree of confidence. For example, FIG. 15 shows an example of a predictive report that describes the type of tissue found in a biological sample, the class of dominant disease identified, the range of tissues in which the disease was identified, and the degree of confidence in the analysis. FIG. 15 shows an example of a bar graph representing the prediction result. FIG. 16 illustrates an example of a validation report in one embodiment of the present invention.

  Therefore, the user (physician or the like) can easily confirm the cancer class specified in the biological sample and the degree of confidence with respect to the cancer class by the prediction report. Further, by using the trust image and the confidence value report, it is possible to visually indicate the degree of trust regarding the state of the comorbidity, the difference in the disease type of the heterogeneous disease, and the state of the complication. Thus, a user (such as a doctor) can use a predictive report to identify significant diseases found in a biological sample along with other diseases that may be present in the biological sample.

  The method further includes a step 318 of performing a difference analysis between the actual image and the predicted image. In one aspect of the system, the actual image of the biological sample and the predicted image can be compared and the difference between the two can be identified. In the difference analysis, texture comparison between the actual image and the predicted image, comparison between the actual image and the predicted image, comparison of spectral fluctuations (magnitude of spectral change, fluctuation width), and spatial locality difference (for example, different) Areas may be clustered together to form larger areas of the same color or spread to other colors), IHC markers (+ or −) molecular markers (+ or −), histopathology, and other suitable Includes but is not limited to metadata or clinical data (patient information). In one aspect of the present system, by applying one or more of the above-described difference analysis to the predicted image, it is possible to specify a region different from the actual image in the predicted image without explaining the difference. The more difference analysis is applied to the predicted image, the higher the possibility that the identified difference is a new cancer class.

  For example, the present system can compare the actual image shown in FIG. 6A with the predicted image shown in FIG. 6B to determine whether a difference exists. For example, the real image of FIG. 6A shows that the biological sample includes squamous cell carcinoma (portion shown in blue). On the other hand, the predicted image indicates that the biological sample includes squamous cell carcinoma (portion shown in blue) and adenocarcinoma (portion shown in reddish purple). This system determines that the reddish purple region of the predicted image in FIG. 6B is different from the same region in the actual image.

  The system includes a step 320 of assigning region of interest pixels to a new class. In one aspect of the system, based on the differential analysis, an annotation region for the region of interest pixel is generated and assigned a new class. For example, the present system determines that the reddish purple region of the predicted image in FIG. 6B is different from the same region in the real image of FIG. 6A, generates an annotation region for the reddish purple region of the predicted image, Assign a new class. In the method, the physician annotates the image by annotating the image indicating whether the biological sample contains a new class (step 308).

  The method includes determining 322 a true positive region of interest or a true negative region of interest. For example, the system identifies pixels of a comparative image that include a true positive region of interest or a true negative region of interest. In the true positive region, for example, the real image indicates that the cancer class exists in it (the doctor has annotated the real class with the cancer class), and the spectrum of the predicted image is the above in the predicted image. A region of a comparison image indicating the presence of a cancer class is included. In the true negative region, for example, the real image of the biological sample indicates that there is no cancer class in it (the doctor has annotated the real image that there is no cancer class), and A region of a comparative image whose spectrum indicates that the cancer class does not exist in the predicted image is included.

  FIG. 8B shows an example of a predicted image including a true negative region. For example, FIG. 8B shows a real image of a biological sample containing squamous cell carcinoma (SQCC +) identified in the blue region. Moreover, FIG. 8B shows the prediction image of the same biological sample as FIG. 8A containing the true positive area | region which identifies squamous cell carcinoma (SQCC +) in the same area | region where the prediction image was specified in the real image. For example, the blue region of the predicted image corresponds to the blue region of the actual image.

  The method further includes determining 324 a false positive region of interest and a false negative region of interest. In one aspect of the system, for example, the system identifies pixels of a comparative image that include a false positive region of interest or a false negative region of interest. The false positive region includes, for example, a region of a comparative image indicating that the actual image indicates that the cancer class is not present therein and the spectrum of the predicted image indicates that the cancer class is present in the predicted image. . In the false negative region, for example, a real image of a biological sample indicates that a cancer class is present therein, and a region of a comparative image in which the spectrum of the predicted image indicates that the cancer class is not present in the predicted image Is included.

  FIG. 8C shows an example of a predicted image including a false negative region. For example, FIG. 8C shows a predicted image of the same biological sample as FIG. 8A. While only the squamous cell carcinoma (SQCC +) is shown in the real image of FIG. 8A, the corresponding false-positive region in FIG. 8C includes a green region indicating the presence of necrosis. .

  The method further includes selecting 330 a region of interest in the confidence prediction image based on the confidence value. Regions of interest include regions that are well differentiated but have low confidence in the type of class or subclass identified in the biological sample. FIG. 7B shows an example of a region of interest that is well differentiated but has low confidence. For example, the region of interest shown in FIG. 7B includes a plurality of white spots having spatial locality. 9A and 9B show examples of images that include regions of interest that are hardly differentiated. The region of interest includes a region that is hardly differentiated but has high reliability of the spectrum signal. For example, the region of interest includes a plurality of color pixels that are located in the poorly differentiated region and that show strong spectral signals from real images that show different classes. In one aspect of the system, the region of interest is identified by receiving the identified false negative region of interest and false positive region of interest and inserting a boundary in the form of a circle, grid, contour, etc. around the region of interest. .

  The method includes a step 332 of assigning region of interest pixels to a new class. The system generates an annotation region for the region of interest pixel and assigns it a new class. In the method, the physician annotates the image by annotating the image indicating whether the biological sample contains a new class (step 308).

  As described above, the combined use of the confidence image can assist diagnosis, prognosis and / or prediction classification of a biological sample. Further, by using the trust image, it is possible to determine a region of interest for microdissection of a biological sample. For example, the region of interest identified in the confidence image can be used to identify a change in gene expression of the biological sample.

  FIG. 4 illustrates a procedure 400 for identifying a region for microdissection according to an aspect of the present invention. Microdissection enables, for example, separation inspection and / or microdetection analysis of biological samples. The micro detection analysis is used, for example, when performing gene expression analysis, gene sequence analysis, molecular analysis (next generation sequencing (NGS), etc.), and targeted therapy.

  The method includes applying 402 a heuristic to the image set. Heuristics include logical rules for identifying data used for microdissection in biological samples. For example, heuristics are areas of biological samples with low confidence, areas with multiple attributes, areas classified by specific tissue types, areas classified into specific cancer classes or subclasses, and clinical data about biological samples Etc. can be specified. In one aspect of the system, one or more heuristics can be applied to the image set to identify data of interest regarding microdissection in a biological sample.

  The method includes steps 404 and 406 for receiving a selection of a sample area to be ablated in the region of interest. The region of interest is identified using the result of applying a heuristic to the image set. In one aspect of the system, a selection of an ablation region by a physician is received. That is, the doctor can highlight or specify the sample area using the interface on the system. For example, a doctor can specify a region for microdissection by providing a boundary around the sample area. In another example, a region for microdissection can be highlighted.

  In another aspect of the system, the sample area to be excised can be automatically selected. For example, the system can receive data from a heuristic and use it to automatically select the sample area to be excised. Thereby, this system can highlight or specify the sample area to be excised.

  18A to 18D illustrate examples of sample areas according to one embodiment of the present invention. The sample area can be highlighted using a circle as shown in FIGS. 18A and 18B. Also, the sample area can be highlighted using a grid, as shown in FIGS. 18C and 18D. In one embodiment of the present invention, as shown in FIG. 18C, the sample area can be selected based on the reliability. For example, as shown in FIG. 18C, a region with low reliability (for example, a region where white spots are densely packed) can be selected as the sample area.

  The method includes a step 408 of aligning between the region of interest and the SHP image. The registration of the region of interest and the SHP image includes, for example, associating the spatial locality of the region of interest with the test identification of the sample area. In one aspect of the system, the spatial locality from the SHP image is associated with the region of interest and the association is stored in a data repository. By performing registration between the region of interest and the SHP image, the system can associate the gene sequence analysis performed on the region of interest with the spectral data of the SHP image. The system can also identify any regularity or change in gene sequence pattern based on gene sequence analysis. In addition, the system can track the examination applied to the sample.

  FIG. 17 illustrates an example of alignment between a clinical image and an SHP image according to an aspect of the present invention. For example, the system can superimpose the SHP image and the clinical image and associate the spectrum of the SHP image with the clinical image.

  The method includes a step 410 of taking material from a selected sample area and performing a molecular test. In one aspect of the system, the selected sample area can be used to control an automated tool for excising the sample area from a biological sample. For example, an automated tool can control a laser or other milling device to excise a sample area from a biological sample.

  It is within the scope of the present description that aspects of the invention can be applied to any particular cell or tissue classification, whether cancerous or not. When the above iteration process is applied, the first iteration analyzes the original sample data for the most extensive cell or tissue classification, and the subsequent iterations have a narrower range of cells or tissues obtained in previous iterations. Analyzing the sample data subset for classification allows the most accurate results to be obtained. It is within the scope of this document that the results of all iterations provided or output indicate data associated with a particular condition. For example, if the first iteration is an analysis of cancer, it can provide or output information about data found to be non-cancerous, as well as performing a second iteration of cancerous data .

  FIG. 19 illustrates various configurations of a computer system 1900 for use with a method according to aspects of the present invention. As shown in FIG. 19, a computer system 1900 includes a personal computer (PC), a minicomputer, a mainframe computer, a microcomputer, a telephone device, a personal digital assistant (PDA), or other device having a processor and input functions. Used by a user 1901 such as a requester / doctor or a representative of a user 1901 such as a requester / doctor. The server model includes, for example, a PC, minicomputer, mainframe computer, microcomputer, or other device that includes or has access to a processor and data repository. Server model 1906 may be associated with an available repository of disease-based data, such as training datasets and / or algorithms for use in diagnostic, prognostic and / or predictive analysis, for example.

  The above-described data can be transmitted between a doctor or the like and the SHP system (or user) via a network 1910 such as the Internet, and also transmitted between the analyst 1901 and the server model 1906. Can do. Communication is performed via connections 1911, 1913 such as, for example, wired, wireless, or fiber optic connections.

  Aspects of the invention can be implemented using hardware, software, or a combination thereof, and can be implemented by one or more computer systems or other processing systems. In one variation, aspects of the present invention implement one or more computer systems capable of performing the functions described herein. FIG. 20 shows an example of such a computer system 2000.

  Computer system 2000 includes one or more processors, such as processor 2004. The processor 2004 is connected to a communication infrastructure 2006 (eg, a communication bus, crossover bar, or network). Various software aspects are described in connection with this exemplary computer system. The description herein will make it apparent to those skilled in the art how to implement aspects of the invention using other computer systems and / or architectures.

  Computer system 2000 can include a display interface 2002 that transfers graphics, text, and other data from communication infrastructure 2006 (or a frame buffer not shown) for display on display unit 2030. The computer system 2000 can include a main memory 2008, preferably random access memory (RAM), and can also include a secondary memory 2010. The secondary memory 2010 can include, for example, a hard disk drive 2012 and / or a removable storage drive 2014 such as a floppy disk drive, magnetic tape drive, optical disk drive, and the like. The removable storage drive 2014 can read from and / or write to the removable storage unit 2018 by a well-known method. The removable storage unit 2018 represents a floppy (registered trademark) disk, a magnetic tape, an optical disk, and the like that can be read from and written to the removable storage drive 2014. In other words, the removable storage unit 2018 may include a computer that can use a storage medium in which computer software and / or data is stored.

  In alternative aspects of the present invention, secondary memory 2010 may include other similar devices to allow computer programs or other instructions to be loaded into computer system 2000. Such a device can include, for example, a removable storage unit 2022 and an interface 2020. Examples include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as erasable programmable read only memory (EPROM), or programmable read only memory (PROM)) and related. There are sockets, as well as other removable storage units 2022 and interfaces 2020 that allow software and data to be transferred from the removable storage unit 2022 to the computer system 2000.

  Computer system 2000 can also include a communication interface 2024. Communication interface 2024 allows software and data to be transferred between computer system 2000 and external devices. Examples of the communication interface 2024 include a modem, a network interface (such as an Ethernet (registered trademark) card), a communication port, a personal computer memory card international association (PCMCIA) slot, a card, and the like. Software and data transferred via the communication interface 2024 can be in the form of a signal 2028 that is an electronic signal, electromagnetic signal, optical signal, or other signal receivable by the communication interface 2024. can do. These signals 2028 are communicated to communication interface 2024 via communication path (eg, channel) 2026. Path 2026 carries signal 2028 and may be configured with wires or cables, fiber optics, telephone lines, cellular links, radio frequency (RF) links and / or other communication channels. As used herein, the terms “computer program medium” and “computer usable medium” generally refer to media such as removable storage drive 2014, hard disk installed in hard disk drive 2012, and signal 2028. Used about. These computer program products can provide software to computer system 2000. An aspect of the present invention implements such a computer program product.

  Computer programs (also called computer control logic) can be stored in main memory 2008 and / or secondary memory 2010. The computer program can also be received via the communication interface 2024. By executing these computer programs, the computer system 2000 can realize the configuration according to the aspect of the present invention described in this specification. In particular, the execution of the computer program enables the processor 2004 to realize those configurations. Therefore, these computer programs correspond to the controller of the computer system 2000.

  In one variation of aspects of the invention implemented using software, the software may be stored in a computer program product and read into the computer system 2000 by the removable storage drive 2014, the hard drive 2012, or the communication interface 2024. Good. When the control logic (software) is executed by the processor 2004, the processor 2004 may implement the functions described herein. In other variations, aspects of the invention are implemented primarily in hardware using hardware components such as application specific integrated circuits (ASICs). Implementation of a hardware state machine to perform the functions described herein will be apparent to those skilled in the art.

  In still other variations, aspects of the invention can be implemented by a combination of both hardware and software.

Claims (35)

Receiving an image of a biological sample in the system;
Applying annotation information associated with a particular disease or condition to the image and spectral data to generate a real image of the biological sample;
Applying an algorithm to the image and spectral data to generate a predictive image including a predictive classification of a disease or condition identified from the biological sample;
A method for classifying a biological sample, comprising: transmitting a predicted image including the predicted classification for display on a display. The predictive classification classifies the biological sample into classes,
The method according to claim 1, wherein the class is used for diagnosis determination, prognosis determination, treatment determination, prediction determination, and companion diagnosis.   The method of claim 2, wherein the class includes one or more sub-classifications.   The method of claim 2, wherein the class identifies a heterogeneity of the biological sample. Calculating the confidence of the predicted classification of the disease or condition identified from the biological sample;
Applying the reliability to the predicted image to generate a reliable predicted image;
The method of claim 1, further comprising: transmitting the confidence prediction image for display on a display. Further comprising determining a differentiation value of the real image,
The method of claim 1, wherein the differentiation value is a quantitative measure of the grade or degree of progression of a disease or condition.   The method according to claim 6, wherein the disease is not progressing when the differentiation value is low, and the disease is progressing when the differentiation value is high. Performing a differential analysis between the real image and the predicted image;
The method according to claim 1, further comprising: identifying a region of interest in the predicted image based on the difference analysis.   9. The method of claim 8, wherein the region of interest is selected from a group consisting of a region with low differentiation and high reliability and a region with high differentiation and low reliability.   9. The method of claim 8, wherein the region of interest is selected from the group consisting of a true positive region of interest, a true negative region of interest, a false positive region of interest, and a false negative region of interest.   9. The method of claim 8, further comprising assigning the region of interest to a new cancer class.   The method of claim 11, further comprising applying annotation information related to a specific disease or condition to the region of interest.   The difference analysis includes comparison of textures in the actual image and the predicted image, comparison between the actual image and the predicted image, comparison of spectral fluctuations, identification of spatial locality differences, comparison of immunohistochemistry (IHC) markers, molecular markers 9. The method of claim 8, wherein the method is selected from the group consisting of comparisons, comparisons based on histopathology, and metadata comparisons. Applying a heuristic to an image set selected from the group consisting of the real image, the predicted image, and the confidence predicted image;
Identifying a sample area region of interest of the biological sample based on the heuristic;
Assisting in collecting a portion of the biological sample corresponding to a sample area of interest;
The method according to claim 5, further comprising performing a molecular test on a part of the biological sample.   The heuristics identify areas of biological samples with low confidence, areas of biological specimens with multiple attributes, identification of areas of biological specimens classified by specific tissue types, specific cancer classes or subclasses 15. The method of claim 14, wherein the method is selected from the group consisting of identifying an area of a biological sample to be classified and processing clinical data about the biological sample.   15. The method of claim 14, further comprising associating the sample area region of interest with the spatial locality of the predicted image. A biological sample classification system,
Having a memory in communication with the processing unit;
The memory and processing device cooperate to receive an image of a biological sample and apply annotation information associated with a particular disease or condition to the image and spectral data to generate an actual image of the biological sample; Applying an algorithm to the image and spectral data to generate a predicted image including a predicted classification of the identified disease or condition from the biological sample, and transmitting the predicted image including the predicted classification for display on a display A biological sample classification system comprising: The predictive classification classifies the biological sample into classes,
The system according to claim 17, wherein the class is used for diagnosis determination, prognosis determination, treatment determination, prediction determination, and companion diagnosis.   The system of claim 18, wherein the class includes one or more sub-classifications.   The system of claim 18, wherein the class identifies a heterogeneity of the biological sample.   The processing apparatus further calculates a reliability of a predicted classification of a disease or a pathological condition identified from the biological sample, generates a reliability predicted image by applying the reliability to the predicted image, and displays the reliability predicted image The system of claim 17, wherein the system is configured to transmit for display above. The processing device is further configured to determine a differentiation value of the real image;
18. The system of claim 17, wherein the differentiation value is a quantitative measure of the grade or degree of progression of a disease or condition.   The system according to claim 22, wherein the disease is not progressing when the differentiation value is low, and the disease is progressing when the differentiation value is high.   The said processing apparatus is further comprised so that a region of interest may be specified in the said prediction image based on the said difference analysis by performing the difference analysis of the said real image and the said prediction image. System.   The system according to claim 24, wherein the region of interest is selected from a group consisting of a region with low differentiation and high reliability and a region with high differentiation and low reliability.   25. The system of claim 24, wherein the region of interest is selected from the group consisting of a true positive region of interest, a true negative region of interest, a false positive region of interest, and a false negative region of interest.   25. The system of claim 24, wherein the processing device is further configured to assign the region of interest to a new cancer class.   28. The system of claim 27, wherein the processing device is further configured to apply annotation information associated with a particular disease or condition to the region of interest.   The difference analysis includes comparison of textures in the actual image and the predicted image, comparison between the actual image and the predicted image, comparison of spectral fluctuations, identification of spatial locality differences, comparison of immunohistochemistry (IHC) markers, molecular markers 25. The system of claim 24, wherein the system is selected from the group consisting of comparisons, comparisons based on histopathology, and metadata comparisons.   The processing apparatus further applies a heuristic to an image set selected from the group consisting of the real image, the prediction image, and the confidence prediction image, specifies a sample area region of interest of the biological sample based on the heuristic, and the biological sample 23. The system of claim 21, wherein the system is configured to assist in collecting a portion corresponding to a region of interest of the sample area and to perform a molecular test on a portion of the biological sample.   The heuristics identify areas of biological samples with low confidence, areas of biological specimens with multiple attributes, identification of areas of biological specimens classified by specific tissue types, specific cancer classes or subclasses 31. The system of claim 30, wherein the system is selected from the group consisting of identifying an area of a biological sample to be classified and processing clinical data about the biological sample.   32. The system of claim 30, wherein the processing device is further configured to associate the sample area region of interest with a spatial locality of the predicted image. A receiving module for receiving an image of the biological sample;
Applying annotation information related to a specific disease or condition to the image and spectral data to generate a real image of the biological sample, and applying an algorithm to the image and spectral data to identify a disease or disease identified from the biological sample A generation module for generating a prediction image including a prediction classification of a disease state;
A biological sample classification system, comprising: a transmission module configured to transmit a predicted image including the predicted classification for display on a display. A heuristic module for applying a heuristic to an image set selected from the group consisting of the real image, the predicted image and the confidence predicted image;
A specific module for identifying a sample area of interest of the biological sample based on the heuristic;
A collection module that assists in collecting a portion of the biological sample corresponding to a sample area of interest;
34. The system of claim 33, further comprising a test module that performs a molecular test on a portion of the biological sample. A computer program product,
Having a computer readable medium;
The computer-readable medium includes instructions for receiving an image of a biological sample and instructions for applying annotation information associated with a particular disease or condition to the image and spectral data to generate an actual image of the biological sample. Instructions for generating a predicted image including a predicted classification of a disease or condition identified from the biological sample by applying an algorithm to the image and spectrum data, and a predicted image including the predicted classification are displayed on a display And a computer program product comprising: instructions for transmitting to transmit.

Images