JP2021506003A - How to store and retrieve digital pathology analysis results - Google Patents

Abstract

The present disclosure is specifically directed to automated systems and methods for analyzing, storing, and / or retrieving information associated with living organisms having irregular shapes. In some embodiments, these systems and methods divide the input image into multiple sub-regions based on the localized color, texture, and / or intensity in the input image, where each sub-region , Represents biologically meaningful data.

Description

[0001] The present application claims the benefit of the filing date of US Provisional Patent Application No. 62 / 595,143 filed on December 6, 2017, all of which are disclosed herein by reference. Is incorporated into.

One embodiment of the present invention relates to, for example, a method for storing and retrieving digital pathological analysis results.

[0002] Digital pathology involves scanning an entire glass slide of pathological tissue or cells into a digital image that can be interpreted on a computer screen. These images will later be processed by an image processing algorithm or interpreted by a pathologist. To examine tissue sections (substantially transparent), tissue sections are created using colored histochemical stains that selectively bind to cellular components. The use of colored or stained cell structures by clinicians or computer-aided diagnosis (CAD) algorithms is used to identify morphological markers of the disease and proceed with treatment accordingly. By observing the assay, a variety of processes are possible, such as diagnosing the disease, assessing the response to treatment, and developing new medicines to combat the disease.

[0003] Immunohistochemical (IHC) slide staining can be used to identify proteins in cells of tissue sections, so that various types of cells, such as cancer cells and immune cells in biological tissues, Widely used in research. Therefore, in the case of immune response studies, IHC staining can be used in studies to understand the distribution and localization of biomarkers of different expression of immune cells (T cells, B cells, etc.) in cancer cells. For example, tumors often contain infiltrates of immune cells, which may prevent tumor growth or prioritize tumor growth.

[0004] In-situ hybridization (ISH) is used to confirm the presence or absence of genetic abnormalities in cells or conditions such as amplification of carcinogenic genes, which are considered to be morphologically malignant, especially when observed under a microscope. It can be used. ISH detects or locates a target nucleic acid target gene in a cell or tissue sample by adopting an antisense labeled DNA or RNA probe molecule for the target gene sequence or transcript. ISH is performed by exposing the cell or tissue sample to a labeled nucleic acid probe that is particularly hybridizable to a given target gene in the cell or tissue sample immobilized on a glass slide. Multiple target genes can be analyzed simultaneously by exposing cell or tissue samples to multiple nucleic acid probes labeled with multiple different nucleic acid tags. By utilizing labels with different emission wavelengths, simultaneous multicolor analysis can be performed in a single step on a single target cell or tissue sample.

One embodiment of the present invention relates to, for example, a method for storing and retrieving digital pathological analysis results.

[0005] The present disclosure relates, among other things, to automated systems and methods for analyzing and storing data associated with living organisms having irregular shapes (eg, fibroblasts or macrophages). The present disclosure also provides a medium resolution (mid-resolution or medium-resolution) analytical method, i.e., a method of grouping pixels with similar characteristics (eg, staining intensity, staining presence or absence, and / or texture) into "subregions". It relates to automated systems and methods for analyzing and storing data associated with living organisms.

[0006] In digital pathology, images are taken from biological specimens (eg, tissue specimens) mounted on glass slides and stained to identify biomarkers. Biological samples can be evaluated under a high magnification microscope or automatically analyzed by a digital pathology algorithm that detects and classifies organisms of interest. For example, the object of interest can be cells, blood vessels, glands, tissue regions, and the like. Any derived information may be stored in a database for later retrieval, which contains statistics on the presence or absence of biological structures of interest, spatial relationships, and / or staining properties. You may. As a matter of course to those skilled in the art, it is relatively easy to store and retrieve the analysis results of clearly distinguished cells (eg, tumor cells or immune cells). This is because such cells are represented by points at the center of each cell and can be stored in a database (see, eg, FIG. 4). Similarly, organisms with well-defined size and shape (eg, blood vessels) are represented by a simple outline, the coordinates of this outline are stored in a database and can be retrieved and / or analyzed separately later. (In this specification, it is also referred to as "polygon" or "polygon contour").

[0007] On the other hand, some biological structures of interest (eg, fibroblasts or macrophages) have an irregular shape. Groups of cells of this type may extend around each other or around other cells (see Figure 5). As a result, it is often difficult to accurately identify these irregularly shaped cells individually by an observer or an automated algorithm. Instead, these cells are very often identified solely by the localization of their respective stained cytoplasm or membrane, without identification of individual cells.

[0008] Such irregularly shaped structures are considered analytic and storable using high resolution analysis, but such techniques require considerable computer resources (calculation time and / or storage resources). Often becomes. In fact, high-resolution analysis techniques that store all pixel information for biological structures of interest (eg, analysis results for all pixels) consume software and hardware resources (eg, memory and processors that process or display the information). ) Too many, and after all, it is thought that it cannot provide meaningful results for a particular organism.

[0009] Such irregular structures could also be analyzed using low resolution analysis, and such low resolution data representation "groups" multiple individual cells into a single object. It may be stored in the database. As an example, FIGS. 6A and 6B show a tumor (yellow, 620) represented by a large polygonal outline (red, 630) that surrounds a group of related cells, excluding "pores" (turquoise, 640) in unwanted regions. ) And fibroblasts (purple, 610) are shown as an example of stained IHC images. In this example, the analysis results are averaged over a large area (red outline, 630) that can contain many individual cells with different characteristics (eg, shape, size, staining intensity, etc.). For example, for FIG. 6B, the contoured FAP positive area is 928.16 μm ² and the calculated fibroblast activated protein (FAP) positive average intensity is 0.26. Given the average intensity in such a large pixel area region, an average intensity of 0.26 is too coarse to show and represent the overall FAP positivity in this image. Without wishing to be bound by any particular theory, it is believed that this low resolution analysis technique can lead to reduced accuracy if the stored results are later used in downstream processing. Therefore, due to the heterogeneity of such stained cells, it is believed that this method does not locally present the actual details of such regions of biological structure of interest.

[00010] In contrast to the high-resolution and low-resolution analytical methods described above, the present disclosure uses medium-resolution analytical techniques to resemble image properties (eg, at least one of texture, intensity, or color). A system and a method for deriving data corresponding to irregularly shaped cells are provided by segmenting an image into a plurality of subregions.

[00011] In view of the above, one aspect of the present disclosure is a method of storing image analysis data derived from an image of a biological specimen having at least one stain, wherein (a) one or one from the image. A step of deriving a plurality of feature metrics and (b) a step of segmenting the image into a plurality of subregions, each subregion being substantially in at least one of the presence or absence of staining, the staining intensity, or the local texture. A step containing uniformly uniform pixels, (c) a step of generating multiple representation objects based on multiple segmented subregions, and (d) a step of associating each of the multiple representation objects with a derived feature metric. A method that includes (e) a step of storing the coordinates of each representation object in a database, along with an association-derived feature metric. As a matter of course to those skilled in the art, at least steps (a) and (b) may be performed in any order. In some embodiments, the step of segmenting an image into multiple subregions involves deriving superpixels. In some embodiments, superpixels are derived by grouping pixels by local k-means clustering and (ii) integrating small separation regions into large closest superpixels using a connected component algorithm. Without wishing to be bound by any particular theory, superpixels (as sub-regions) are perceptually consistent units, i.e. the color and texture of all pixels in the superpixel. Is considered to be perceptually meaningful so that can be uniform. In some embodiments, connected component labeling scans an image and groups the pixels into components based on pixel connectivity. That is, all the pixels of the connected component share similar pixel intensity values and are somehow connected to each other.

[00012] In some embodiments, the step of segmenting the image into multiple subregions is to superimpose the sampling grid on the image, where the sampling grid defines a non-superimposed area having a predetermined size and shape. Including that. In some embodiments, the subregion has a size of M × N, where M ranges from 50 pixels to 100 pixels and N ranges from 50 pixels to approximately 100 pixels.

[00013] In some embodiments, the representation object comprises a subregion outline that meets a predetermined staining intensity threshold. In some embodiments, the representation object comprises a seed point. In some embodiments, seed points are derived by calculating the centroids of each of the plurality of subregions. In some embodiments, the derived feature metric is the staining intensity, and the average staining intensity of all pixels within each generated representation object outline is calculated. In some embodiments, the derived feature metric is the expression score, and the average expression score corresponding to the area within each generation subregion is associated with the plurality of generation expression objects. In some embodiments, the method further comprises reading stored coordinates and associated feature metrics data from the database and projecting the read data onto an image. In some embodiments, the analysis results (eg, intensity, area) within the corresponding sub-region can be stored in the form of average pixel measurement results representing the pixel data of the sub-region.

[00014] In some embodiments, the biological sample is stained with membrane stain. In some embodiments, the biological sample is stained with membrane stain and nuclear stain. In some embodiments, the biological sample is stained with at least FAP and one or more derived feature metrics include at least one of FAP staining intensity or FAP percentage positive. In some embodiments, the average FAP percentage positiveness is calculated for all pixels in the subdomain. In some embodiments, the average FAP staining intensity is calculated for all pixels in the subregion. In some embodiments, the sample is stained with FAP and H & E. In some embodiments, the sample is stained with FAP as well as another core or membrane stain.

[00015] In some embodiments, the image received as input is initially separated into an image channel image (eg, an image channel image of a particular stain). In some embodiments, the region of interest is selected prior to image analysis.

[00016] Another aspect of the present disclosure is a system for deriving data corresponding to irregularly shaped cells from an image of a biological sample containing at least one stain, wherein (i) one or more. A processor and (ii) a memory coupled to one or more processors that, when executed by one or more processors, (a) derive one or more feature metrics from the image. And (b) to generate multiple sub-regions in the image, each sub-region having pixels with similar qualities selected from color, brightness, and / or texture. c) compute a series of representation objects based on multiple generated subregions, and (d) associate one or more derived feature metrics from the image with the computational coordinates of each of the sequence representation objects. A system comprising a memory that stores computer-executable instructions that cause one or more processors to perform operations including. In some embodiments, the subregions are (i) adjacent cells, (ii) pixels with perceptually meaningful similar properties (eg, color, brightness, and / or texture), and (iii) organisms. It is formed by grouping pixels that are substantially homogeneous with respect to biological properties (eg, biological structure, staining properties of biological structure, cell properties, groups of cells). In some embodiments, the pixels in the subregion have similar properties and descriptive statistics with respect to the organism of interest (eg, irregularly shaped cells, including, but not limited to, fibroblasts and macrophages). Has.

[00017] In some embodiments, segmenting an image into multiple subregions involves deriving superpixels. In some embodiments, superpixels are derived using either a graph-based approach or a gradient rise-based approach. In some embodiments, superpixels are derived by grouping pixels by local k-means clustering and (ii) integrating small separation regions into large closest superpixels using a connected component algorithm.

[00018] In some embodiments, the representation object comprises a subregion outline that meets a predetermined staining intensity threshold. In some embodiments, the representation object comprises a seed point. In some embodiments, the system further includes instructions for storing the coordinates of one or more derived feature metrics and associated computational representation objects in the database. In some embodiments, one or more derived feature metrics include at least one expression score selected from percentage positiveness, H-score, or staining intensity. In some embodiments, the data corresponding to the irregularly shaped cells is derived for the region of interest in the image. In some embodiments, the area of interest is the area of the image annotated by a healthcare professional.

[00019] Another aspect of the present disclosure is a non-temporary computer-readable medium containing instructions for analyzing data associated with an irregularly shaped organism, wherein the instructions are (a) of a biological sample. An instruction that derives one or more feature metrics from an image, in which the biological sample contains at least one stain, and (b) a series of subordinates to the image by grouping pixels with similar traits. An instruction that divides into regions, the characteristics of which are selected from color, brightness, and / or texture, and (c) an instruction that calculates a plurality of representation objects based on a series of division sub-regions. d) A non-temporary computer-readable medium that includes an instruction that associates one or more derived feature metrics from an image with the calculated coordinates of each of a plurality of computational representation objects.

[00020] In some embodiments, dividing an image into a series of sub-regions involves computing superpixels. In some embodiments, superpixels are calculated using one of a normalized cut algorithm, a cohesive clustering algorithm, a quickshift algorithm, a turbopixel algorithm, or a simple linear iterative clustering algorithm. In some embodiments, superpixels are generated using simple iterative clustering, with the superpixel size parameter set to about 40 pixels to about 400 pixels and the density parameter set to about 10 to about 100. To. In some embodiments, superpixels are calculated by grouping the pixels by local k-means clustering and using (ii) the connected component algorithm to integrate the small separation regions into the larger closest superpixels.

[00021] In some embodiments, the biological sample is stained with at least FAP and one or more derived feature metrics include at least one of FAP staining intensity or FAP percentage positive. In some embodiments, the average FAP percentage positiveness is calculated for all pixels in the subdomain. In some embodiments, the average FAP staining intensity is calculated for all pixels in the subregion. In some embodiments, the representation object comprises at least one of a polygon outline and a seed point. In some embodiments, the memory includes instructions for storing the coordinates of one or more derived feature metrics and association computational representation objects in the database. In some embodiments, the memory further comprises an instruction to project the stored information onto an image of a biological sample.

[00022] Applicants provide an improved solution in which the systems and methods described herein store biological analysis results that cannot be defined in a single position or outline for each object of interest. showed that. In addition, Applicants believe that the systems and methods described herein can reduce the storage space for storing analysis results as compared to pixel-level high-resolution analysis techniques. This is because the analysis results of a specific pixel and the pixels around it are integrally stored in the sub-region, and the pixels in the sub-region have similar characteristics or characteristics (for example, color, brightness, texture). In addition, Applicants have generated image complexity from thousands of pixels to a smaller and more manageable number of subregions that allow for significantly faster retrieval and reporting of analysis results. Therefore, it is considered that the above system and method are computationally efficient. Applicants also consider that the sub-regions are neither too small nor too large for the storage and representation of the analysis results, which is expressively efficient. Finally, Applicants believe that the systems and methods disclosed herein can improve accuracy, especially as compared to low resolution analytical techniques. The generated subregion is to describe the characteristics or statistical information of the biologically relevant object of interest as compared to the storage of information from the large domain representation (ie, the subregion is stained, stained, stained. Includes pixels where the intensity and texture are as uniform as possible). The above and other advantages are described separately herein.

[00023] Reference is made to the drawings for a general understanding of the features of the present disclosure. The same reference numbers are used to identify the same elements throughout the drawing.

[00024] FIG. 6 shows a typical digital pathology system comprising an image acquisition device and a computer system according to some embodiments. [00025] FIG. 5 illustrates various modules available in a digital pathology system or digital pathology workflow, according to some embodiments. [00026] FIG. 5 is a flow chart showing various steps for deriving image analysis data and related image analysis data by a generation subregion, according to some embodiments. [00027] A diagram showing an example of digital pathological images of liver cancer cells at a high level resolution according to some embodiments, after image analysis processing and classification, the analysis results (eg, cells). Annotation points (red = positive stained tumor cells (410), green = negative stained tumor cells (420)) can be stored and read from the database and displayed, and each annotation point can display read information. It may include (eg, descriptive statistics of the presence or absence of biological structures of interest, spatial relationships, and staining properties). [00028] FIG. 5A is a diagram showing normal fibroblasts with different appearances (eg, irregular size, shape, and boundaries of cells) and morphologically heterogeneous appearances of fibroblasts. is there. FIG. 5B shows the appearance of morphologically heterogeneous fibroblasts with different appearances (eg, irregular size, shape, and boundaries of cells). FIG. 5C shows the appearance of morphologically heterogeneous fibroblasts with different appearances (eg, irregular size, shape, and boundaries of cells), with hematoxylin and eosin staining of normal activated fibroblasts (eg H & E) It is a figure which showed the image. FIG. 5D shows the appearance of morphologically heterogeneous fibroblasts with different appearances (eg, irregular size, shape, and boundaries of cells), with hematoxylin and eosin staining (H & E) of activated fibroblasts. It is a figure which showed the image. [00029] FIG. 6A shows an example of fibroblast immunohistochemistry (IHC) associated with tumor cells, where fibroblasts (610) are stained purple and tumors (620) It is a diagram that stains yellow and can have a very irregular shape extending to the opposite side or perimeter of other cells as well as the fibroblasts in contact with other cells. [00030] FIG. 6B shows an example of a low resolution polygon outline (red, 630) in an area with turquoise positive fibroblast expression and exclusion regions (pores, 640). [00031] FIG. 6 shows a subregion (710) of a simple shape (eg, a circle) that can be associated with image data using the medium resolution technique described herein. [00032] It is a figure which showed an example of the super pixel generated by using SLIC of a fibroblast region on an IHC image. [00033] FIG. 6 shows a high magnification original IHC image in which tumor cells (830) were stained yellow and fibroblasts (840) were stained purple. [00034] FIG. 5 shows the initial shape of a superpixel that looks similar to a square before adjustment of the regularization parameters, according to some embodiments. [00035] FIG. 5 shows the final representation of superpixels with adjusted regularization parameters of the SLIC algorithm, according to some embodiments. [00036] It is a figure which showed the polygon outer shape (black, 910) of the sub-region (here, superpixel) belonging to the region of interest (fibroblast region) which concerns on some embodiments. [00037] It is a figure which showed the polygon outer shape (black, 920) and the central seed (green dot, 930) of the sub-region (super pixel) belonging to the organism of interest (fibroblast) which concerns on some embodiments. [00038] Head and neck cancer tissue stained purple with fibroblast-activating protein (FAP) for fibroblasts (1010) and yellow with pancytokeratin (PanCK) for epithelial tumors (1020) It is a figure which showed an example of all slide IHC images of. [00039] It is a figure which showed an example of the polygon outer shape which was given the analysis result of the superpixel (blue, 1030) belonging to the fibroblast region which can be stored in a database. [00040] It is a figure which showed an example of the central seed to which the analysis result of the superpixel (red, 1140) belonging to the fibroblast region which can be stored in a database was given. [00041] FIG. 6 shows an example of a histogram plot of FAP intensities read from all slide superpixels. [00042] It is a flowchart which showed the step of area selection which concerns on some embodiments. [00043] FIG. 6 shows six different annotation shapes and regions in an image of a biological sample. [00044] Concordance of FAP positive area ratios between FAP + area determined using (i) high resolution analytical techniques and (ii) exemplary medium resolution (subregion) techniques described herein was shown. It is a figure.

[00045] Unless otherwise expressly specified, in any method claimed herein that includes two or more steps or actions, the order of the steps or actions shall be in the order in which the steps or actions are listed. It shall be understood that it is not necessarily limited.

[00046] As used herein, the singular terms "a", "an", and "the" include a plurality of referents unless expressly specified in the context. Similarly, the word "or" is intended to include "and" unless otherwise explicitly specified in the context. The term "includes" is defined inclusively to mean that "includes A or B" includes A, B, or A and B.

[00047] As used herein and in the claims, it is understood that "or" has the same meaning as "and / or (and / or)" as defined above. For example, when separating items in a list, "or (or)" or "and / or (and / or)" is interpreted as inclusive, with many or list elements and optionally additional. It is possible to include at least one of the items not on the list, and to include two or more of them. Used in terms or claims with other explicit specifications, such as "only one of" or "exactly one of". Only the term "consisting of" refers to the inclusion of just one element of many or list elements. In general, the term "or" as used herein is exclusive when preceded by an exclusive term such as "eacher", "one of", "only one of", or "exactly one of". (Ie, one or the other, but not both (one or the other but not both) "). As used in the claims, "consisting essentially of" shall have the usual meaning when used in the field of patent law.

[00048] The terms "comprising", "inclusion", "having" and the like are used interchangeably and have the same meaning. Similarly, "comprises", "includes", "has" and the like are used without distinction and have the same meaning. Specifically, each of these terms is open, meaning "at least the following," because each of these terms is defined consistently with the general definition of "comprising" in 35 USC. It is interpreted as a term and does not exclude additional features, limitations, aspects, etc. For this reason, for example, "a device having components a, b, and c" means that the device includes at least the components a, b, and c. .. Similarly, the expression "method invoking steps a, b, and c" means that the method comprises at least steps a, b, and c. Moreover, although the steps and processes may be described herein in a particular order, those skilled in the art will recognize that the order of the steps and processes can vary.

[00049] As used herein and in the claims, the expression "at least one" refers to one or more elements of the list and is any one of the elements in the element list. Or, it means at least one element selected from a plurality of elements, but does not necessarily include at least one of all the elements specifically listed in the element list, and does not exclude any combination of elements in the element list. Shall be understood. Also, according to this definition, elements other than these elements may or may not relate to the specifically identified elements in the element list referenced by the expression "at least one". Can exist as an optional choice. Therefore, as a non-limiting example, "at least one of A and B (at least one of A and B)" (or equivalently, "at least one of A or B (at least one of)". "A or B)" or "at least one of A and / or B (at least one of A and / or B)") in one embodiment (two or more as an option). A (B does not exist (optionally, includes elements other than B)), and in another embodiment, at least one (optionally, two or more) B (A does not exist (optional)). (Including elements other than A as an option)), and in yet another embodiment, at least one (two or more as an optional option) A and at least one (two or more as an optional option) B (optional). As a choice, other elements may be included) and the like.

[00050] In the present specification, the term "biological sample" (used herein without distinction from the term "biological specimen" or "specimen") or A "tissue sample" (used herein indistinguishable from the term "tissue specimen") is a biomolecule (protein, peptide) obtained from any organism, including a virus. , Nucleic acid, lipid, sugar, or a combination thereof, etc.). Other examples of organisms include mammals (eg, domestic animals such as humans, cats, dogs, horses, cows, and pigs, and laboratory animals such as mice, rats, and primates), insects, ring-shaped animals, and arachnids. Examples include arachnids, bagging animals, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (eg, tissue sections and needle biopsy specimens of tissue), cell samples (eg, cytological smears such as Pap or blood smears, or cells obtained by microanalysis. Samples), or cell fractions, cell debris, or organelles (eg, cells lysed and cell components separated by centrifugation or the like). Other examples of biological samples are blood, serum, urine, semen, fecal matter, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, (eg, obtained by surgical biopsy or needle biopsy). T) Biopsy tissue, papillary inhalation fluid, ear stains, breast milk, vaginal fluid, saliva, swabs (oral swabs, etc.), or any substance containing biomolecules derived from a first biological sample. In certain embodiments, the term "biological sample" herein refers to a sample (homogenized or liquefied sample) made from a tumor or portion thereof obtained from a subject.

[00051] As used herein, the term "biomarker" or "marker" refers to a measurable indicator of any biological condition or condition. In particular, the biomarker may be a protein or peptide (eg, a surface protein) that is particularly stainable and exhibits the biological characteristics of the cell (eg, cell type or physiological state of the cell). Immune cell markers are biomarkers that selectively indicate features of a mammalian immune response. Biomarkers may be used to determine the body's response to a disease or treatment of a disease or to determine whether a subject is vulnerable to a disease or disease. With respect to cancer, biomarkers represent biological substances that indicate the presence or absence of cancer in the body. The biomarker may be a molecule secreted by the tumor or a specific response of the body to the presence of cancer. Genetic biomarkers, epigenetic biomarkers, proteome biomarkers, glicocom biomarkers, and imaging biomarkers can be used for cancer diagnosis, prognosis, and epidemiology. Such biomarkers can be tested in non-invasively collected biofluids such as blood or serum. Multiple gene and protein-based biomarkers have already been used in patient care, including AFP (liver cancer), BCR-ABL (chronic myeloid leukemia), BRCA1 / BRCA2 (breast / ovarian cancer), BRAF V600E ( Black tumor / colorectal cancer), CA-125 (ovarian cancer), CA19.9 (pancreatic cancer), CEA (colorectal cancer), EGFR (non-small cell lung cancer), HER-2 (breast cancer), KIT ( Gastrointestinal stromal tumors), PSA (Prostate Specific Antigen), S100 (Breast Cancer), and many others, but are not limited to these. Biomarkers provide diagnostic methods (identifying early-stage cancer) and / or prognosis (predicting the progression of cancer and / or the subject's response to specific treatments and / or the likelihood of cancer recurrence). It is considered to be useful as a diagnostic method.

[00052] As used herein, the term "image data" is, as understood herein, raw image data or preprocessing obtained from a biological tissue sample by an optical sensor or sensor array or the like. Includes image data. In particular, the image data may include a pixel matrix. As used herein, the term "immunohistochemistry" refers to a method of detecting the interaction of an antigen with a particular binder, such as an antibody, to determine the presence or absence or distribution of the antigen in a sample. The sample is contacted with the antibody under conditions that allow antibody-antigen binding. The antibody-antigen binding can be detected by a detectable label attached to the antibody (direct detection) or by a detectable label attached to a sub-antibody that specifically binds to the main antibody (indirect detection). .. As used herein, "mask" is a derivative of a digital image in which each pixel of the mask is a binary value (eg, "1" or "0" (or "true" or "false"). )). By overlaying the mask on the digital image, in a separate processing step applied to the digital image, all pixels of the digital image mapped to one particular mask pixel of the binary value are hidden, deleted, or hidden. Ignored or filtered out. For example, a mask is generated from the original digital image by assigning an intensity value above the true or false threshold to all pixels in the original image and removing all pixels where the "false" masked pixels overlap. obtain. A "multi-channel image", as understood herein, is a specific fluorescent dye, quantum dot, chromogen, etc. that constitutes one of the channels of a multi-channel image by fluorescence or detectability in different spectral bands. Includes digital images obtained from biological tissue samples that simultaneously stain different biological structures such as nuclear and tissue structures.

[00053] Overview
[00054] Applicants have developed a system and method for storing the analysis results of an irregularly shaped organism such as fibroblasts or macrophages in a non-temporary memory such as a database. The results of the analysis may later be read from the database or memory for further analysis or use in other downstream processes. In addition, the analysis result may be projected onto an input image or another derived image, or may be visualized by other means. The present disclosure also stores analysis results at an adjustable level of detail by allowing the size of the generated subregions to be adjusted (eg, by increasing or decreasing the size of the simple shape or adjusting the parameters of the Superpixel algorithm). May facilitate reporting. This is believed to be able to improve efficiency and accuracy as compared to the low resolution analysis techniques described herein in which average analysis results from the global area of interest are stored.

[00055] As described separately herein, the disclosed systems and methods are based on medium resolution analytical techniques that store analytical results using locally similar small regions (subregions). The sub-regions can be simple shapes (eg, circles, squares) or complex shapes (eg, superpixels) to store local analysis of each subregion across the entire slide. Used for. The sub-regions defined by this medium resolution technique are similar (or homogeneous) properties (eg, with or without staining (ie, with or without specific stains), staining intensity (ie, relative intensity (or amount) of stain)). , Grouping pixels with local texture (ie, information about the spatial arrangement of colors or intensities in the image or selection area of the image) to allow identification of irregularly shaped objects. In some embodiments, the sub-region in the medium resolution approach has a size in the range of about 50 to about 100 pixels or a pixel area of about 2,500 pixels ² to about 10,000 pixels ² . Of course, the subregion may have any size, which size may be based on the type of analysis performed and / or the type of cell being investigated.

[00056] As a matter of course to those skilled in the art, the medium-level method is between the high-resolution analysis method and the low-resolution analysis method described herein, so that data is collected at the subregion level as well as being collected. The sub-region is proportionally smaller than the region of interest in the low-resolution technique and clearly larger than the pixels in the high-resolution analysis technique. "High resolution analysis" means that image data is captured at the pixel level or substantially at the pixel level. On the other hand, "low resolution analysis" represents an area level analysis, such as an area having a size of at least 500 pixels x 500 pixels or an area having a size greater than 250,000 pixels ² . As a matter of course to those skilled in the art, low resolution analytical techniques will involve many living organisms (eg, cells of multiple irregular shapes).

[00057] The present disclosure may represent the analysis and storage of living organisms with irregular shapes and / or sizes, such as fibroblasts or macrophages. It is understood that the present disclosure is not limited to fibroblasts or macrophages and can be extended to any organism whose size or shape is not well defined.

[00058] With respect to fibroblasts, they are the cells that make up the structural skeleton or stroma composed of extracellular matrix and collagen in animal tissues. These cells are the most common type of connective tissue in animals and are important for wound healing. Fibroblasts come in a variety of shapes and sizes, as well as activated and deactivated forms (see, eg, FIGS. 5A-5D). Fibroblasts are in the activated form (the suffix "blast" represents metabolically active cells), while fibrosis-related cells are considered hypoactive. However, both fibroblasts and fibrosis-related cells are not designated as different, and may be simply referred to as fibroblasts. Fibroblasts are morphologically distinguishable from fibrosis-related cells by the abundance and relatively large size of the rough endoplasmic reticulum. In addition, fibroblasts are thought to come into contact with their respective adjacent cells, and these contacts are considered attachments that can distort the morphology of isolated cells. The medium resolution analytical techniques presented herein can take into account these morphological differences and are considered optimal for storing information about fibroblasts, macrophages, and other irregular organisms.

[00059] FIG. 1 shows a digital pathology system 200 for imaging and analyzing specimens, according to some embodiments. The digital pathology system 200 includes an imaging device 12 (for example, a device having means for scanning a microscope slide on which a specimen is placed) and a computer 14, so that the imaging device 12 and the computer are integrally (for example, a network 20). They may be communicably coupled (directly or indirectly) through. The computer system 14 contains digital electronic circuits, firmware, hardware, memory, computer storage media, computer programs or instruction sets (eg, programs are stored in memory or storage media, such as desktop computers, laptop computers, tablets, etc.). If), it may include one or more processors (including programmed processors), as well as any other hardware, software, or firmware module, or a combination thereof. For example, the computer system 14 shown in FIG. 1 may include a computer having a display device 16 and a housing 18. A computer can store a binary digital image (locally, such as in memory, on a server, or on another network-attached device). Also, the digital image can be divided into matrix pixels. Pixels may contain digital values of one or more bits defined by bit depth. Other computer devices or systems may, of course, be utilized by those skilled in the art, and the computer systems described herein are additional components (eg, sample analyzers). , Microscopes, other imaging systems, automated slide creation devices, etc.) may be communicatively coupled. Some of these additional components and the various computers, networks, etc. available are described separately herein.

[00060] Generally, the image pickup device 12 (or another image source in which the preliminary scan image is stored in the memory) includes, but is not limited to, one or more image capture devices. The image capture device includes a camera (for example, an analog camera, a digital camera, etc.), an optical element (for example, one or more lenses, a sensor focal lens group, a microscope objective lens, etc.), and an imaging sensor (for example, a charge-coupled device (CCD). ), Complementary metal oxide semiconductor (CMOS) image sensor, etc.), photographic film, etc., but is not limited thereto. In a digital embodiment, the image capture device may include multiple lenses that work together to prove on-the-fly focusing. An image sensor (eg, a CCD sensor) can capture a digital image of the specimen. In some embodiments, the imaging device 12 is a brightfield imaging system, a multispectral imaging (MSI) system, or a fluorescence microscopy system. Digitized tissue data can be obtained, for example, from VENTANA MEDICAL SYSTEMS, Inc. It may be generated by an image scanning system such as a VENTANA iScan HT scanner by (Tucson, Arizona) or other suitable imaging device. Additional imaging devices and systems are described separately herein. As a matter of course to those skilled in the art, the digital color image acquired by the image pickup apparatus 12 may conventionally be composed of primary color pixels. Each colored pixel can be encoded on three digital components, each containing the same number of bits, and each component is a primary color (generally red, green, or blue, also represented as the term "RGB" component. Corresponds to.

[00061] FIG. 2 outlines the various modules utilized in the digital pathology system disclosed above. In some embodiments, the digital pathology system employs a computer device 200 or computer mounting method having one or more processors 203 and at least one memory 201, with at least one memory 201 being one. Or stores non-temporary computer-readable instructions that cause one or more processors to execute instructions (or stored data) in one or more modules (eg, modules 202 and 205-209) by execution by multiple processors. doing.

[00062] With reference to FIGS. 2 and 3, the present disclosure stores analysis of living organisms having irregular shapes, such as fibroblasts or macrophages, and / or storage of analysis results in non-temporary memory such as databases. Provides a computer implementation method to do. In this method, for example, (a) the image acquisition module 202 is operated to generate or receive multichannel image data (for example, an acquired image of a biological sample stained with one or more stains) (step). 300) and (b) the step (step 310) of operating the image analysis module 205 to derive one or more metrics from the features in the acquired image, and (c) operating the segmentation module 206. , A step of segmenting the acquired image into a plurality of sub-regions (step 320) and (d) a step of operating the expression object generation module 207 to generate a polygon, a central seed, or another object that identifies the sub-region. (Step 330), (e) run the labeling module 208 to associate one or more derived metrics with the generated representation object (step 340), and (f) the representation object and associated metrics. It may include a step (step 350) to be stored in the database 209. As a matter of course to those skilled in the art, the workflow may include additional modules or databases. For example, the behavior of the image processing module may apply a particular filter to the acquired image or identify a particular histological and / or morphological structure within the tissue sample. In addition, a specific part of the image to be analyzed may be selected by using the region of interest selection module. Similarly, the operation of the separation module may be to provide an image channel image corresponding to a particular stain or biomarker.

[00063] Image acquisition module
[00064] In some embodiments, the digital pathology system 200, as a first step, operates the image acquisition module 202, with reference to FIG. 2, a biological sample having one or more stains. Import the image or image data of. In some embodiments, the received or acquired image is an RGB image or a multispectral image (eg, a multiplexed brightfield and / or darkfield image). In some embodiments, the captured image is stored in memory 201.

[00065] The image or image data (used in this specification without distinction) may be acquired by the image pickup apparatus 12 in real time or the like. In some embodiments, the image is obtained from an instrument such as a microscope capable of capturing image data of a microscope slide on which the specimen is placed, as described herein. In some embodiments, the image is acquired using a line scanner capable of scanning the image line by line, such as a 2D scanner capable of scanning image tiles or a VENTANA DP 200 scanner. Alternatively, the image may be an image previously acquired (eg, scanned) and stored in memory 201 (or, in this regard, read from a server via network 20).

[00066] The biological sample may be stained by application of one or more stains, resulting in the image or image data containing a signal corresponding to each of the one or more stains. As such, the systems and methods described herein are capable of estimating or normalizing a single stain (eg, hematoxylin), but are not limited in the number of stains in a biological sample. In fact, the biological sample may be stained in a multiplex assay of two or more stains as an addition or inclusion of any counterstain.

Of course to those skilled in the art, biological samples may be adapted to be stained against different types of core and / or cell membrane biomarkers. Methods for staining and guiding tissue structures in the selection of stains suitable for a variety of purposes are described, for example, in Sambrook et al. "Molecular Cloning: A Laboratory Manual (Molecular Cloning: Laboratory Manual)", Cold Spring Harbor Laboratory Press (1989) and Ausube et al. It is described in "Curent Protocols in Molecular Biology", Greene Publishing Associates and Willy-Intersciences (1987), the contents of which are incorporated herein by reference.

[00068] As a non-limiting example, in some embodiments, tissue samples are stained in an IHC assay for the presence of one or more biomarkers containing a fibroblast activating protein (FAP). To. Overexpression of FAP in fibroblast cell lines is thought to promote malignant behavior. Interstitial fibroblasts, an essential component of the tumor microenvironment and often designated as cancer-related fibroblasts (CAFs), are by multiple mechanisms such as proliferation, angiogenesis, invasion, survival, and immunosuppression. It has been shown that it can promote tumorigenesis and progression. Without wishing to be bound by any particular theory, cancer cells activate interstitial fibroblasts and induce FAP expression, which affects the growth, invasion, and migration of cancer cells. Conceivable. FAP is thought to be highly expressed in reactive stromal fibroblasts in 90% of human epithelial cell carcinomas, including breast, lung, colorectal, ovary, pancreas, and head and neck. For this reason, the amount of FAP is most likely to provide an important prognostic diagnosis for the clinical behavior of the tumor (which is a measure of certain metrics that can be associated with the generated subregion or representation object after derivation). This is an example).

[00069] Chromatic stains include hematoxylin, eosin, fast red, or 3,3'-diaminobenzidine (DAB). Also, of course to those skilled in the art, any biological sample may be stained with one or more fluorophores. In some embodiments, the tissue sample is stained with a major stain (eg, hematoxylin). In some embodiments, tissue samples are stained in the IHC assay for specific biomarkers. The sample may also be dyed with one or more fluorescent dyes.

[00070] Normal biological samples are processed in an automated staining / assay platform where stains are applied to the samples. There are a variety of products on the market suitable for use as staining / assay platforms, for example Ventana Medical Systems, Inc. (Tucson, AZ) Discovery ™ products are available. Camera platforms also include brightfield microscopes (such as Ventana Medical Systems, Inc.'s VENTANA iScan HT or VENTANA DP 200 Scanner) or any microscope with one or more objective lenses and digital imaging devices. Other techniques for capturing images at various wavelengths may be used. Other camera platforms suitable for imaging stained biological specimens are known in the art and are commercially available from companies such as Zeiss, Canon, and Applied Spectral Imaging. Also, such platforms are readily adaptable for use in the systems, methods, and devices of the present disclosure.

[00071] In some embodiments, the input image is masked so that only tissue regions are present in the image. In some embodiments, a tissue area mask is generated to mask the non-tissue area from the tissue area. In some embodiments, the tissue region is identified and the background region (eg, the region of the entire slide image corresponding to the sampleless glass, such as where only white light from the imaging source is present) is automatically or Tissue area masks may be formed by semi-automatically (ie, with minimal user input) exclusion. Of course, those skilled in the art need masking from tissue areas of non-tissue areas, as well as tissue masking modules such as parts of tissue identified as belonging to a particular tissue type or area suspected of being a tumor. Other areas of interest may be masked accordingly. In some embodiments, segmentation techniques are used to generate a tissue region masking image by masking the tissue region from the non-tissue region in the input image. Suitable segmentation techniques are known from prior art (“Digital Image Processing”, Third Edition, Rafael C. Gonzarez, Richard E. Woods, chapter 10, page 689 and “Hand”. Medical Imaging (Medical Imaging Handbook) ”, Processing and Analisis, Isaac N. Bankman Academic Press, 2000, chapter 2). In some embodiments, image segmentation techniques are used to distinguish between digitized tissue data in images and slides, in which case the tissue corresponds to the foreground and the slides to the background. In some embodiments, the component computes the area of interest (AoI) in the entire slide image to limit the amount of background non-organizational area analyzed while covering all tissue areas in the AoI. To detect. For example, extensive image segmentation techniques (eg, HSV color-based image segmentation, Lab image segmentation, average shift color image segmentation, region expansion, level) are used to determine the boundaries between tissue data and non-organizational or background data. Setting method, high-speed marching method, etc.) can be used. The components can also generate tissue foreground masks that can be used to identify parts of digitized slide data that correspond to tissue data, at least in part based on segmentation. Alternatively, the component can generate a background mask used to identify a portion of digitized slide data that does not correspond to tissue data.

[00072] This identification can be made possible by an image analysis operation such as edge detection. Tissue area masks may be used to remove non-tissue background noise in images (eg, non-tissue areas). In some embodiments, the generation of the tissue area mask is such that the pixels whose brightness is above a given threshold are set to 1 and the pixels below the threshold are set to 0 so that the tissue area mask is generated. One or more of operations such as calculating the luminance of a low-resolution input image, generating a luminance image, applying a standard deviation filter to a luminance image, generating a filtering luminance image, and applying a threshold to a filtered luminance image. Includes (but is not limited to these behaviors). For additional information and examples regarding the generation of tissue area masks, see "An Image Processing Method and System For Analyzing a Multi-Channel Image System Obtiined from System System System (Multiple Tissue). An image processing method and system for analyzing a multi-channel image obtained from a tissue sample) ”is disclosed in PCT / EP / 2015/062015, the entire disclosure of which is incorporated herein by reference. ..

[00073] In some embodiments, the region of interest identification module is used to select the portion of the biological sample from which the image or image data should be obtained (eg, the region of interest with a high concentration of fibroblasts). You may be able to do it. FIG. 13 is a flowchart showing the steps of region selection according to some embodiments. In step 420, the region selection module receives the identified region of interest or field of view. In some embodiments, the area of interest is identified by a user of the system of the present disclosure or another system communicatively coupled to the system of the present disclosure. Alternatively, in another embodiment, the region selection module reads the location or identification information of the region of interest from the storage / memory. In some embodiments, as shown in step 430, the region selection module automatically generates a field of view (FOV) or region of interest (ROI), eg, by the method described in PCT / EP2015 / 062015. All disclosures thereof are incorporated herein by reference. In some embodiments, the region of interest is automatically determined by the system in the image or based on some predetermined criterion or characteristic of the image (eg, a biological sample stained with three or more stains). In the case of, identify the area of the image containing only two stains). At step 440, the region selection module outputs the ROI.

[00074] Image analysis module
[00075] In some embodiments, features in the image received as input derive specific metrics (eg, FAP-positive area, FAP-positive intensity) (step 300) (see FIG. 3). .. Derived metrics may be correlated with the subregions generated herein (steps 320, 330, and 340), and the metrics (or their mean, standard deviation, etc.) and The location of the sub-region may be stored in the database as a unit (step 350) for later reading and / or downstream processing. The procedures and algorithms described herein are the derivation of metrics from different types of cells or cell nuclei, including the derivation of metrics from fibroblasts and / or macrophages, and / or different types of cells. Alternatively, it may be configured to classify cell nuclei.

[00076] In some embodiments, the metric is the detection of nuclei in the input image and / or the detected nuclei (such as image patches around the detected nuclei) and / or the cell membrane (of course, in the input image). Derived by extracting features from (determined by the biomarkers used). In other embodiments, metrics are derived by analyzing cell membrane staining, cytoplasmic staining, and / or emphasis staining (eg, to distinguish between membrane stained and non-membrane stained areas). As used herein, the term "cytoplasmic staining" refers to a group of pixels arranged in a pattern with morphological characteristics of the cytoplasmic region of a cell. As used herein, the term "membrane staining" refers to a group of pixels arranged in a pattern with morphological characteristics of a cell membrane. As used herein, the term "punctate staining" refers to a group of pixels containing localized staining intensities that appear as spots / dots scattered in the membrane area of a cell. As a matter of course to those skilled in the art, cell nuclei, cytoplasms, and membranes have different properties, so tissue samples with different stains can reveal different biological characteristics. In fact, as will be appreciated by those skilled in the art, certain receptors on the cell surface may have a staining pattern localized to the membrane or cytoplasm. For this reason, the "membrane" staining pattern is analytically different from the "cytoplasmic" staining pattern. Similarly, the "cytoplasmic" and "nucleus" staining patterns are analytically different. For example, stromal cells can be strongly stained with FAP, while tumor epithelial cells can be strongly stained with EpCAM, while cytokeratin can be stained with PanCK. Therefore, during image analysis, the use of different stains may allow different cell tumors to be differentiated and distinguished, or different metrics may be derived.

[00077] US Pat. No. 7,760,927 ("'". For methods of identifying and / or scoring nuclei, cell membranes, and cytoplasms in images of biological samples with one or more stains. 927 patents ”), the entire disclosure of which is incorporated herein by reference. For example, the '927 patent is an automated method for simultaneously identifying multiple pixels in an input image of biological tissue stained with biomarkers, with multiple in the foreground of the input image for simultaneous identification of cytoplasmic and cell membrane pixels. A step of considering the first color plane of a pixel, a step in which the input image has been processed to remove a background portion of the input image and a counter-staining component of the input image, and a digital image. The step of determining the threshold level between the pixels of the cytoplasm and the cell membrane in the foreground, and where the selected pixels from the foreground and their eight adjacent pixels are used, at the same time, the selected pixels are the cytoplasmic pixels in the digital image. A method is described that includes a step of determining, whether it is a cell membrane pixel or a transition pixel, using a determination threshold level. In addition, the '927 patent states that the step of simultaneously determining the selected pixel and its eight adjacent pixels determines the square root of the product of the selected pixel and its eight adjacent pixels, and compares the product to the determination threshold level. And, based on the comparison, incrementing the first counter for the cell membrane, the second counter for the cytoplasm, or the third counter for the transition pixel, and the first counter, the second counter, Alternatively, it is determined whether or not the third counter exceeds the predetermined maximum value, and if it exceeds the predetermined maximum value, it is described that the selected pixels are classified based on the counter that exceeds the predetermined maximum value. .. In addition to nuclear scoring, the '927 patent provides examples of cytoplasmic and membrane scoring based on computational cytoplasmic pixel volume index, cytoplasmic median intensity, membrane pixel volume, membrane pixel median intensity, etc., respectively.

[00078] Another method for identifying and / or scoring membrane, nucleus, and other cell features of interest is described in PCT Publication WO 2017/037180 (“'180 Publication”), the entire disclosure of which is described. Incorporated herein by reference. In addition, the '180 publication describes a method of quantifying membrane staining of a specimen of interest in a biological sample when the area of membrane staining is mixed with cytoplasmic staining and / or emphasis staining. To achieve this, the '180 publication is a step of (A) segmenting a digital image of a tissue or cytological sample into a plurality of different regions based on a sample staining pattern, with the plurality of regions being at least one. A region of an image having a sample-positive stain in a first biological compartment mixed with a composite stained region, i.e. a sample-positive stain in at least one second biological compartment, comprising the first biology. The candidate biological compartment, i.e. at least the first biological compartment, is separate from the step and (B) step (A), where the compartment and the at least one second biological compartment are analytically different. Separately from the step of identifying the pixel cluster in the digital image corresponding to (C) and (A) and (B), the pixel cluster corresponding to the sample staining is divided into high-intensity bins, low-intensity bins, and background intensity. The steps to generate a sample intensity map by segmenting into bins and (D) matching the candidate biological compartments within the composite staining region with the appropriate bins from the sample intensity map are analytical for each composite staining region. Specimen staining and staining of analytically different biological compartments are mixed by the step of identifying the part associated with (E) and the step of quantifying the sample staining in the analytically related part of the complex staining region. Quantify specimen staining of biological compartments in these regions (eg, (i) regions where diffuse membrane staining is mixed with cytoplasmic staining or (ii) regions where diffuse membrane staining is mixed with emphasis staining). Describe the method. Pixels in any discriminating compartment can then be quantified so that the compartment or area of staining intensity quantification can be determined. The '180 publication also describes scoring of membrane-specific expression levels.

[00079] In some embodiments, scoring is performed on the classified nuclei to obtain a percentage positive or H-score metric for a particular biomarker. By identifying the nucleus, the corresponding cell can be identified. In other embodiments, cells are scored by associating each of the associated nuclei with the surrounding staining membrane. Based on the presence or absence of a stained membrane around the nucleus, for example, unstained (no stained membrane found around the nucleus), partial staining (part of the cell nucleus is surrounded by a stained membrane), or complete staining (of the nucleus) The cells may be classified as (whole surrounded by a staining membrane).

[00080] In some embodiments, tumor nuclei are automatically identified by first identifying candidate nuclei and then automatically distinguishing between tumor nuclei and non-tumor nuclei. Many methods are known in the art to identify candidate nuclei in tissue images. For example, a method based on radial symmetry, as described herein, Ruiflok et al. For hematoxylin image channels, biomarker image channels, and the like obtained using the color deconvolution described in the above, as described herein, Paravin et al. Automatic candidate nucleus detection can be performed by applying a method based on the radial symmetry of. In one exemplary embodiment, a nuclear detection operation based on radial symmetry is used as described in Simultaneously Pending Patent Application WO2014 / 140855A1 assigned to the assignee of the present invention, all of which is hereby referenced Incorporated into the book. Other methods are described in U.S. Patent Application Publication No. 2017/01/40246, the disclosure of which is incorporated herein by reference.

[00081] Candidate nuclei are identified and then analyzed separately to distinguish tumor nuclei from other candidate nuclei. Other candidate nuclei may be further classified (eg, by identification of lymphocyte nuclei and stromal nuclei). In some embodiments, a trained supervised classifier is applied to identify tumor nuclei. For example, a trained supervised classifier is applied to identify tumor nuclei by training on nuclear features and then classify candidate nuclei as tumor nuclei or non-tumor nuclei on test images. Optionally, the trained supervised classifier may be trained separately to distinguish between different classes of non-tumor nuclei, such as lymphocyte nuclei and stromal nuclei. In some embodiments, the trained supervised classifier used to identify tumor nuclei is a random forest classifier. For example, training in a random forest classifier would (i) generate a training set of tumor and non-tumor nuclei, (ii) extract the characteristics of each nucleus, and (iii) extract the characteristics of the tumor nuclei and non-tumor nuclei. This can be done by training a random forest classifier to distinguish it from tumor nuclei. Then, by applying a trained random forest classifier, the nuclei may be classified into tumor nuclei and non-tumor nuclei in the test image. Optionally, the random forest classifier may be trained separately to distinguish between different classes of non-tumor nuclei, such as lymphocyte nuclei and stromal nuclei.

[00082] In some embodiments, the image received as input is processed to perform nuclear center (seed) detection and / or nuclear segmentation. For example, a command to detect a nuclear center may be given based on a radiosymmetric vote using a technique commonly known to those skilled in the art (Parvin, Bahram, et al., "Institute of Electrical and Electronics Engineer of". Structural saliency and symmetry of subcellular events (repeated voting for estimation of structural features and characterization of intracellular events) ”, Image Processing, IEEE Transitions on 16.3 (2007): 61 (2007). (Incorporated herein by reference)). In some embodiments, the nuclear center is detected by detection of the nucleus using radial symmetry, after which the nucleus is classified based on the intensity of stain around the cell center. For example, the size of the image may be calculated within the image, and by adding the sum of the sizes within the selected area, one or more votes at each pixel are accumulated. Mean shift clustering may be used to find local centers in the region that represent the actual location of the nucleus. Nuclear detection based on radial symmetry voting is performed on color image intensity data and explicitly uses speculative domain knowledge that the nucleus is an elliptical spot that varies in size and eccentricity. To achieve this, image gradient information, along with color intensity in the input image, is also used for radiosymmetric voting and, in combination with an adaptive segmentation process, provides accurate detection and localization of cell nuclei. As used herein, a "gradient" is a pixel intensity gradient calculated for a particular pixel, for example by considering the intensity value gradient of a set of pixels surrounding the particular pixel. Each gradient may have a particular "orientation" with respect to a coordinate system whose x-axis and y-axis are defined by the two orthogonal edges of the digital image. For example, nuclear seed detection involves defining seeds as points that are assumed to be inside the cell nucleus and serve as a starting point for locating the cell nucleus. In the first step, seed points associated with each cell nucleus are detected by detecting elliptical spots and structures similar to cell nuclei using a highly robust technique based on radial symmetry. The radial symmetry method operates on a gradient image by using a kernel-based voting procedure. A voting response matrix is generated by processing each pixel that has accumulated votes through the voting kernel. The kernel is based on the gradient direction calculated at the particular pixel, the expected range of maximum and minimum nucleus sizes, and the voting kernel angle (usually the range [π / 4, π / 8]). In the resulting voting space, the maximal locations with voting values above a predetermined threshold are stored as seed points. Unrelated seeds may be discarded in subsequent segmentation or classification processes.

[00083] The nucleus may be identified using other techniques known to those of skill in the art. For example, the size of an image may be calculated from one particular image channel of an H & E or IHC image, and each pixel around the specified size is around that pixel. Many votes may be assigned based on the total size within the area. Alternatively, the average shift clustering operation may be performed to find the local center in the voting image that represents the actual location of the nucleus. In other embodiments, the use of nuclear segmentation may be used to segment the entire nucleus based on the currently known center of the nucleus via morphological arithmetic and local thresholding. In yet other embodiments, the use of model-based segmentation may be used to detect nuclei (ie, by learning a nuclei shape model from a training dataset and using it as prior knowledge. Segment the nucleus in the test image).

[00084] In some embodiments, the nuclei are then segmented using individually calculated thresholds for each. For example, the Otsu method may be used for segmentation in the region around the identified nucleus because the pixel intensity of the nuclear region is expected to fluctuate. Of course to those skilled in the art, the Otsu method is used to determine the optimal threshold by minimizing intraclass variance, but is known to those skilled in the art. More specifically, the Otsu method is used to automatically perform clustering-based image thresholding, i.e. reduction of gray level images to binary images. The algorithm assumes that the image contains two classes of pixels (foreground and background pixels) that follow a bimodal histogram. Then, by calculating the optimal threshold for separating the two classes so that the combined dispersion (intraclass variance) of the two classes is minimal or equivalent (because the sum of the pairwise squared distances is constant), each intraclass variance Is the maximum.

[00085] In some embodiments, these systems and methods include automatically analyzing spectral and / or morphological features of the discriminating nuclei in images that identify the nuclei of non-tumor cells. For example, in the first step, the spots may be identified in the first digital image. As used herein, a "blob" can be, for example, a region of a digital image in which some properties (eg, intensity or gray values) are constant or fluctuate within a predetermined range of values. is there. In a sense, all the pixels in the spot are considered to be similar to each other. For example, finite difference methods based on derivatives of functions of position on a digital image and extremum-based methods may be used to identify spots. Nuclear spots are spots whose pixels and / or outer shape indicate that they were probably produced by nuclei stained with a first stain. For example, the evaluation of the radial symmetry of the spots can be used to determine whether the spots should be identified as nuclear spots or as any other structure (eg, stained artifacts). For example, if the spots have a very long shape and are not radial symmetric, the spots may be identified as stained artifacts rather than as nuclear spots. According to this embodiment, the spots identified as "nuclear spots" may represent a set of pixels that are identified as candidate nuclei and can be further analyzed to determine if the nuclear spots represent a nucleus. In some embodiments, any type of core spot is used directly as the "identifying core". In some embodiments, filtering operations are applied to the identified nuclei or nuclear spots to identify nuclei that do not belong to biomarker-positive tumor cells and to remove the identified non-tumor nuclei from the identified nuclei list. Or, do not add the nucleus to the list of identified nuclei from the beginning. For example, analysis of the additional spectral and / or shape features of the discriminating nuclear spots may be used to determine if the nucleus or nuclear spots are the nucleus of a tumor cell. For example, the core of lymphocytes is larger than the core of other tissue cells (eg, lung cells). When tumor cells are derived from lung tissue, lymphocyte nuclei are identified by identifying all nuclear spots of minimum size or diameter that are much larger than the average size or diameter of normal lung cell nuclei. To. Discriminating nuclear spots on the lymphocyte nuclei may be removed (ie, "filtering depletion") from the identified set of nuclei. Filtering and removal of nuclei of non-tumor cells can improve the accuracy of this method. According to the biomarker, non-tumor cells can also represent the biomarker to some extent, so that an intensity signal can be generated in a first digital image not derived from the tumor cells. By identifying nuclei that do not belong to tumor cells and filtering out the entire identified nuclei, the accuracy of identifying biomarker-positive tumor cells can be improved. The above and other methods are described in US Patent Application Publication No. 2017/0103521 and all disclosures thereof are incorporated herein by reference. In some embodiments, once seeds are detected, local adaptive thresholding methods may be used to generate spots around the detection center. In some embodiments, other methods may be embodied, for example, by using a marker-based watershed algorithm, it is possible to identify nuclear spots around the detection nucleus center. .. The above and other methods are described in the co-pending application PCT / EP2016 / 051906 (published as WO2016 / 120442), the entire disclosure of which is incorporated herein by reference.

[00086] The system can use at least one image trait metric and at least one morphological metric to determine if a feature in an image corresponds to a structure of interest ("feature metric"). Collectively). Examples of the image characteristic measurement standard (derived from the features in the image) include color, color balance, and intensity. Morphological metrics (derived from features in an image) include, for example, feature size, feature color, feature orientation, feature shape, relationship or distance between features (eg, adjacent features), and features for another anatomy. Relationships or distances can be mentioned. As described herein, image quality metrics, morphological metrics, and other metrics can be used for classifier training. Specific examples of the measurement criteria derived from the image features are shown below.

[00087] (A) Measurement criteria derived from morphological features
[00088] As used herein, a "morphological feature" is, for example, a feature that indicates the shape or size of a nucleus. Without wishing to be bound by any particular theory, morphological features are believed to provide some essential information about the size and shape of a cell or its nucleus. For example, morphological features can be calculated by applying various image analysis algorithms to the pixels contained in the nuclear spots or seeds or the pixels surrounding the nuclear spots or seeds. In some embodiments, morphological features include area, minor and major axis lengths, perimeters, radii, solidity and the like.

[00089] (B) Measurement criteria derived from appearance features
[00090] In the present specification, the "appearance feature" refers to, for example, by comparing the pixel intensity values of the pixels contained in the nuclear spots or seeds used for identifying the nucleus or the pixels surrounding the nuclear spots or seeds. Derived from image channels (eg, background channels, channels for staining biomarkers, etc.) that are calculated features for a particular nucleus and have different pixel intensities to be compared. In some embodiments, the metrics derived from the appearance features are percentiles of pixel intensity and gradient magnitude calculated from different image channels (eg, 10th, 50th, and 95th hundreds). Calculated from the percentile). For example, first of all, the X percentile value (X = 10, 50,) of the pixel value of each of the plurality of ICs of the image channels (for example, three channels of HTX, DAB, and luminance) in the nuclear spot representing the nucleus of interest. The number P of 95) is identified. It is considered convenient to calculate the appearance feature metric because the derived metric can represent the characteristics of the nuclear region as well as the membrane region around the nucleus.

[00091] (C) Measurement criteria derived from background features
[00092] “Background features” are, for example, features that indicate appearance and / or the presence or absence of stain in the cytoplasm and cell membrane features of cells containing nuclei extracted from the image. For example, by identifying a seed that represents a nuclear spot or nucleus and analyzing a pixel area that is directly adjacent to the identified cell assembly (eg, a 20 pixel (approximately 9 micron) thick ribbon around the nuclear spot boundary). Background features and corresponding metrics of the nucleus and corresponding cells shown in digital images by capturing the cytoplasm and in-membrane appearance and stain presence of the cell, integrally with the area directly adjacent to the cell bearing the nucleus. Can be calculated. These metrics are similar to nuclear appearance features, but are calculated on a ribbon approximately 20 pixels (approximately 9 microns) thick around each nuclear boundary, so they are integrated with the area directly adjacent to the cell with the discriminant nucleus. The appearance and presence or absence of stain in the cytoplasm and membrane of the cell are incorporated into the cell. Without wishing to be bound by any particular theory, the ribbon size is chosen so that the ribbon captures a sufficient amount of background tissue area around the nucleus that can be used to provide useful information for identifying the nucleus. This is because it is considered. These features are characterized by J. Kong, et al. "A comprehensive framework for framework of glioma in digital microscopic imaging: An application to glioma: application to gliomas: application to gliomas: application to gliomas: application to gliomas, application to gliomas, gliomas, gliomas, gliomas, gliomas, gliomas, gliomas, gliomas, gliomas, gliomas, gliomas, gliomas. Similar to the characteristics of the disclosures in 2128-2131, all disclosures are incorporated herein by reference. These features may be useful in determining whether the surrounding tissue is stromal or epithelial (such as in H & E stained tissue samples). Also, without wishing to be bound by any particular theory, these background features are believed to incorporate membrane staining patterns that are useful when tissue samples are stained with a suitable membrane stain.

[00093] (D) Metrics derived from color
[00094] In some embodiments, the metric derived from color includes a color ratio R / (R + G + B) or a color principal component. In other embodiments, color-derived metrics include local statistics (mean / median / variance / standard deviation) and / or color intensity correlation in the local image window for each color.

[00095] (E) Metrics derived from strength characteristics
[00096] Adjacent cell populations with certain characteristic values are set between the black and white tones of the gray colored cells represented in the histopathological slide image. The intensity of these colored cells determines the cells affected by the cluster of dark cells surrounding them, as the correlation of color features defines size class instances. Examples of texture features are described in PCT Publication WO 2016/075095, the entire disclosure of which is incorporated herein by reference.

[00097] (F) Spatial features
[00098] In some embodiments, spatial features include local density of cells, the average distance between two adjacent detection cells, and / or the distance from the cell to the segmented region.

[00099] (G) Metrics derived from nuclear features
[000100] Of course to those skilled in the art, metrics can also be derived from nuclear features. Calculations of such nuclear features are performed by Xing et al. "Robust Nucleus / Cell Detection and Segmentation in Digital Pathology and Microscopy Images: A Comprehensive Review: A Comprehensive Review (Digital Pathology / Microscopic Imaging) Robust Nucleus / Cell Detection and Segmentation in Digital Pathology / Microscopic Imaging" 263, January 2016, the entire disclosure of which is incorporated herein by reference. Of course, other features known to those of skill in the art may also be considered and used as the basis for the calculation of features.

[000101] After deriving the feature metrics, either by feature alone or in combination with training data (eg, during training, combined with grand truth identification information provided by an observation expert according to procedures known to those of skill in the art). An exemplary cell is presented), the nucleus or cell may be classified. In some embodiments, the system may comprise a classifier trained at least partially based on a per-biomarker training set or reference slide set. As a matter of course to those skilled in the art, various slide sets can be used for training the classifier for each biomarker. Therefore, for a single biomarker, a single classifier is obtained after training. Also, of course, those skilled in the art will have variations between image data obtained from different biomarkers, so training different classifiers for different biomarkers will result in better performance against unknown test data. Can be guaranteed, and the biomarker species of the test data will be known. The trainable classifier can be selected for slide interpretation, at least in part, based on the best treatment of variability in training data, for example in tissue type, staining protocol, and other features of interest.

[000102] In some embodiments, the classification module is a support vector machine (“SVM”). In general, SVM is a classification technique based on statistical learning theory in which the non-linear case kernel transforms a non-linear input dataset into a high-dimensional linear feature space. Without wishing to be bound by any particular theory, support vector machines are thought to project training datasets E representing two different classes into higher dimensional space by means of kernel function K. In this transformed data space, the nonlinear data is transformed so that flat lines (discriminating hyperplanes) that separate the classes can be generated so as to maximize class separation. The K then projects the test data into a higher dimensional space and classifies it based on its position relative to the hyperplane. The kernel function K defines how data is projected into higher dimensional space.

[000103] In other embodiments, classification is performed using the AdaBoost algorithm. AdaBoost is an adaptive algorithm that combines many weak classifiers to produce a strong classifier. Image pixels identified by a pathologist during the training phase (eg, specific) to generate a probability density function for each individual texture feature Φj (j ∈ {1, ..., K}) that is considered a weak classifier. Pixels with stains or pixels belonging to a particular tissue species) are used. Then, by using Bayes' theorem, the likelihood scene Lj = (Cj, 1j ∈ {1, ..., K}) constituting the weak learner is generated for each Φj. These are combined by the AdaBoost algorithm to give a strong classifier Πj = ΣTi = 1αjilji, but for all pixels cj ∈ Cj, Πj (cj) is the coupling likelihood that pixel cj belongs to class ωT, and αji is , A weight determined during training of feature Φi, where T is the number of iterations.

[000104] In some embodiments, different marker expression scores such as percentage positiveness or H-score, depending on the derived stain intensity value, the number of specific nuclei, or the use of other classification results (as used herein). May be used to determine the term "expression score" (i.e., the classification feature may allow the expression score to be calculated). The scoring method is as follows: Simultaneous pending patent application WO2014 / 102130A1 "Image analysis for breast cancer prediction (image analysis for breast cancer prediction)" filed on December 19, 2013 and 2014 3 WO2014 / 14085A1 filed on 12th May 2014, "Tisse invention-based machine learning system for automatic scoring of digital color slides" (detailed description in machine learning system for automatic scoring of digital hall slides) All the contents of each are incorporated herein by reference. For example, a score (eg, whole slide score) can be determined based at least in part on the number of biomarker-positive tumor cells / biomarker-positive non-tumor cells. In some embodiments, the average spot intensity, color, and geometric features (such as the area and shape of the detected nuclear spots) may be calculated for each detected nuclear spot, and the nucleus. The spots are classified into tumor nuclei and non-tumor cell nuclei. The number of discriminative nuclear outputs corresponds to the total number of biomarker-positive tumor cells detected in the FOV, as indicated by the count of tumor nuclei.

[000105] In some embodiments, characteristic metrics are also derived for FAP staining and the percentage of FAP positive or negative cells (eg, positive or negative stained stromal cells) (eg, for example). The classifier is trained so that the percentage positive expression score) can be revealed. In some embodiments, less than 10% of the tumor cells are assigned a score of 0, more than 11% to less than 25% of the tumor cells are assigned a score of 1, and more than 26% to 50 of the tumor cells. 2 may be assigned to an area of% or less, and 3 may be assigned to an area of more than 51% of tumor cells. For staining intensity, zero / weak staining (negative control) is assigned a score of 0, weak staining that is clearly stronger than the negative control level is assigned 1, 2 is assigned to medium intensity staining, and strong staining is assigned. 3 may be assigned to. In some embodiments, a final score of 3 or higher may be recognized as indicating positive expression of FAP.

[000106] Segmentation module
[000107] The medium resolution analytical technique employs a segmentation algorithm that produces subregions within the input image that are defined to capture the biologically meaningful region of interest. After deriving the measurement reference from the input image by the image analysis module 205 (step 310), the input image is segmented into a plurality of subregions by using the segmentation generation module 206 (step 320).

[000108] In some embodiments, segmentation is performed on a single channel image (eg, the "purple" channel in the separated FAP image). Separation methods are known to those skilled in the art (eg, Zimmermann "Spectral Imaging and Linear Imaging in Light Microscopy", Adv Biochem Engin / Biotechn / Biotechn / Biotechn. C. L. Lawson and R.J. Hanson, "Solving first microscopes (solving the least squares problem)", Prentice Hall, 1974, Chapter 23, p.161 describes linear separations, all of which are disclosed. The contents are incorporated herein by reference). In the present specification, other separation methods are disclosed (Ruifok et. Al. "Quantification of histochemical staining by color deconvolution (quantification of histochemical staining by color deconvolution)", Anal Quant Cylt 200 23 (4): 291-9 (see also, all disclosures are incorporated herein by reference).

[000109] In some embodiments, the generated subregions are within a given size or within a specified range within the image processing algorithm (eg, parameters of the SLIC superpixel generation algorithm, as described herein). Captures information about the area of the input image that has a size.

[000110] In some embodiments, the input image is segmented into subregions with a given shape, size, area, and / or spacing. For example, the sub-region (710) may be oval, circular, square, rectangular or the like, as shown in FIG. In some embodiments, the oval, circular, square, or rectangular subregion may have a size in the range of 50 pixels to approximately 100 pixels and has similar properties or properties (eg, color,). It may have some other size so that a group of pixels with brightness and / or texture) is selected. In some embodiments, the subregions may not overlap and may be generated via a sampling grid. As used herein, the term "sampling grid" is a network of horizontal and vertical lines that are superimposed on an image at uniform intervals and ultimately used to locate points that do not overlap points in the image. Regarding. In some embodiments, any number of adjacent positions established by horizontal and vertical lines may be used to define the image segment. In some embodiments, the subregions are distributed throughout the image to capture representative samples of relevant regions for analysis (eg, areas where irregularly shaped cells are the dominant feature).

[000111] In another embodiment, the input image is segmented by applying a set of algorithms to the image, such as a global thresholding filter, a locally adaptive thresholding filter, a morphological calculation, and a watershed transformation. These filters may operate sequentially, or may operate in any order that one of ordinary skill in the art deems necessary. Of course, any filter may be applied iteratively until the desired result is achieved. In some embodiments, the application of the first filter to the input image may have nuclei, such as removal of white image regions (corresponding to regions in unstained or substantially unstained tissue samples). Remove areas with low. In some embodiments, this is achieved by applying a global thresholding filter. In some embodiments, global thresholding is based on the median and / or standard deviation calculated on the first principal component channel (eg, similar to a grayscale channel). By determining the global threshold, it is believed that any white image region representing an unstained or nearly unstained region that is likely to have no nuclei can be discarded. The image is then filtered to selectively remove and / or fill holes with artifacts (eg, small spots, small cuts, other small objects). In some embodiments, morphological operators are applied to remove artifacts and / or fill holes. In some embodiments, a distance-based watershed is applied based on a binary image introduced as input (eg, a binary image as a result of a preceding filtering step).

[000112] In some embodiments, the input image is segmented into superpixels. The superpixel algorithm is thought to divide an image into many segments (pixel groups) that represent perceptually meaningful entities. Each superpixel is obtained by a low level grouping process and has a perceptually consistent unit. That is, all the pixels in the living body contained in the superpixel are stained (for example, the pixel existing in the superpixel is of a certain type of stain) and the staining intensity (for example, the pixel is a constant relative intensity value or value). (Has a range of), and texture (eg, pixels have a specific spatial arrangement with respect to color or intensity) is as uniform as possible. The local analysis results for each superpixel can be stored and reported to represent the analysis results on a digital pathological image.

[000113] A superpixel is a collection of pixels with similar characteristics such as color, brightness, and texture. The image contains multiple combinatorial properties of pixels and can be composed of a fixed number of superpixels that can store the edge information of the original image. Compared to a single pixel, superpixels contain a wealth of property information, which can significantly reduce the complexity of image post-processing and significantly increase the speed of image segmentation. Superpixels are also useful for estimating and determining probabilities with small neighborhood models.

[000114] The superpixel algorithm is a method of grouping pixels into meaningful atomic regions of similar size. Without wishing to be bound by any particular theory, superpixels are often located at important boundaries in the image and tend to take on non-standard or unique shapes if they contain prominent object features. Therefore, it is considered to be effective. Consistent with the desire to obtain and store information in medium-resolution analysis, superpixels are positioned between the pixel level and the object level, and are a group of perceptually meaningful pixels without a comprehensive representation of image objects. Brings more information than a pixel by representing. Superpixels can be understood as a form of image segmentation that oversegments an image in a short computational time. The outer shape of superpixels has been found to well follow natural image boundaries, as most structures in the image are protected. Subsequent processing tasks are less complex and less time consuming because image features are calculated for each superpixel rather than each pixel. For this reason, superpixels are considered useful as a pre-processing step for object-level analysis such as image segmentation.

[000115] Without wishing to be bound by any particular theory, superpixels are thought to oversegment an image, for example by forming dense and uniform pixels with similar qualities of color or shape. .. In the past, multiple superpixel methods have been developed. These can be classified into (i) graph-based methods and (ii) gradient rise-based methods. In a graph-based approach, each pixel is considered a node in the graph. Edge weights proportional to their similarity are defined between all node pairs. Then, the superpixel segment is extracted by formulating and minimizing the cost function specified on the graph. In the ascending gradient approach, iterative mapping of pixels to feature space draws a dense region representing clusters. Each iteration obtains better segmentation until convergence by refining each cluster.

[000116] Many superpixel algorithms have been developed, including normalized cuts, aggregated clustering, quickshifts, and turbopixel algorithms. The normalized cut algorithm recursively splits the graph for all pixels by using contour and texture cues and minimizing the cost function defined on the edges of the split boundary. This results in very regular and visually pleasing superpixels (Jianbo Sh and Jitendra Malik "Normalized cuts and image segmentation", IEEE Transition Engineering Technology). (PAMI), 22 (8): 888-905, Aug 2000 (see, all disclosures are incorporated herein by reference). Alastair Moore, Simon Prince, Johann Warrell, Umar Mohammed, and Graham Jones "Superpixel Lattices", IEEE Computer Vision 2 Division from IEEE Computer Vision and Pattern Describes how to generate superpixels that follow a grid by finding the optimal path or seam. The optimal path can be found by using the graph cut method. (Shai Avidan and Ariel Shamir "Seam carving for content-aware image resizing", ACM Transitions on Graphics (see SIGGRAPH), ACM Transitions on Graphics (SIGGRAPH) Incorporated herein by)). Quick Shift (A. Vedaldi and S. Soutto "Quick shift and kernel methods for mode seeding", European Conference on Computer Vision (details in EC2) (Incorporated in) uses the mode search segmentation method. It initializes the segmentation using the mediad shift procedure. It then moves each point in the feature space to the closest adjacent point that increases the Parzen density estimate. The turbopixel method uses a level set-based geometric flow to gradually expand a set of seed positions (A. Levinshtein, A. Stereo, K. Kuturakos, D. Fleet, S. Dickinson, and K. .Siddiqi "Turbopixels: Fast superpixels using geometric flows", IEEE Transitions on Pattern Analysis in IEEE (Notes) See (embedded)). Geometric flow relies on a local image gradient for the purpose of regularly distributing superpixels on the image plane. Turbopixel Superpixels, unlike other methods, are constrained to have uniform size, denseness, and boundary tracking. For yet another method of generating superpixels, see Radhaklishna Achanta "SLIC Superpixels Compared to State-of-the-art", Journal of Latex Class Files. 6, No. 1, December 2011 (all disclosures thereof are incorporated herein by reference).

[000117] A superpixel algorithm called simple linear iterative clustering (SLIC) has been introduced, which is superior in both boundary tracking and efficiency compared to the state-of-the-art superpixel method. SLIC has two steps. First, superpixels are generated by grouping pixels by the local k-means clustering (KMC) method, the distance being measured as an Euclidean distance integrated with data and spatial distance. Second, it is removed by integrating the generated micro-separation regions as the closest large superpixels by using the Connected Component Algorithm (CCA).

[000118] k-means clustering aims to divide n observations into k clusters, each observation belonging to the cluster closest to the mean and acting as a prototype of the cluster. Connected component labeling works by scanning the image pixel by pixel (top to bottom, left to right) to identify connected pixel regions, that is, regions of adjacent pixels that share the same set of intensity values (V). In the case of a binary image, V = {1}, but in a gray level image, V takes a range of values (for example, V = {51, 52, 53, ..., 77, 78, 79). , 80}))). Connected component labeling works for binary or gray level images, allowing different measures of connectivity. However, in the following, a binary input image and eight degree of connection are assumed. The connected component labeling operator is by moving along a line to the point p where V = {1}, where p indicates the pixel to be labeled at any stage of the scanning process. Scan the image. If this is true, the four adjacent pixels of p that have passed during the scan (ie, (i) to the left of p, (ii) above, and (iii and iv) two upper diagonal adjacent pixels). Inspect. Based on this information, p is labeled as follows. If all four adjacent pixels are 0, a new label is assigned to p, and if only one adjacent pixel is V = {1}, that label is assigned to p, and two or more adjacent pixels V = { In the case of 1}, assign one of the labels to p and write down the equivalent.

[000119] After the scan is complete, the equivalent label pair is stored in the equivalent class and a unique label is assigned to each class. As a final step, a second scan is made on the image, during which each label is replaced by the label assigned to its equivalent class. For display purposes, each label may have a different gray level or color.

[000120] SLIC is an adaptation of k-means for the generation of superpixels, but there are two important differences. (I) By limiting the search space to a region proportional to the size of the superpixels, the number of distance calculations in the optimization is dramatically reduced (thus, regardless of the number of superpixels k, the number of pixels. Complexity is expected to decrease linearly). (Ii) The weighted distance scale combines color and spatial proximity while controlling the size and density of superpixels (Achanta, et al. "SLIC Superpixels Compared to State-of-the-Art Superpixel Methods". SLIC Super Pixel by Comparison with Advanced Super Pixel Method) ”, IEEE Transitions on Pattern Accessibility and Machine Intelligence, Vol. 34, No. l1, November 2012 (all disclosures are incorporated herein by reference). ).

[000121] SLIC considers the L * a * b values in the CIELAB color space and the image pixels in the 5D space defined by their respective x and y coordinates. Pixels in 5D space are clustered based on adaptive k-means clustering that integrates color similarity and proximity in the image plane. This clustering is based on a distance scale D that measures color similarity (dc) in L * a * b space and pixel proximity (ds) in x, y space. The latter is normalized by a grid spacing (S) that defines the square root of the total number of image pixels divided by the number of superpixels (k). The density and regularity of superpixels is controlled by the constant m. This parameter serves as a weighting criterion between the spatial distance (dc) and the spectral distance (ds). The larger m is, the heavier the weight of spatial proximity is, and the denser superpixels can be obtained with less boundary tracking to the spectral outline in the image.

[000123] The SLIC algorithm can be applied as follows. Let N _{p be the} number of pixels in a given image (or part or region of interest), and let k be the number of superpixels to generate. Next, the main steps of the SLIC algorithm are as follows.

[000124] (1) Initialize the cluster center. K cluster centers on a regular grid

After being separated by pixels, the centers of these clusters are moved to the position where the gradient is minimal near 3x3. Without wishing to be bound by any particular theory, this is thought to avoid setting the center of the superpixel to the edge and reducing the chances of setting the seed of the superpixel to a noisy pixel.

[000125] (2) Allocate pixels. The local KMC specifies each pixel at the center of the closest cluster in the local search space.
[000126] (3) Update the cluster center. Set each cluster center as the average of all pixels in the corresponding cluster.

[000127] (4) Steps (2) and (3) are repeated until the cluster remains unchanged or another given criterion is met.
[000128] (5) Post-processing. If the size of the separation area is less than the minimum size S _min , the separation area is reassigned to nearby superpixels by using CCA.

[000129] In step (2) of the SLIC method, local KMC is applied, in which case each pixel is associated with the nearest cluster center whose search area covers the location. In the conventional KMC, since the search area at the center of each cluster is the entire image, the distance from the center of each cluster in the image to all the pixels is calculated. However, in the local KMC, the search space at the center of the cluster is limited to the local 2S × 2S square area. Therefore, SLIC only calculates the distance from the center of each cluster to the pixels in its search area.

[000130] In local KMC, Euclidean distance is used in clustering. The z _i is the spatial position and _{(x i, y} i) i th cluster center data. Let z _{j be} the intensity of the pixels in the central search area. And the integration distance between this pixel and the center is as follows.

[000132] Here, d _f = | z _{i −} z _j | and

Each is the distance between the strength and the pixel and the center, m is the regularization parameter which weights the relative contributions of d _f and d _s for the integrated distance D _I. A larger m indicates that d _s is more significant than d _f . These two integrated distance D _I of directly describing equivalent contribution of distance can be given below.

[000134] Here, N _f is the average intensity of the entire screen, and w ∈ [0, 1] is a regularization parameter. In this connection, w and (1-w), respectively, the ratio of the respective spatial distance in the normalized intensity and D _I.

[000135] In some embodiments, the parameter k of the SLIC algorithm specifies the number of superpixels of approximately equal size. In some embodiments, the density parameter m can be set to control the trade-off between superpixel homogeneity and boundary tracking. By changing the density parameter, without wanting to be bound by any particular theory, regular shaped superpixels are generated in the non-textured area and highly irregular superpixels are generated in the textured area. It is thought to get. Again, without wishing to be bound by any particular theory, the parameter m would be considered to allow weighting of relative importance between color similarity and spatial proximity. When m is large, spatial proximity is more important and the resulting superpixels are denser (ie, the area-to-perimeter ratio is smaller). If m is small, the resulting superpixels will follow the image boundaries more closely, but will be irregular in size and shape.

[000136] In some embodiments, both superpixel size and density parameters are adjusted. In some embodiments, superpixel sizes ranging from about 40 pixels to about 400 pixels are used. In other embodiments, superpixel sizes ranging from about 60 pixels to about 300 pixels are used. In yet another embodiment, superpixel sizes in the range of about 70 pixels to about 250 pixels are used. In yet another embodiment, superpixel sizes ranging from about 80 pixels to about 200 pixels are used.

[000137] In some embodiments, the density parameter ranges from approximately 10 to approximately 100. In other embodiments, the density parameter ranges from approximately 20 to approximately 90. In other embodiments, the density parameter ranges from about 40 to about 80. In other embodiments, the density parameter ranges from about 50 to about 80.

[000138] FIG. 8A shows an example of superpixels generated using SLIC, as described herein, superpixels to suit the localized attributes of the region of interest, without superposition and gaps. Is segmented. In addition, each superpixel subregion has a particular final shape, depending on the local intensity (810) and direction (820) of the presence of biomarker expression. For this reason, superpixels are perceptually meaningful for such biological structures of interest. 8B, 8C, and 8D show the original IHC image at high magnification, the initialization of the superpixel generation process, and the locally homogeneous final superpixels, respectively, and the regularity of their shapes is As mentioned above, it has been adjusted by the technical parameters of the SLIC algorithm.

[000139] Representation object generation module
[000140] After the subregion is generated by the subregion generation module (step 320), the expression object or the point of interest is determined for each subregion using the module 207 (step 330). In some embodiments, the representation object is a subregion or superpixel outline for a cell of interest or a group of cells of interest (eg, fibroblasts or macrophages). In other embodiments, the representation object is a seed point. As described herein, an object of the present disclosure is to characterize cells of interest (irregularly shaped cells) based on subregions with similar staining presence, staining intensity, and / or local texture, and these. It is to automatically save the subregion of the homogeneity property of. The representation object or its coordinates is one way to store the generated subregion. 9A and 9B provide examples of superpixel polygon outlines and central seeds containing living organisms of interest.

[000141] In some embodiments, algorithms are utilized that separate subregions of different colors or textures and form boundaries that match the dominant edges in the image, and thus the organism of interest (eg, fibers). Boundaries representing cells of irregular size or shape, such as blasts or macrophages, are generated. In some embodiments, a thresholding algorithm (eg, Otsu method, average clustering, etc.) is used to exclude sub-regions that do not have stains and provide only sub-regions that contain a threshold amount of stain as representation objects. ) May be applied to the stain channel image. In some embodiments, threshold parameters (eg, threshold staining parameters provided by a pathologist) may be used to generate a binary mask of the subregion. In some embodiments, a series designed to enhance the image so that (i) a subregion that is unlikely to represent an object of interest is separated from (ii) a subregion that represents a cell that has the object of interest. Segmentation is achieved by applying the filter of. Additional filters may be selectively applied to remove artifacts, remove small spots, remove small cuts, fill holes, and divide large spots.

[000142] In some embodiments, irregularly shaped cells, such as by removal of a white image region (corresponding to a region in an unstained or substantially unstained tissue sample) in a binary image of a stain channel. Regions that are unlikely to have sub-regions that identify the are removed. In some embodiments, this is achieved by applying a global thresholding filter. Thresholding turns an intensity image (I) into a binary image (I') by assigning a value of 1 or 0 to all pixels when the intensity is above or below some threshold (here, the global threshold). The method used to convert. In other words, global thresholding is applied to divide the pixels according to the intensity value. In some embodiments, global thresholding is based on the median and / or standard deviation calculated on the first principal component channel (eg, similar to a grayscale channel). By determining the global threshold, it is believed that any white image region representing an unstained or substantially unstained region that is likely to be free of irregularly shaped cells can be discarded.

[000143] In some embodiments, with respect to FAP stains, 1) the purple channels are separated, 2) the FAP positive regions are identified by thresholding the purple channels, and 3) superpixel segmentation for the purple channels Boundaries can be formed by applying and 4) applying feature metrics to superpixel objects. In some embodiments, the presence or absence of a FAP-positive region may be identified using supervised generation rules trained on the basis of grand truth obtained from a pathologist. In some embodiments, FAP positive threshold parameters may be provided by the pathologist, such as by identifying thresholds on a training set of images. Then, a binary mask may be generated by using this threshold parameter. These methods are described in Auranuch Losakul et al. "Automate-slide-slide-analysis of multiplex-brightfield IHC images for cancer cells and carcinoma-associated fibroblasts" (cancer cells) It is described in detail in SPIE 10140, Medical Imaging 2017: Digital Pathology, 1014007 (2017/03/01), the entire disclosure of which is incorporated herein by reference.

[000144] In some embodiments, subregion boundaries are traced. For example, an algorithm may be provided that traces the outer boundaries of the sub-region, as well as the outer boundaries of the "holes" inside or between the sub-regions. In some embodiments, the boundaries of the subregions are generated by forming boundary traces using a matlab function called bwboundaries (html: //www.mathworks.com/help/images/ref/bwboundaries. html).

[000145] After boundary formation, the boundary trace is transformed into a polygon outline consisting of x, y coordinates. The x, y coordinates of the trace boundary may be stored in memory or database, for example, the row and column coordinates of all pixels of the trace boundary of the subregion object are determined and stored. You may.

[000146] In some embodiments, the calculation or calculation of the center of gravity or mass center of each subregion derives seed points. Those skilled in the art will know how to determine the centroid of an irregular object. After the calculation, the center of gravity of the sub-region is labeled and / or the x and y coordinates of the seed are stored in the memory or database. In some embodiments, the position of the center of gravity or locating the center of mass may be superimposed on the input image.

[000147] Labeling module
[000148] After the sub-region is generated by using the segmentation module 206 and the representation object is calculated by using the module 207, the measurement reference and the like derived from the image analysis module 202 by using the labeling module 208 ( Step 310), the representation object is annotated, labeled, or associated with the data (step 330). The labeling module 208 may generate database 209, which is a non-temporary memory for storing data, as described herein. In some embodiments, the database 209 stores the image received as input, the coordinates of any polygon and / or seed point, and any relevant data or label from image analysis (see FIG. 11).

[000149] In this regard, a data vector may be stored for each segmentation subregion of the image. For example, a data vector may be stored for each sub-region, including any representation object and associated image analysis data. As an example, the data points "a", "b", and "c" are the coordinates of the representation object, and "x", "y", and "z" are the metrics (or specific) derived by image analysis. The database is [a, b, c, x, y, z] ¹ , [a, b, c, x, y, z] ² , [a] if it is the average of the metrics corresponding to the subregions of , B, c, x, y, z] ^{N and} other data vectors will be stored. Here, N is the number of subregions generated by the segmentation module 206.

[000150] In some embodiments, the data from the image analysis module represents individual pixels in the image. As a matter of course to those skilled in the art, the data of all pixels in a particular sub-region can be averaged to give the average value of the pixel data in the sub-region. For example, each individual pixel may have a certain intensity. The intensities of all pixels in a particular sub-region can be averaged to give the average pixel strength of that sub-region. The average pixel of the sub-region may be associated with the representation object of the sub-region, or the data may be stored as a unit in the memory.

[000151] For staining with FAP, the FAP positive area can be given another feature / measurement result to the superpixel object. The FAP positive area refers to the total number of pixels having FAP intensity above the set threshold. For selection of thresholds, see Auranuch Lorsakal et al. "Automate-slide-slide-analysis of multiplex-brightfield IHC images for cancer cells and carcinoma-associated fibroblasts" (cancer cells) SPIE 10140, Medical Imaging 2017: Digital Pathology, 1014007 (2017/03/01), the entire disclosure of which is incorporated herein by reference.

[000152] As an example of the data stored by the labeling module, and for staining biological samples with FAP biomarkers, the average intensity of FAP stains within the subregion is derived by image analysis of the particular subregion. The FAP stain intensity may be stored in the database together with the coordinates of any representation object in the sub-region. Similarly, by image analysis, a specific expression score of a sub-region such as a FAP expression score may be derived, and the FAP expression score of the sub-region is an expression of the specific sub-region. It may be stored together with the object. In addition to the average intensity score and average expression score of the image portion in any subregion, other parameters may be stored, such as the distance between seed points, discriminating tumor cells and irregularly shaped cells. Distances between (eg, distances between tumor cells and fibroblasts), and FAP-positive areas are, but are not limited to.

[000153] In some embodiments, as an example, analytical results calculated within the corresponding superpixels (eg, average local intensity, positive staining area) are given to each corresponding polygon outline and seed. In the case of all slide images, these representation objects (for example, polygon outlines and seeds) to which the respective analysis results are given are stored in the database in xy coordinates. FIG. 10A is a head and neck cancer stained purple with fibroblast-activating protein (FAP) for fibroblasts (1010) and yellow with pancytokeratin (PanCK) for epithelial tumors (1020). An example of all slide IHC images of tissue is shown. 10B and 11 show examples of polygon outlines and seeds with analysis results of superpixels belonging to the fibroblast region that can be stored in the database, respectively.

[000154] Data Read or Projection Module
[000155] As a matter of course to those skilled in the art, the stored analytical results and associated biological features can be retrieved later and the data are reported in various formats (eg, histogram plots of the analytical results). Alternatively, it may be visualized. More specifically, the coordinate data of the representation object and the related image analysis data may be read from the database 209 and used separately for analysis. In some embodiments, and as an example, the representation object is readable from the database for visualization or reporting of analysis results within the entire slide image or in the user annotation area. As shown in FIG. 12, the associated or assigned image analysis results can be reported by plots in the histogram of FAP intensity read from all slide superpixels. Alternatively, the data can be visualized in a full slide image, a field image, or a portion of an image separately annotated by a healthcare professional for scrutiny.

[000156] Other components that implement the embodiments of the present disclosure.
[000157] The system 200 of the present disclosure may be connected to a sample processing device capable of performing one or more preparation processes on a tissue sample. Preparation processes include deparaffinization of specimens, preparation of specimens (eg, cell regulation), staining of specimens, performing antigen readout, immunohistochemical staining (including labeling) or other reactions, and / or Specimens for in-situ hybridization (eg, SISH, FISH, etc.) staining (including labeling) or performing other reactions, as well as microscopy, microanalysis, mass spectroscopy, or other analytical methods. Other processes to create include, but are not limited to.

[000158] The treatment device can apply the fixative to the specimen. Fixatives include cross-linking agents (aldehydes (eg, formaldehyde, paraformaldehyde, and glutaaldehyde) as well as non-aldehyde cross-linking agents, etc.), oxidizing agents (eg, metal ions and complexes such as osmium tetroxide and chromium acid) Protein denaturants (eg, acetic acid, methanol, and ethanol), fixatives of unknown mechanism (eg, mercuric chloride, acetone, and picric acid), combination reagents (eg, Carnois fixatives, metacan, Buan solution, B5). Fixants, Rothman's solution, and Jundle's solution), microwaves, and mixed fixatives (eg, exclusion volume fixatives and steam fixatives).

[000159] If the sample is a sample embedded in paraffin, this sample can be deparaffinized by using a suitable deparaffinizing solution. After the paraffin is removed, any number of substances can be applied to the specimen in succession. These substances include pretreatment (eg, protein cross-linking reversal, nucleic acid exposure, etc.), denaturation, hybridization, washing (eg, stringency washing), detection (eg, visual or marker molecules). (Link to probe), for amplification (for example, amplification of proteins, genes, etc.), for counter staining, for encapsulation, etc.

[000160] Specimen processing equipment can apply a wide range of substances to specimens. These materials include, but are not limited to, stains, probes, reagents, cleaning solutions, and / or modifiers. These substances can be fluids (eg, gases, liquids, or gas / liquid mixtures) and the like. As the fluid, a solvent (for example, a polar solvent, a non-polar solvent, etc.), a solution (for example, an aqueous solution or a solution of another kind) or the like can be used. Reagents include, but are limited to, stains, wetting agents, antibodies (eg, monoclonal antibodies, polyclonal antibodies, etc.), antigen recovery solutions (eg, aqueous or non-aqueous antigen buffers, antigen recovery buffers, etc.). Not done. The probe can be an isolated nucleic acid or an isolated synthetic oligonucleotide attached to a detectable label or reporter molecule. Labels include radioisotopes, enzyme substrates, cofactors, ligands, chemical luminescent or fluorescent agents, haptens, and enzymes.

[000161] As a sample processing apparatus, Ventana Medical Systems, Inc. Automation devices such as Benchmark XT appliances and SYMPHONY appliances sold by are possible. Ventana Medical Systems, Inc. Is the assignee of many US patents that have disclosed systems and methods for performing automated analysis, US Pat. Nos. 5,650,327, 5,654,200, 6,296,809, No. Including 6,352,861, 6,827,901, and 6,943,029, and US Patent Application Publications 2003/0211630 and 2004/0052685, all of which are referenced. Is incorporated herein by. Alternatively, the specimen can be processed manually.

[000162] After the specimen has been processed, the user can transfer the specimen support slides to the imaging device. In some embodiments, the imaging device is a brightfield imaging slide scanner. As one of the bright-field imaging devices, Ventana Medical Systems, Inc. There is an iScan HT and DP200 (Griffin) brightfield scanner for sale by. In an automated embodiment, the imaging device is an international patent application PCT / US2010 / 002772 (Patent Publication WO2011 / 049608) or September 9, 2011, entitled "IMAGING SYSTEM AND TECHNIQUES". Digital pathology as disclosed in US Patent Application No. 61 / 533,114 entitled "IMAGING SYSTEMS, CASSETTES, AND METHODS OF USING THE SAME" (imaging system, cassette, and how to use them). It is a device. All of International Patent Application PCT / US2010 / 002772 and US Patent Application No. 61 / 533,114 are incorporated herein by reference.

[000163] The imaging system or device may be a multispectral imaging (MSI) system or a fluorescence microscopy system. The imaging system used here is MSI. MSI generally prepares a computerized microscope-based imaging system for the analysis of pathological specimens by making the spectral distribution of the image accessible at the pixel level. There are a variety of multispectral imaging systems, but a common behavioral aspect of all of these systems is the ability to compose multispectral images. A multispectral image is one that captures image data at a specific wavelength or a specific spectral bandwidth of the entire electromagnetic spectrum. These wavelengths may be selected by the use of optical filters or other instruments capable of selecting predetermined spectral components, including electromagnetic radiation at wavelengths beyond the visible light range, such as infrared (IR). ..

[000164] MSI systems include optical imaging systems, some of which include a spectrum selection system that is adjustable to define a discrete optical band of a predetermined number N. The optical system may be configured to image a tissue sample transmitted and irradiated by a broadband light source on a photodetector. In one embodiment, the optical imaging system may include a magnifying system such as a microscope and has one optical axis that is roughly spatially aligned with one optical output of the optical system. The system constructs a series of images of tissue such that the spectrum selection system is tuned or tuned so that images are acquired in different discrete bands of spectrum (eg, by a computer processor). The device may also include a display, on which an image of at least one visually perceptible tissue appears from the acquired series of images. The spectrum selection system responds to either a group of optical filters such as gratings, thin film interference filters, or user input or pre-programmed processor commands to spectrum the light transmitted from the source through the sample to the detector. It may include an optical grating element, such as any other system configured to select a particular pass band from.

[000165] In an alternative embodiment, the spectrum selection system defines multiple optical outputs corresponding to N discrete bands of spectrum. This type of system captures the transmitted light output from the optical system, and in the identified spectral band, N lines of samples are taken on the detection system along the optical path corresponding to this identified spectral band. At least a portion of this light output is spatially redirected along spatially different optical paths.

[000166] Embodiments of the subject matter and operation described herein are digital electronic circuits or computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or these. It can be implemented in one or more combinations of them. Embodiments of the subject matter described herein are one or more computer programs, i.e., computer program instructions encoded on a computer storage medium for execution by a data processor or control of the operation of the data processor. It can be implemented as one or more modules. Any of the modules described herein may include logic executed by a processor. As used herein, "logic" refers to any information having a form of instruction signal and / or data that may be applied to affect the operation of the processor. Software is an example of logic.

[000167] The computer storage medium can be, and includes, computer-readable storage devices, computer-readable storage boards, random or sequential access memory arrays or devices, or combinations thereof. It is also possible. Further, the computer storage medium can be a source or destination of computer program instructions encoded as artificially generated propagating signals, although not propagating signals. The computer storage medium can also be, or can include, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices). is there. The operations described herein can be implemented as operations performed by a data processor with respect to data stored in one or more computer-readable storage devices or data received from other sources.

[000168] The term "programmed processor" includes all types of devices, devices, and machines for processing data, such as programmable microprocessors, computers, system-on-chips, or these. Multiple or combinations of them can be mentioned. Examples of the device include a dedicated logic circuit (for example, FPGA (field programmable gate array) or ASIC (application specific integrated circuit)). The device can also be hardware, as well as code that generates an execution environment for the target computer program (eg, processor firmware, protocol stack, database management system, operating system, cross-platform runtime environment, virtual machine, or Codes that make up one or more combinations of these). The device and execution environment can implement a variety of different computing model infrastructures, including web services, distributed computing, and grid computing infrastructure.

[000169] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiler or interpreter languages, declarative or procedural languages. It can also be deployed in any form, such as a single program or module, component, subroutine, object, or other unit suitable for use in a computing environment. The computer program may, but does not necessarily, support files in the file system. A program is part of a file that holds other programs or data (eg, one or more scripts stored in a markup language document), one file dedicated to the target program, or multiple tailors. It can be stored in a file (eg, a file containing one or more modules, subprograms, or parts of code). The computer program may be distributed on one computer or on multiple computers or sites located on one site and deployed to run on multiple computers interconnected by a communication network.

[000170] The processes and logic flows described herein are performed by one or more programmable processors that execute one or more computer programs and perform operations by producing operations on input data and output. Can be done. These processes and logic flows can be performed by dedicated logic circuits (eg FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits)). The device may also be implemented as a dedicated logic circuit (eg FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit)).

[000171] Suitable processors for executing computer programs include, for example, both general purpose and dedicated microprocessors, as well as any one or more processors of any type of digital computer. In general, the processor will receive instructions and data from read-only memory, random access memory, or both. Essential elements of a computer are a processor that performs operations according to instructions and one or more memory devices that store instructions and data. Also, in general, a computer includes one or more mass storage devices (eg, magnetic, magneto-optical disk, or optical disk) that store data, or for one or more mass storage devices. The operations will be combined to receive, transmit, or both of the data. However, the computer does not necessarily have to have such a device. In addition, the computer may be another device (eg, a mobile phone, mobile information terminal (PDA), mobile audio or video player, game console, global positioning system (GPS) receiver, or portable storage device (eg, universal serial). It can be embedded in a bus (USB) flash drive) or many other devices). Suitable devices for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (eg, EPROM, EEPROM, and flash memory devices). , Magnetic disks (eg, internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be complemented by a dedicated logic circuit or incorporated into a dedicated logic circuit.

[000172] In order to allow interaction with the user, embodiments of the subject described herein are display devices that display information to the user (eg, LCD (LCD), LED (Light Emitting Diode) displays, etc. Alternatively, it can be implemented on a computer having an OLED (organic light emitting diode) display) and a keyboard and pointing device (eg, mouse or trackball) that allows the user to give input to the computer. In some embodiments, a touch screen may be used to display information and accept input from the user. Also, other types of devices may be used as well to allow interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback), and the user input can be acoustic input, audio input, Alternatively, it can be received in any form such as tactile input. The computer can also interact with the user by sending and receiving documents to the device used by the user (eg, sending a web page to a web browser on the user's client device in response to a request received from the web browser). Is.

[000173] An embodiment of a subject described herein is a back-end component (eg, a data server), a middleware component (eg, an application server), or a front-end component (eg, one of the subjects described herein). Implemented in a computer system that includes a graphical user interface or web browser that allows the user to interact with the embodiment, or in any combination of one or more such backends, middleware, or frontend components. Can be done. The components of the system can be interconnected by digital data communication in any form or medium (eg, a communication network). Examples of communication networks are local area networks (“LAN”) and wide area networks (“WAN”), interconnect networks (eg, the Internet), and peer-to-peer networks (eg, ad hoc peer-to-two). Peer network). For example, the network 20 in FIG. 1 may include one or more local area networks.

[000174] A computer system can include any number of clients and servers. Clients and servers are generally remote from each other and typically interact through communication networks. The client-server relationship is created by computer programs that run on their respective computers and have a client-server relationship with each other. In some embodiments, the server sends data (eg, an HTML page) to the client device (eg, for the purpose of displaying data to the user interacting with the client device and accepting user input from the user). .. Data generated on the client device (eg, the result of user interaction) can be received from the client device on the server.

[000175] Additional Separation Method / Optional Separation Module
[000176] Separation is a procedure in which the measurement spectrum of a mixed pixel is decomposed into a group of constituent spectra or endmembers and a set of corresponding fragments or individuals to indicate the proportion of each endmember present in the pixel. Specifically, the separation process can determine local concentrations of individual stains by extracting stain-specific channels using well-known reference spectra for standard tissue and stain combinations. The separation may use a reference spectrum read from the control image or a reference spectrum estimated from the image being observed. By separating the component signal of each input pixel, stain-specific channels such as hematoxylin and eosin channels in H & E images or diaminobenzidine (DAB) and counterstain (eg, hematoxylin) channels in IHC images can be read and analyzed. It becomes. The terms "unmixing" and "color deconvolution" (or "deconvolution"), etc. (eg, "deconvolving", "unmixed") are used interchangeably in the art. To. In some embodiments, the multiplex images are separated by a separation module that uses linear separation. For linear separation, for example, Zimmermann "Spectral Imaging and Linear Microscope", Adv Biochem Engin / Biotechnol (2005) 95: 25. L. Lawson and R. J. Hanson, "Solving first squares Problems", Plentice Hall, 1974, Chapter 23, p. It is described in 161 and all the disclosures thereof are incorporated herein by reference. In linear stain separation, the measurement spectrum (S (λ)) at any pixel is considered to be a linear mixture of stain spectrum components, and is the proportion of the color reference (R (λ)) of each stain represented at that pixel. Equal to the sum of the weights (A).

[000177] S (λ) = A ₁ · R ₁ (λ) + A ₂ · R ₂ (λ) + A ₃ · R ₃ (λ) ... A _i · R _i (λ)
[000178] More generally, it can be expressed in the form of a matrix as follows.

[000179] S (λ) = ΣA i · R i (λ) or S = R · A
[000180] In the presence of the acquired M channel images and N individual stains, the sequence of the M × N matrix R is the optimal color system and is an N × 1 vector, as derived herein. A is an unknown number of individual stain ratios, and M × 1 vector S is a multi-channel spectrum vector measured in pixels. In these equations, the signal (S) for each pixel is measured during acquisition of the multiplex image and reference spectrum or optimal color system and is derived as described herein. The contribution of the various stains ( _Ai ) can be determined by calculating the contribution to each point in the measurement spectrum. In some embodiments, a solution is obtained using an inverse least squares fitting technique that minimizes the square difference between the measured spectrum and the calculated spectrum by solving the following set of equations.

[000181] [∂Σ _j {S (λ _j ) -Σ _i A _i · R _i (λ _j )} 2] / ∂A _i = 0
[000182] In this equation, j represents the number of detection channels and i is equal to the number of stains. In the solution of linear equations, conditional separation often results in a sum of weights (A) of 1.

[000183] In another embodiment, described in WO 2014/195193, entitled "Image Adaptive Physiological Color Separation", filed May 28, 2014. Separation is achieved using the above methods, all of which are incorporated herein by reference. In general, WO2014 / 195193 describes a separation method by separating the component signals of an input image using iteratively optimized reference vectors. In some embodiments, the correlation of image data from the assay to expected or ideal results specific to the nature of the assay determines quality metrics. If the correlation to poor quality images or ideal results is inadequate, until one or more reference sequence vectors in matrix R are adjusted to show good quality images that match physiological and anatomical requirements. The separation is iteratively repeated with the adjusted reference vector. Anatomical, physiological, and assay information may be used to define the rules that are applied to the measured image data to determine quality metrics. This information includes the method of staining the tissue, the tissue structure intended for staining or the tissue structure not intended for staining, and the relationships between the assay-specific structures, stains, and markers to be treated. The iterative process provides a stain-specific vector that can accurately identify structures of interest and biologically relevant information and produce images suitable for analysis with no noise or unwanted spectra. The reference vector is adjusted in the search space. The search space defines the range of values that the reference vector can take to represent a stain. The search space may be determined by scanning a variety of representative training assays, including known or commonly occurring problems, and determining a high quality reference vector set for the training assay. ..

[000184] In another embodiment, the method described in WO 2015/124772, entitled "Group Sparseness Model for Image Unising", filed February 23, 2015. Separation is achieved using, the entire disclosure of which is incorporated herein by reference. In general, WO2015 / 124772 describes separation using a group spasity framework, where the percentage of stain contributions from multiple collocation markers is modeled within the "same group" and multiple non-collocation markers. By giving co-localization information of multiple collocation markers to the modeling group spasity framework and solving the modeling framework using group lasso, modeled in groups with different percentages of stain contribution from , Produce a minimum squared solution within each group. This least squares solution corresponds to the separation of collocation markers and, within the group, produces a sparse solution corresponding to the separation of non-collocation markers. Further, WO2015 / 124772 can input image data obtained from a biological tissue sample, read reference data describing the stain color of each of a plurality of stains from an electronic memory, and localize the reference data in the biological tissue sample. By reading the collocation data describing the stain groups having at least one size of 2 or more from the electronic memory and using the reference data as the reference matrix. The separation method by calculating the solution of the group lasso standard for obtaining the separation image is described. In some embodiments, the method of separating images assigns the percentage of stain contributions from localized markers within a single group and the percentage of stain contributions from delocalized markers within separate groups. It may include a step of generating a group spasity model assigned in and a step of generating a least squares solution in each group by solving the group spasity model using a separation algorithm.

[000185] Example-Comparison of FAP positive areas between high resolution and medium resolution analysis methods
[000186] The accuracy of the FAP positive area results was compared by experiments using:
[000187] 1) FAP positive high resolution analysis. For this measurement, all FAP-positive pixels after thresholding at high magnification (20X) were accumulated at a spatial resolution of pixel size of 0.465 micrometers. After that, the reported area selected from the preliminary annotation area was obtained as the FAP positive area for each pixel of the area of interest.

[000188] 2) In the preliminary annotation region, the FAP positive area measured using the medium resolution analysis technique described herein is obtained by summing the FAP positive areas of the FAP superpixel object, seed, or polygon outline. calculated.

[000189] Six different annotation areas (see FIG. 14), each with a different shape (large or small, circular, or unusual shape, etc.), were analyzed according to both methods. As shown in FIG. 15 and the table below, there was no significant difference in the comparison results of FAP positive areas measured using these two methods (R ² = 0.99, p <0.001).

[000190] In conclusion, when summing the area features calculated within a superpixel for a particular annotation, the total area is equal to the area directly calculated within the annotation by the high resolution analytical technique. The FAP positive area results show that there is no significant difference in the calculations by the two methods (with or without superpixels) with different annotation regions.

[000191] All U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent publications subject to reference and / or publication in the application datasheet herein are in their entirety. Is incorporated herein by reference. The embodiments may be modified to provide yet another embodiment as needed to adopt the concepts of various patents, applications, and publications.

[000192] Although the present disclosure has been described with reference to many exemplary embodiments, many other improvements and embodiments within the spirit and scope of the principles of the present disclosure may be devised by those skilled in the art. It shall be understood. More specifically, without departing from the gist of the present disclosure, within the scope of the above disclosure, drawings, and accompanying claims, reasonable modifications and improvements in the component parts and / or arrangement of the subject combination configuration. Is possible. In addition to deformations and improvements in component parts and / or placement, those skilled in the art will also see alternative use.

Claims (32)

A method of storing image analysis data derived from an image of a biological specimen having at least one stain.
(A) A step of deriving one or more feature metrics from the image, and
(B) The step of segmenting the image into a plurality of sub-regions, where each sub-region is substantially in at least one of the presence or absence of staining, the staining intensity, or the local texture. Including uniform pixels, steps and
(C) A step of generating a plurality of representation objects based on the plurality of segmentation subregions, and
(D) A step of associating each of the plurality of representation objects with the derived feature metric,
(E) A step of storing the coordinates of each representation object in the database together with the association derived feature metric (along with).
Including methods. The method of claim 1, wherein the step of segmenting the image into the plurality of subregions comprises deriving a superpixel. The superpixels (i) group pixels by local k-means clustering and (ii) integrate small isolated regions into large nearest tangent superpixels using the connected components algorithm. The method according to claim 2, which is derived by the above method. The step of segmenting the image into the plurality of subregions comprises overlaying the sampling grid on the image, wherein the sampling grid defines a non-overlapping area having a predetermined size and shape. The method according to any one of claims 1 to 3. Any one of claims 1 to 4, wherein the subregion has a size of M × N, M is in the range of 50 pixels to 100 pixels, and N is in the range of 50 pixels to approximately 100 pixels. The method described in. The method according to any one of claims 1 to 5, wherein the representation object includes an outer shape of a sub-region that satisfies a predetermined dyeing intensity threshold. The method according to any one of claims 1 to 6, wherein the representation object includes seed points. The method according to claim 7, wherein the seed point is derived by calculating the center of gravity of each of the plurality of sub-regions. The method of claim 6, wherein the derived feature metrics include staining intensity and the average staining intensity of all pixels within the outline of each generated representation object is calculated. Claim 1 wherein the derived feature metric includes an expression score and an average expression score corresponding to an area within each generation subregion is associated with the generated plurality of representational object. The method according to any one of 7 to 7. The method according to any one of claims 1 to 7, further comprising a step of reading the stored coordinates and the association feature measurement reference data from the database and a step of projecting the read data onto the image. A system for deriving data corresponding to irregularly shaped cells from images of biological samples containing at least one stain, wherein (i) one or more processors and (ii) the one or more. Memory combined to one of the above processors, when executed by one or more of the above processors
(A) Derivation of one or more feature measurement criteria from the image, and
(B) To generate a plurality of sub-regions in the image, each sub-region having pixels with similar properties selected from color, brightness, and / or texture.
(C) Computing a series of representation objects based on the plurality of generated sub-regions,
(D) Associating the one or more derived feature metrics from the image with the calculated coordinates of each of the series of computational representation objects.
A system with memory, which stores computer executable instructions that cause the system to perform operations including. 12. The system of claim 12, wherein dividing the image into the plurality of subregions comprises deriving superpixels. 12. The system of claim 12 or 13, wherein the superpixels are derived using either a graph-based approach or a gradient climb-based approach. The superpixels are derived by (i) grouping the pixels by local k-means clustering and (ii) integrating a small separation region into a large closest superpixel using a connected component algorithm. The system according to any one of 14. The system according to any one of claims 12 to 15, wherein the representation object includes an outer shape of a sub-region that satisfies a predetermined staining intensity threshold. The system according to any one of claims 12 to 16, wherein the representation object includes a seed point. The system according to any one of claims 12 to 17, wherein the operation further comprises storing the coordinates of the one or more derived feature metrics and association computational representation objects in a database. One of claims 12-18, wherein the one or more derived feature metrics include at least one expression score selected from percent positivity, H-score, and staining intensity. Described system. The system according to any one of claims 12 to 19, wherein data corresponding to irregularly shaped cells are derived with respect to the region of interest in the image. The system of claim 20, wherein the area of interest is an area of the image annotated by a medical professional. A non-transitory computer-readable medium containing instructions for analyzing data associated with a living body having an irregular shape.
(A) An instruction to derive one or more feature metrics from an image of a biological sample, wherein the biological sample contains at least one stain.
(B) A command that divides the image into a series of subregions by grouping pixels with similar characteristics, the characteristic being selected from color, brightness, and / or texture.
(C) An instruction for calculating a plurality of expression objects based on the series of division subregions, and
(D) An instruction for associating the one or more derived feature measurement criteria from the image with the calculated coordinates of each of the plurality of arithmetic expression objects.
Non-transitory computer-readable media, including. 22. The non-transitory computer-readable medium of claim 22, wherein dividing the image into the series of sub-regions comprises computing superpixels. 22. The superpixel is calculated using one of a normalized cuts algorithm, an agglomerative clustering algorithm, a quickshift algorithm, a turbopixel algorithm, or a simple linear iterative clustering algorithm. Alternatively, the non-temporary computer-readable medium according to 23. 22. To claim 22, the superpixels were generated using simple iterative clustering, the superpixel size parameter was set to about 40 pixels to about 400 pixels, and the density parameter was set to about 10 to about 100. The non-temporary computer-readable medium according to any one of 24. 22. To claim 22, the superpixels are calculated by (i) grouping pixels by local k-means clustering and (ii) integrating small separation regions into large closest superpixels using a concatenated component algorithm. The non-transitory computer-readable medium according to any one of 25. Any one of claims 22-26, wherein the biological sample is stained with at least FAP and the one or more derived feature metrics comprises at least one of FAP staining intensity or FAP percentage positiveness. Non-temporary computer-readable medium described in. 28. The non-transitory computer-readable medium of claim 27, wherein the average FAP percentage positiveness is calculated for all pixels in the subdomain. 28. The non-transitory computer-readable medium of claim 27, wherein the average FAP staining intensity is calculated for all pixels in the sub-region. The non-transitory computer-readable medium according to any one of claims 22 to 26, wherein the representation object includes at least one of a polygon outline and a seed point. The non-transitory computer-readable medium of any one of claims 22-26, further comprising an instruction to store the coordinates of the one or more derived feature metrics and association computational representation objects in a database. The non-transitory computer-readable medium of claim 31, further comprising an instruction to project the stored information onto the image of the biological sample.