WO2017151799A1 - Improved image analysis algorithms using control slides - Google Patents
Improved image analysis algorithms using control slides Download PDFInfo
- Publication number
- WO2017151799A1 WO2017151799A1 PCT/US2017/020245 US2017020245W WO2017151799A1 WO 2017151799 A1 WO2017151799 A1 WO 2017151799A1 US 2017020245 W US2017020245 W US 2017020245W WO 2017151799 A1 WO2017151799 A1 WO 2017151799A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- control
- ihc
- artifacts
- image analysis
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
- G06T7/0014—Biomedical image inspection using an image reference approach
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/754—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries involving a deformation of the sample pattern or of the reference pattern; Elastic matching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/757—Matching configurations of points or features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30072—Microarray; Biochip, DNA array; Well plate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
Definitions
- the present disclosure generally relates to the field of medical imaging. Some examples discussed in the present disclosure relate to methods of automatically detecting and excluding artifacts from image analysis of images of biological specimens.
- the biological specimens are sometimes stained with one or more different stains or dyes in order to highlight various types of structures within the specimen, such as specific types of tissue, cells, cell organelles, and so forth. This can enable pathologists and other health care professionals to better assess the specimen's morphology and deliver a more accurate diagnosis, prognosis, and treatment plan for the patient.
- H&E staining One method of staining the specimen is hematoxylin and eosin (H&E) staining, in which a dark blue or violet hematoxylin binds to basophilic substances such as DNA and RNA, and a red or pink eosin binds to acidophilic substances such as positively charged amino acid side chains.
- basophilic substances such as DNA and RNA
- red or pink eosin binds to acidophilic substances such as positively charged amino acid side chains.
- cell nuclei on specimen slides stained with H&E are likely to appear in blue or purple, while structures like the cytoplasm, basophils, muscles, erythrocytes, collagen, and mitochondria are likely to appear in different shades of pink or red.
- ISH staining is a technique that uses a labeled complementary DNA, RNA or modified nucleic acids strand (i.e., probe) to localize a specific DNA or RNA sequence in a portion or section of tissue.
- In situ hybridization can be a powerful technique for identifying specific mRNA species within individual cells in tissue sections, providing insights into physiological processes and disease pathogenesis.
- Immunohistochemistry (IHC) staining is another staining technique that has become popular due to its significant efficiencies and the rich diagnostic information that it generates.
- IHC slide staining can be utilized to identify proteins in cells of a tissue section and hence is widely used in the study of different types of cells, such as cancerous cells and immune cells in biological tissue.
- IHC staining may be used in research and clinical settings to understand the distribution and localization of the differentially expressed biomarkers of immune cells in a cancerous tissue for an immune response study.
- tumors often contain infiltrates of immune cells, which may prevent the development of tumors or favor the outgrowth of tumors.
- multiple stains may be used to target different types of immune cells, and the population distribution of each type of immune cell is used in studying the clinical outcome of the patients.
- Digital pathology systems include slide scanners that can scan tissue slides to produce very high resolution (e.g., 0.5 or 0.275 microns per pixel) digital images of the tissue. Digital pathology systems also include hardware and/or software solutions for automatically processing, viewing, segmenting, analyzing (e.g., scoring) and otherwise managing tissue images or other types of biological specimen images.
- a negative control slide can include an adjacent tissue section prepared in the same way as the tissue on the main IHC slide, with the exception that the primary antibody is either omitted or replaced with some other solution such as isotype, tissue culture media, buffer, or dilute non-immune serum of the same species as the primary antibody.
- Such slides can be used, for example, to determine the level of background stain and make sure that it does not exceed acceptable limits.
- Disclosed herein are systems and methods for automatically excluding artifacts from an analysis of a biological specimen image.
- an image analysis system for analyzing biological specimen images.
- the system may include an image registration engine that can be configured to obtain an immunohistochemistry (IHC) image and a control image, and to determine whether the IHC image and the control image are structurally similar.
- the system may also include an artifact detection engine configured to determine whether the control image includes one or more artifacts, and upon a determination that the control image includes one or more artifacts, to identify one or more artifact regions within the IHC image by mapping the one or more artifacts from the control image to the IHC image.
- the system may further include, among other things, an image analysis engine configured to perform an image analysis of the IHC image, where the one or more artifact regions are excluded from the image analysis based on a determination that the IHC image and the control image are structurally similar.
- a method of detecting and excluding artifact regions from image analysis may include, among other things, obtaining an immunohistochemistry (IHC) image and a control image; determining whether the control image includes one or more artifacts; upon a determination that the control image includes one or more artifacts, identifying one or more artifact regions within the IHC image by mapping the one or more artifacts from the control image to the IHC image; and performing an image analysis of the IHC image, wherein the one or more artifact regions are excluded from the image analysis.
- a non-transitory computer-readable medium storing instructions is disclosed.
- the instructions when executed by a processor of an image analysis system, may cause the image analysis system, for example, to obtain a field of view (FOV) of an immunohistochemistry (IHC) image of an IHC slide; obtain a control image of a control slide associated with the IHC slide; map the FOV from the IHC image to the control image to obtain a control FOV; detect artifacts within the control FOV; and notify a user if any artifacts have been detected in the control FOV.
- FOV field of view
- IHC immunohistochemistry
- FIG. 1 shows an exemplary image acquisition system and an exemplary image analysis system, in accordance with some embodiments.
- FIG. 2 shows various examples of image registration, in accordance with some embodiments.
- FIG. 3 shows examples of regions of IHC and negative control (NC) images.
- FIG. 4 shows other examples of regions of IHC and NC images.
- FIG. 5 shows yet other examples of regions of IHC and NC images.
- FIG. 6 illustrates an FOV-based workflow, in accordance with some embodiments.
- FIG. 7 shows an exemplary method of automatically detecting and excluding artifact regions from image analysis, such as a whole-slide image analysis, in accordance with some embodiments.
- FIG. 8 shows an exemplary method of automatically detecting artifacts in an FOV, in accordance with some embodiments.
- Methods and systems described herein are directed to using previously unutilized control slides to identify regions in the IHC image that contain artifacts and excluding those areas from the analysis, thereby improving the algorithm's overall accuracy.
- Such methods and systems can be especially beneficial when applied to whole-slide-analysis workflows, where the pathologist cannot avoid the artifact problem by finding a field of view (FOV) free of artifacts.
- FOV field of view
- the techniques discussed herein can also improve an FOV-based workflow, because some artifacts can appear so substantially similar to a positive IHC stain, that distinguishing them from the positive stain based on an IHC image alone can be a very time consuming and difficult (if not impossible) task even for trained pathologists.
- the methods and systems disclosed herein can be used to detect and exclude artifacts that appear on both the IHC slide and the adjacent control slides.
- artifacts may include, for example, tissue pigments (e.g., melanin, hemosiderin, etc.) or other artifacts associated with the tissue itself.
- FIG. 1 is a block diagram of an image analysis system 100, according to an exemplary embodiment of the subject disclosure.
- Image analysis system 100 may include one or more computing devices such as desktop computers, laptop computers, tablets, smartphones, servers, application- specific computing devices, or any other type(s) of electronic device(s) capable of performing the techniques and operations described herein.
- image analysis system 100 may be implemented as a single device. In other embodiments, image analysis system 100 may be implemented as a combination of two or more devices together achieving the various functionalities discussed herein.
- image analysis system 1 00 may include one or more server computers and a one or more client computers communicatively coupled to each other via one or more local-area networks and/or wide-area networks such as the Internet.
- image analysis system 100 may include a memory 1 16, a processor 1 1 7, and a display 1 18.
- Memory 1 16 may include any combination of any type of volatile or non-volatile memories, such as random-access memories (RAMs), read-only memories such as an Electrically-Erasable Programmable Read-Only Memory (EEPROM), flash memories, hard drives, solid state drives, optical discs, and the like.
- RAMs random-access memories
- EEPROM Electrically-Erasable Programmable Read-Only Memory
- flash memories hard drives, solid state drives, optical discs, and the like.
- FIG. 1 memory 1 16 is depicted in FIG. 1 as a single device, but it is appreciated that memory 1 16 can also be distributed across two or more devices.
- Processor 1 1 7 may include one or more processors of any type, such as central processing units (CPUs), graphics processing units (GPUs), special-purpose signal or image processors, field-programmable gate arrays (FPGAs), tensor processing units (TPUs), and so forth.
- processors such as central processing units (CPUs), graphics processing units (GPUs), special-purpose signal or image processors, field-programmable gate arrays (FPGAs), tensor processing units (TPUs), and so forth.
- processor 1 17 is depicted in FIG. 1 as a single device, but it is appreciated that processor 1 17 can also be distributed across any number of devices.
- Display 1 18 may be implemented using any suitable technology, such as LCD, LED, OLED, TFT, Plasma, etc.
- display 1 1 8 may be a touch-sensitive display (a touchscreen).
- image analysis system 100 may also include an image registration engine 1 1 1 , an artifact detection engine 1 12, and an image analysis engine 1 13. While these modules are depicted in FIG. 1 as standalone modules, it will be evident to persons having ordinary skill in the art that each module may instead be implemented as a number of sub- modules, and that in some embodiments any two or more modules can be combined into a single module. Furthermore, in some embodiments, system 100 may include additional engines and modules (e.g., input devices, networking and communication modules, etc.) not depicted in FIG. 1 for brevity. Furthermore, in some embodiments, some of the blocks depicted in FIG. 1 may be disabled or omitted. As will be discussed in more detail below, the functionality of any module of system 100 can be implemented in hardware, software, firmware, or as any combination thereof.
- image analysis system 100 may be communicatively coupled to an image acquisition system 120.
- Image acquisition system 120 may obtain images of biological specimens and provide those images to image analysis system 1 00 for analysis and presentation to the user.
- Image acquisition system 120 may include an automated staining platform 123 for staining biological specimen slides using H&E, IHC, ISH, FISH, or any other staining process.
- Commercially available staining platforms include, for example, HE 600, BENCHMARK XT, and BENCHMARK ULTRA products by Ventana Medical Systems, Inc.
- Image acquisition system 120 may also include a scanning platform 1 25 such as a slide scanner that can scan the stained slides at 20x, 40x, or other magnifications to produce high resolution whole-slide digital images.
- Commercially available slide scanners include, for example, VENTANA iScan HT and iScan Coreo products by Ventana Medical Systems, Inc.
- Images generated by scanning platform 125 may be transferred to image analysis system 1 00 or to a server or database accessible by image analysis 100.
- the images may be transferred automatically via one or more local-area networks and/or wide-area networks.
- image analysis system 100 may be integrated with or included in scanning platform 1 25 and/or other modules of image acquisition system 120, in which case the image may be transferred to image analysis system, e.g., through a memory accessible by both platform 125 an system 120.
- image acquisition system 120 may not be communicatively coupled to image analysis system 100, in which case the images may be stored on a nonvolatile storage medium of any type (e.g., a flash drive) and downloaded from the medium to image analysis system 1 00 or to a server or database communicatively coupled thereto.
- image analysis system 100 may obtain an image of a biological sample, where the sample may have been affixed to a slide and stained by staining platform 123, and where the slide may have been scanned by a slide scanner or another type of scanning platform 125. It is appreciated, however, that in other embodiments, below-described techniques may also be applied to images of biological samples acquired through other means.
- image analysis system 100 may obtain and pass to image registration engine 1 1 1 a control image (an image of a negative control slide or a positive control slide) and an IHC image (an image of an IHC slide adjacent to the control slide).
- adjacent slides refer to slides containing tissue sections that were cut from the same tissue block.
- the two tissue sections can be the closest two sections that were cut from that block.
- the two sections may not necessarily be the closest ones from that block, but may still be within sufficiently close proximity (e.g., within ⁇ ⁇ or 20 ⁇ ) of each other.
- image registration engine 1 1 1 may align the two images in a process referred to as "registration.” In some embodiments, engine 1 1 1 may use the following registration method to align the images.
- 11 represents the IHC image and I2 represents the control image.
- ⁇ x1 ,i, y1 ,i ⁇ represent the i-th cell center for image 11 .
- ⁇ x2,i, y2,i ⁇ represent the i-th cell center for image I2.
- engine 1 1 1 may calculate the affine transformation between the two images using Procrustes method or any other suitable method. After computing the transformation function, engine 1 1 1 may use it to transform point set P1 into a transformed point set P1 '.
- D(P1 , P2) can then be computed as the mean distance between every point in P2 and the nearest point in P1 '.
- engine 1 1 1 may instead transform point set P2 into a transformed point set P2' and compute D(P1 , P2) as the mean distance between every point in P1 and the nearest point in P2'.
- engine 1 1 1 may determine, based on the mean distance, whether images 11 and I2 are structurally similar.
- "structurally similar" can mean that the location of most or all of the cell nuclei, TMA cores, needle biopsies, and/or any other types of objects of interest will be substantially similar in both images.
- engine 1 1 1 may compare the mean distance D(P1 , P2) to a predefined threshold and determine that the images are structurally similar if and only if the mean distance is below that predefined threshold. To illustrate, FIG.
- FIG. 2 shows at the top row image 21 1 containing point set P1 , adjacent image 21 2 containing point set P2, and image 213 that shows transformed (90-degree rotated) point set P1 ' overlaid upon point set P2.
- point sets P1 ' and P2 are well matched, indicating structural similarity between adjacent images 21 1 and 212.
- FIG. 2 shows image 231 containing point set P1 , adjacent image 232 containing point set P2, and image 233 that shows the two point sets overlaid on each other. It is apparent from this example that even the most optimal transformation function could not transform P1 such that it would match P2, i.e., such that D(P1 , P2) would be relatively low. Accordingly, in this example there is probably no structural similarity between adjacent images 231 and 232. ln some embodiments, upon a determination by image registration engine
- engine 1 12 may obtain the control image and process it to detect artifacts.
- engine 1 12 may run the artifact detection process or portions thereof in parallel with the image registration process performed by image registration engine 1 1 1 .
- Artifact detection engine 1 12 can use any suitable algorithm to detect artifacts in the control image.
- the artifacts can be detected using an intensity-based thresholding method.
- engine 1 12 may obtain a grayscale version of a negative control image and determine that higher intensity pixels (e.g., pixels whose intensity is higher than a predefined threshold) correspond to counter-stain regions, while the rest of the pixels (e.g., pixels whose intensity is lower or equal to the predefined threshold) correspond to artifacts.
- engine 1 12 may obtain a grayscale version of a positive control image and determine that lower intensity pixels (e.g., pixels whose intensity is lower than a first threshold) correspond to positively stained regions; higher intensity pixels (e.g., pixels whose intensity is higher than a second threshold) correspond to the image background, and that the rest of the pixels (e.g., pixels whose intensity is between the first and the second thresholds) correspond to artifacts.
- lower intensity pixels e.g., pixels whose intensity is lower than a first threshold
- higher intensity pixels e.g., pixels whose intensity is higher than a second threshold
- the rest of the pixels e.g., pixels whose intensity is between the first and the second thresholds
- artifact detection engine 1 1 2 may obtain (e.g., from memory 1 16 or via input from the user) a predefined color range corresponding to all possible non-artifact colors (e.g., counter-stain colors in negative control slides or stain colors in positive control slides).
- the predefined color range may be defined, for example, in a one-dimensional, two-dimensional, or three-dimensional space (e.g., the LAB color space), where a one-dimensional color range can also be referred to as an intensity range.
- engine 1 12 may determine that any pixels whose color/intensity falls outside of the predefined color/intensity range are artifacts.
- engine 1 12 may also obtain a color/intensity range representing the background, and determine that any pixels whose color/intensity range falls outside both the predefined color/intensity range and the background color/intensity range are artifacts.
- engine 1 12 may analyze the control image and automatically determine these ranges, for example, by finding one or two highest peaks on the color/intensity histogram, and establishing that the peaks correspond to the background and either the stain or the counter- stain.
- artifact detection engine 1 1 2 can map all artifact candidates from the control image onto the IHC image using, for example, the transformation function computed by image registration engine 1 1 1 (discussed above). After the artifacts are mapped onto the IHC image, artifact mapping engine 1 1 2 may pass the locations of the artifacts on the IHC image to image analysis engine 1 13.
- image analysis engine 1 13 After image analysis engine 1 13 obtains the locations of the artifacts on the IHC image, it may exclude such artifacts from its analysis. In other words, when image analysis engine 1 13 subsequently runs an image analysis algorithm on the IHC image, it may disregard any artifact, or any region containing an artifact or at least a predefined percentage of artifacts.
- region as used herein can correspond, for example, to an area of a predefined size, such as a circle of a predefined radius, a rectangle of a predefined size (e.g., a slide tile or an FOV), and so forth.
- analysis engine 1 13 may highlight or outline those regions on display 1 18, thereby allowing the user to modify the regions, e.g., by making them larger or smaller, by manually drawing additional regions, by cancelling some of the identified regions (i.e., marking them as non-artifact regions), and so on.
- image analysis engine 1 13 can perform the image analysis on the IHC image while excluding those regions from the analysis.
- the image analysis may be a whole-slide scoring algorithm that counts various types of cells (e.g., all or a certain type of stained and/or unstained cells), their intensities, ratios, etc., throughout the entire slide or at least a large portion (e.g., >50%) of the area covered by tissue. Excluding (i.e., disregarding) the artifact regions from the algorithm's calculations can improve the algorithm's accuracy, because the algorithm could have falsely identified some artifacts as stained cells or other types of significant objects.
- FIGS. 3-5 illustrate some of the methods described above using by showing various samples with different types of artifacts.
- FIG. 3 shows two regions of a melanoma specimen, where region 1 of the negative control (NC) image has dark spots attributable to melanin, while region 2 of the NC image has no such spots.
- image registration engine 1 1 1 to be structurally similar to the adjacent NC image
- melanin pigments detected in region 1 of the NC image by artifact detection engine 1 12 will be mapped by that engine into a corresponding region 1 of the IHC image (top left) and the entire region can be excluded by image analysis engine 1 13 when it runs a whole-slide image analysis algorithm on the IHC image.
- FIG. 4 shows an example of pigment detection in renal cell carcinoma (RCC) where region 1 contains hemosiderin pigment and region 2 does not.
- NC regions 5 shows two additional NC regions, one of a triple negative breast cancer (TNBC) sample and one from a renal cell carcinoma (RCC) sample.
- TNBC triple negative breast cancer
- RNC renal cell carcinoma
- the techniques discussed above can also be applied to FOV-based algorithms, where the user selects (e.g., on an IHC image) one or more FOVs where the user wishes to run the image analysis algorithm.
- image registration engine 1 1 1 can use the above- discussed techniques to determine whether the IHC image and the adjacent control slide are structurally similar. If so, artifact engine 1 1 2 can use the above-discussed techniques to map the selected FOVs from the IHC image onto the adjacent control slide, and to determine whether the mapped FOVs of the control slide include any artifacts (or an amount/percentage of artifacts that exceeds a predefined threshold).
- FIG. 6 illustrates an example IHC image 610, an adjacent NC image 620, and two FOVs (FOV1 and FOV2) mapped from IHC image to adjacent NC image 620.
- FOV1 and FOV2 two FOVs mapped from IHC image to adjacent NC image 620.
- neither of the NC FOVs appear to have any dark stain, meaning that all the stain in the corresponding IHC FOVs is likely attributable to DAB stain and not to artifacts such as pigment.
- FIG. 7 illustrates a flowchart of an exemplary method 700 for automatically (without any input from the user) detecting and excluding artifact regions from an image analysis (e.g., a whole-slide image analysis).
- Method 700 may be executed, for example, by various modules of image analysis system 1 00 discussed above.
- Method 700 may be implemented, for example, in the form of executable instructions stored on at least one non- transitory machine-readable storage medium (e.g., memory 1 16) and executed by at least one processor or processor core (e.g., processor 1 1 7).
- Method 700 may also be implemented in the form of electronic circuitry (e.g., hardware), or in the form of some combination of electronic circuitry and instructions executable by a processor.
- Method 700 may begin at block 710 where an IHC image can be aligned with an adjacent control image, as discussed above.
- block 720 it may be determined whether or not the IHC image is structurally similar to the adjacent control image. If so, at block 730 artifacts may be detected on the control image.
- any artifacts have been detected. If so, at block 750, regions containing artifacts can be mapped from the control image to the IHC image, and at block 760 the mapped IHC regions can be excluded from the whole-slide image analysis.
- FIG. 8 illustrates a flowchart of an exemplary method 800 for automatically (without any input from the user) detecting artifacts in an FOV.
- Method 800 may be executed, for example, by various modules of image analysis system 1 00 discussed above.
- Method 800 may be implemented, for example, in the form of executable instructions stored on at least one non- transitory machine-readable storage medium (e.g., memory 1 16) and executed by at least one processor or processor core (e.g., processor 1 1 7). Method 800 may also be implemented in the form of electronic circuitry (e.g., hardware), or in the form of some combination of electronic circuitry and instructions executable by a processor. It will be appreciated by persons skilled in the art based on the following description of method 800 that some blocks of method 800 may be executed substantially concurrently or in a different order than shown in FIG. 8; that method 800 may include more or less blocks than are shown in FIG. 8; and that some blocks of method 800 may, at certain times, be ongoing and/or may repeat.
- non- transitory machine-readable storage medium e.g., memory 1 16
- processor or processor core e.g., processor 1 1 7
- Method 800 may also be implemented in the form of electronic circuitry (e.g., hardware), or in the form of some combination of electronic
- Method 800 may begin at block 805 where an FOV may be obtained (e.g., selected by the user) on an IHC image.
- the IHC image may be aligned with an adjacent control image using a whole-slide registration method.
- IC integrated circuits
- ASICs application-specific integrated circuits
- FPGAs field-programmable gate arrays
- the software or firmware may include any type of processor executable instructions that can be stored on any type of tangible non-transitory computer-readable medium, where the instructions can be executed by a processing resource, causing the processing resource to implement the functionality of the respective component.
- the executable instructions can be stored in memory 1 1 6, or in any other non-transitory computer-readable storage medium, which can be implemented as any combination of any type of volatile or non-volatile memories, such as random-access memories (RAMs), read-only memories such as an Electrically-Erasable Programmable Read-Only Memory (EEPROM), flash memories, hard drives, solid state drives, optical discs, and the like.
- RAMs random-access memories
- EEPROM Electrically-Erasable Programmable Read-Only Memory
- flash memories hard drives, solid state drives, optical discs, and the like.
- the non-transitory computer-readable medium can be integrated in the same device as the processor resource or it may be separate but accessible to that device and the processor resource.
- the program instructions can be part of an installation package that when installed can be executed by the processor resource to implement the corresponding component.
- the computer- readable medium may be a portable medium such as a CD, DVD, or flash drive or a memory maintained by a server from which the installation package can be downloaded and installed.
- the program instructions may be part of an application or applications already installed, and the computer-readable medium may include integrated memory such as a hard drive, solid state drive, random access memory (RAM), read-only memory (ROM), and the like.
- the processing resource for executing the instructions can be, for example, processor 1 17, or it can be any other processing resource, such as a CPU, a GPU, an FPGA, a TPU, another type of processor configured to retrieve and execute instructions, or a combination thereof.
- the processing resource can be integrated in a single device or distributed across devices, which may be integrated with image analysis system 100 or communicatively coupled to image analysis system 100.
- the specification may have presented the method and/or process of the present subject disclosure as a particular sequence of steps. However, to the extent that the method or process does not rely on the particular order of steps set forth herein, the method or process should not be limited to the particular sequence of steps described. As one of ordinary skill in the art would appreciate, other sequences of steps may be possible. Therefore, the particular order of the steps set forth in the specification should not be construed as limitations on the claims. In addition, the claims directed to the method and/or process of the present subject disclosure should not be limited to the performance of their steps in the order written, and one skilled in the art can readily appreciate that the sequences may be varied and still remain within the spirit and scope of the present subject disclosure.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Multimedia (AREA)
- Artificial Intelligence (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Quality & Reliability (AREA)
- Radiology & Medical Imaging (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Investigating Or Analysing Biological Materials (AREA)
- Image Analysis (AREA)
Abstract
Description
Claims
Priority Applications (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
AU2017226252A AU2017226252A1 (en) | 2016-03-01 | 2017-03-01 | Improved image analysis algorithms using control slides |
CA3012360A CA3012360C (en) | 2016-03-01 | 2017-03-01 | Improved image analysis algorithms using control slides |
EP17760737.1A EP3424019A4 (en) | 2016-03-01 | 2017-03-01 | Improved image analysis algorithms using control slides |
JP2018545836A JP6960935B2 (en) | 2016-03-01 | 2017-03-01 | Improved image analysis algorithm using control slides |
US16/119,520 US10937162B2 (en) | 2016-03-01 | 2018-08-31 | Image analysis algorithms using control slides |
US17/158,121 US11620751B2 (en) | 2016-03-01 | 2021-01-26 | Image analysis algorithms using control slides |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201662302062P | 2016-03-01 | 2016-03-01 | |
US62/302,062 | 2016-03-01 | ||
US201762464972P | 2017-02-28 | 2017-02-28 | |
US62/464,972 | 2017-02-28 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/119,520 Continuation US10937162B2 (en) | 2016-03-01 | 2018-08-31 | Image analysis algorithms using control slides |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2017151799A1 true WO2017151799A1 (en) | 2017-09-08 |
Family
ID=59744373
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2017/020245 WO2017151799A1 (en) | 2016-03-01 | 2017-03-01 | Improved image analysis algorithms using control slides |
Country Status (6)
Country | Link |
---|---|
US (2) | US10937162B2 (en) |
EP (1) | EP3424019A4 (en) |
JP (1) | JP6960935B2 (en) |
AU (1) | AU2017226252A1 (en) |
CA (1) | CA3012360C (en) |
WO (1) | WO2017151799A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10937162B2 (en) | 2016-03-01 | 2021-03-02 | Ventana Medical Systems, Inc. | Image analysis algorithms using control slides |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10453221B2 (en) * | 2017-04-10 | 2019-10-22 | Intel Corporation | Region based processing |
KR102037117B1 (en) * | 2017-08-02 | 2019-11-29 | 재단법인 아산사회복지재단 | Method and program for reducing artifacts by structural simirality, and medial imaging device |
JP7504116B2 (en) * | 2019-03-28 | 2024-06-21 | エフ・ホフマン-ラ・ロシュ・アクチェンゲゼルシャフト | Machine learning using distance-based similarity labels |
US20230053128A1 (en) * | 2019-12-20 | 2023-02-16 | Roche Diagnostics Hematology, Inc. | Linearity control compositions and methods of use |
WO2022225845A1 (en) * | 2021-04-18 | 2022-10-27 | Mary Hitchcock Memorial Hospital, For Itself And On Behalf Of Dartmouth-Hitchcock Clinic | System and method for automation of surgical pathology processes using artificial intelligence |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050123181A1 (en) * | 2003-10-08 | 2005-06-09 | Philip Freund | Automated microscope slide tissue sample mapping and image acquisition |
US20140233797A1 (en) * | 2008-11-14 | 2014-08-21 | The Scripps Research Institute | Image analysis platform for identifying artifacts in samples and laboratory consumables |
WO2015052128A1 (en) * | 2013-10-07 | 2015-04-16 | Ventana Medical Systems, Inc. | Systems and methods for comprehensive multi-assay tissue analysis |
US20160042511A1 (en) * | 2013-03-15 | 2016-02-11 | Ventana Medical Systems, Inc. | Tissue Object-Based Machine Learning System for Automated Scoring of Digital Whole Slides |
Family Cites Families (25)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7171030B2 (en) * | 2000-11-30 | 2007-01-30 | University Of Medicine & Denistry Of New Jersey | Systems for analyzing microtissue arrays |
US8185317B2 (en) * | 2003-06-12 | 2012-05-22 | Cytyc Corporation | Method and system of determining the stain quality of slides using scatter plot distribution |
EP1650705B1 (en) * | 2003-07-28 | 2013-05-01 | Olympus Corporation | Image processing apparatus, image processing method, and distortion correcting method |
US20070237714A1 (en) * | 2006-03-31 | 2007-10-11 | Alvarez Vernon L | Diagnosis and treatment of tumors |
WO2008016679A2 (en) * | 2006-08-02 | 2008-02-07 | 24Eight Llc | Wireless detection and alarm system for monitoring human falls and entries into swimming pools by using three dimensional acceleration and wireless link energy data method and apparatus |
US8060348B2 (en) * | 2006-08-07 | 2011-11-15 | General Electric Company | Systems for analyzing tissue samples |
US8975026B2 (en) * | 2007-01-16 | 2015-03-10 | Somalogic, Inc. | Method for generating aptamers with improved off-rates |
WO2008123969A1 (en) * | 2007-04-02 | 2008-10-16 | The Trustees Of The University Of Pennsylvania | Combined feature ensemble mutual information image registration |
EP2227774B1 (en) * | 2007-12-04 | 2014-07-09 | University College Dublin, National University of Ireland Dublin | Method and system for image analysis |
US8416405B2 (en) * | 2008-08-08 | 2013-04-09 | Chemimage Corporation | Raman chemical imaging of implantable drug delivery devices |
US9601393B2 (en) * | 2009-02-06 | 2017-03-21 | Kla-Tencor Corp. | Selecting one or more parameters for inspection of a wafer |
US8077959B2 (en) * | 2009-09-30 | 2011-12-13 | General Electric Company | Stain-based optimized compression of digital pathology slides |
EP2369551B1 (en) * | 2010-03-25 | 2019-10-30 | Emory University | Imaging system and method |
US20120259574A1 (en) * | 2011-04-05 | 2012-10-11 | Chenmin Hu | Designed-based yield management system |
US8718350B2 (en) * | 2011-04-08 | 2014-05-06 | Regents Of The University Of Minnesota | Computerized methods for tissue analysis |
WO2013072816A1 (en) * | 2011-11-17 | 2013-05-23 | Koninklijke Philips Electronics N.V. | Image processing |
US9488639B2 (en) * | 2013-02-25 | 2016-11-08 | Flagship Biosciences, Inc. | Cell-based tissue analysis |
US9786050B2 (en) * | 2013-03-15 | 2017-10-10 | The Board Of Trustees Of The University Of Illinois | Stain-free histopathology by chemical imaging |
US10820881B2 (en) * | 2013-08-02 | 2020-11-03 | Adm Diagnostics, Inc. | Method for analyzing and correcting measurement variability in PET images |
CA2935473C (en) * | 2014-02-21 | 2022-10-11 | Ventana Medical Systems, Inc. | Medical image analysis for identifying biomarker-positive tumor cells |
WO2015135550A1 (en) * | 2014-03-10 | 2015-09-17 | Visiopharm A/S | Assessment of staining quality |
WO2016015108A1 (en) * | 2014-08-01 | 2016-02-04 | Katholieke Universiteit Leuven | System for interpretation of image patterns in terms of anatomical or curated patterns |
US9558389B2 (en) * | 2015-03-24 | 2017-01-31 | Intel Corporation | Reliable fingertip and palm detection |
CA3012360C (en) | 2016-03-01 | 2021-12-14 | Ventana Medical Systems, Inc. | Improved image analysis algorithms using control slides |
US10504224B1 (en) * | 2016-12-31 | 2019-12-10 | Flagship Biosciences, Inc. | Method for detecting immunostaining below the threshold of visual perception |
-
2017
- 2017-03-01 CA CA3012360A patent/CA3012360C/en active Active
- 2017-03-01 EP EP17760737.1A patent/EP3424019A4/en active Pending
- 2017-03-01 AU AU2017226252A patent/AU2017226252A1/en not_active Abandoned
- 2017-03-01 WO PCT/US2017/020245 patent/WO2017151799A1/en active Application Filing
- 2017-03-01 JP JP2018545836A patent/JP6960935B2/en active Active
-
2018
- 2018-08-31 US US16/119,520 patent/US10937162B2/en active Active
-
2021
- 2021-01-26 US US17/158,121 patent/US11620751B2/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050123181A1 (en) * | 2003-10-08 | 2005-06-09 | Philip Freund | Automated microscope slide tissue sample mapping and image acquisition |
US20140233797A1 (en) * | 2008-11-14 | 2014-08-21 | The Scripps Research Institute | Image analysis platform for identifying artifacts in samples and laboratory consumables |
US20160042511A1 (en) * | 2013-03-15 | 2016-02-11 | Ventana Medical Systems, Inc. | Tissue Object-Based Machine Learning System for Automated Scoring of Digital Whole Slides |
WO2015052128A1 (en) * | 2013-10-07 | 2015-04-16 | Ventana Medical Systems, Inc. | Systems and methods for comprehensive multi-assay tissue analysis |
Non-Patent Citations (1)
Title |
---|
See also references of EP3424019A4 * |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10937162B2 (en) | 2016-03-01 | 2021-03-02 | Ventana Medical Systems, Inc. | Image analysis algorithms using control slides |
US11620751B2 (en) | 2016-03-01 | 2023-04-04 | Ventana Medical Systems, Inc. | Image analysis algorithms using control slides |
Also Published As
Publication number | Publication date |
---|---|
EP3424019A4 (en) | 2019-10-23 |
CA3012360A1 (en) | 2017-09-08 |
EP3424019A1 (en) | 2019-01-09 |
CA3012360C (en) | 2021-12-14 |
US10937162B2 (en) | 2021-03-02 |
US11620751B2 (en) | 2023-04-04 |
US20210150716A1 (en) | 2021-05-20 |
AU2017226252A1 (en) | 2018-07-26 |
US20190026895A1 (en) | 2019-01-24 |
JP6960935B2 (en) | 2021-11-05 |
JP2019513255A (en) | 2019-05-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11620751B2 (en) | Image analysis algorithms using control slides | |
US11600087B2 (en) | Convolutional neural networks for locating objects of interest in images of biological samples | |
CN107111874B (en) | Systems and methods for co-expression analysis | |
JP7228688B2 (en) | Image enhancement enabling improved nuclear detection and segmentation | |
JP6604960B2 (en) | Medical image analysis to identify biomarker positive tumor cells | |
US10388014B2 (en) | Vessel analysis in multiplexed images | |
RU2690224C2 (en) | Image processing and analysis device | |
EP3729371B1 (en) | System and method for generating selective stain segmentation images for cell types of interest | |
JP7214756B2 (en) | Quantification of signal in stain aggregates | |
CN111095352B (en) | Automated method and system for detecting cells in stained sample images | |
JP7011067B2 (en) | Systems and methods for classifying cells in tissue images based on membrane characteristics | |
WO2023107844A1 (en) | Label-free virtual immunohistochemical staining of tissue using deep learning | |
Shimoni et al. | Normalized polarization ratios for the analysis of cell polarity |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 3012360 Country of ref document: CA |
|
ENP | Entry into the national phase |
Ref document number: 2017226252 Country of ref document: AU Date of ref document: 20170301 Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2018545836 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2017760737 Country of ref document: EP |
|
ENP | Entry into the national phase |
Ref document number: 2017760737 Country of ref document: EP Effective date: 20181001 |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 17760737 Country of ref document: EP Kind code of ref document: A1 |