WO2025245280A1 - Use of color lookup tables to generate singleplex pathology images from multiplex images - Google Patents

Abstract

Some embodiments relate to techniques for efficiently generating a synthetic singleplex image. A multiplex image is accessed that includes red, green, and blue (RGB) pixel values representing a tissue section stained with two or more biomarkers. A color lookup table (cLUT) is accessed that maps RGB pixel values from the multiplex image to RGB pixel values of a synthetic singleplex image corresponding to a selected biomarker. For each pixel in the multiplex image, an index is determined based on its RGB value and retrieving from the cLUT a corresponding output RGB value. A synthetic singleplex image is generated by assigning, to each pixel location, the corresponding output RGB value retrieved from the cLUT.

Description

USE OF COLOR LOOKUP TABLES TO GENERATE

SINGLEPLEX PATHOLOGY IMAGES FROM MULTIPLEX IMAGES

CROSS-REFERENCE OF RELATED APPLICATIONS

[0001] This application claims the benefit of and the priority to U.S. Provisional Patent Application 63/650,630, filed on May 22, 2024, which is hereby incorporated by reference in its entirety for all purposes.

FIELD

[0002] The present disclosure generally relates systems, methods and computer-readable media that use of color lookup tables (cLUTs) in digital pathology to generate a singleplex biomarker image from a multiplex image.

BACKGROUND

[0003] In digital pathology, stain unmixing, also known as color deconvolution or color unmixing, is a widely used technique for estimating the intensity of various stains in digitized tissue slide images. This process assists in the identification, classification, and analysis of cellular structures by separating contributions of different stains. For singleplex slides, which typically include one biomarker stain and a counterstain, the output of traditional unmixing methods consists of a separate intensity image corresponding to a given biomarker stain and a corresponding image representing intensity of a counterstain.

[0004] With the growing use of multiplex immunohistochemistry (IHC) assays, which involve the application of multiple biomarkers to a single tissue section, there is an increasing need for effective methods to process and interpret multiplex images. A key challenge is the generation of synthetic singleplex images from a multiplex image. These synthetic images enable more accurate detection and classification of cell types, support phenotype-level analysis, and facilitate the creation of ground truth annotations for training machine learning models used in clinical or research workflows. [0005] Synthetic singleplex images are a combination of the unmixed counterstain intensity image and one of the multiple unmixed biomarker stain intensity images. This combination is carried out using the pure stain color vector array to create optical density images, which are then multiplied together and converted to red, green, blue (RGB). Examples of this combination of stain intensity images are widely known.

[0006] Accurate generation of synthetic singleplex images is particularly important for downstream applications, such as cell scoring, phenotype quantification, and annotation by human pathologists. However, pathologists often report a lack of confidence in scoring or annotating directly from duplex or multiplex images, due to visual complexity and overlapping stain patterns. This limitation can hinder both diagnostic accuracy and the development of automated tools based on human-annotated data.

[0007] Further, as clinical laboratories seek to reduce tissue consumption and streamline diagnostic workflows, there is a growing demand for technologies that enable reliable multiplex imaging while reducing the number of physical slide sections required. These needs are compounded by objectives to improve computational efficiency and compatibility with real-time image rendering on digital pathology platforms.

[0008] Accordingly, there remains a need for improved techniques to derive high-quality, diagnostically useful singleplex biomarker images from multiplex pathology images in a manner that is accurate, efficient, and scalable for clinical use. In particular, there is a need for methods that can generate and display such singleplex images rapidly and reliably, while minimizing tissue usage and reducing the risk of error in image derivation. Existing approaches often require extensive tissue sampling or involve complex workflows that hinder routine clinical implementation. Thus, there is a demand for solutions that streamline this process to enable fast, reproducible, and tissue-conserving generation of clinically meaningful singleplex biomarker images from multiplex data.

SUMMARY

[0009] In some embodiments, a method is provided that includes: accessing a multiplex image comprising red, green, and blue (RGB) pixel values representing a tissue section stained with two or more biomarkers; accessing a color lookup table (cLUT) that maps RGB pixel values from the multiplex image to RGB pixel values of a synthetic singleplex image corresponding to a selected biomarker, the cLUT having been precomputed from a plurality of aligned multiplex and singleplex image pairs;, wherein the cLUT was generated by: accessing a plurality of aligned image pairs, each comprising a multiplex image and a synthetic singleplex image for a specific biomarker; computing, for each unique RGB value in the multiplex images, a representative RGB value based on corresponding pixels in the synthetic singleplex images; and storing the mapping from each multiplex RGB value to the computed representative RGB value in the cLUT. For each pixel in the multiplex image, an index is determined based on its RGB value and retrieving from the cLUT a corresponding output RGB value. A synthetic singleplex image is generated by assigning, to each pixel location, the corresponding output RGB value retrieved from the cLUT; outputting the synthetic singleplex image for rendering or downstream analysis.

[0010] Computing the representative RGB value may include computing a statistical measure selected from the group consisting of: a mean, a median, or a mode of RGB values from the corresponding pixels in the synthetic singleplex images.

[0011] The RGB values of the multiplex image may be binned into discrete intervals prior to computing the representative RGB values, such that each RGB bin maps to a common representative RGB value.

[0012] The multiplex and synthetic singleplex images may be transformed into a standardized color space prior to generating the cLUT, the standardized color space comprising sRGB obtained using an ICC color profile.

[0013] The method may further include applying two or more distinct cLUTs to the same multiplex image, each cLUT corresponding to a different biomarker, to generate multiple synthetic singleplex images; and evaluation of the synthetic singleplex image includes comparing the image to a reference image using one or more quantitative image quality metrics selected from the group consisting of: Structural Similarity Index Measure (SSIM), mean squared error (MSE), and Delta-E color difference. [0014] The cLUT may be configured as a multidimensional array indexed by RGB values, and interpolation is used to compute output RGB values for input values that do not have an exact match in the cLUT.

[0015] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0016] In some embodiments, a computer-program product tangibly embodied in a non- transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform part or all of one or more methods or processes disclosed herein.

[0017] In some embodiments, a system is provided that includes one or more means to perform part or all of one or more methods or processes disclosed herein.

[0018] The terms and expressions which have been employed are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention claimed. Thus, it should be understood that although the present invention as claimed has been specifically disclosed by embodiments and optional features, modification and variation of the concepts herein disclosed may be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this invention as defined by the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0020] The present disclosure is described in conjunction with the appended figures: [0021] FIG. 1 illustrates a workflow for generating synthetic singleplex images based on multiplex images.

[0022] FIG. 2 illustrates a workflow for generating for generating synthetic singlex images based on input specimens.

[0023] FIG. 3 illustrates a typical digital scanner red, green, blue filter transmission performance according to an embodiment of the present disclosure.

[0024] FIG. 4 illustrates a) Chromaticity diagram showing the full RGB gamut with the triangular boundary of the sRGB color gamut superimposed. The dot in the middle is called the white point and b) Chromaticity diagram showing the triangular boundary of the Adobe Wide Gamut color space according to an embodiment of the present disclosure.

[0025] FIG. 5 illustrates, on the left is a digital slide scanner raw image (Dabsyl PDL1, Tamra EGFR, Green cMET) and, on the right, is the Tamra singleplex image generated from that according to an embodiment of the present disclosure.

[0026] FIG. 6 illustrates the images in FIG. 5, transformed to sRGB color space according to an embodiment of the present disclosure.

[0027] FIG. 7 illustrates the triplex image pixels provide the index into the cLUT, and the synthetic singleplex image pixels provide the value of the cLUT at that index according to embodiments of the present disclosure.

[0028] FIG. 8 illustrates a different way of showing the concept in Figure 5 wherein the value of the RGB vector assigned to the index location in the cLUT is determined by the RGB values of the synthetic singleplex image according to another embodiment of the present disclosure.

[0029] FIG. 9 illustrates a cLUT table, showing the numeric values according to an embodiment of the present disclosure.

[0030] FIG. 10 illustrates a Dabsyl synthetic singleplex image from the test set on the left, with the cLUT output on the right (CD8(green)-Lag3(tamra)-PDLl (dabsyl)) according to an embodiment of the present disclosure. [0031] FIG. 11 illustrates a Tamra synthetic singleplex image from the test set on the left, with the cLUT output on the right (CD8(green)-Lag3(tamra)-PDLl(dabsyl)), Lag3 is faintly expressed in this case according to an embodiment of the present disclosure.

[0032] FIG. 12 illustrates a Green synthetic singleplex image from the test set on the left, with the cLUT output on the right (CD8(green)-Lag3(tamra)-PDLl(dabsyl)) according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

[0033] In some embodiments, color lookup tables (cLUTs) may be employed as lightweight and deterministic mechanisms for translating multiplex-stained image data into single-biomarker views. As illustrated in FIG. 1, a multiplex image 102 is input into a synthetic singleplex image detection module 104, which utilizes one or more precomputed color lookup tables (cLUTs) 106. In some instances, the cLUTs are generated by associating pixel-level color values from multiplex image data with corresponding pixel values from known or synthesized singleplex images. The associations can be derived from matched image pairs, including aligned images from adjacent tissue sections or computationally unmixed data. The computationally unmixed data may be generated using linear deconvolution, a machine-learning technique (e.g., a deeplearning technique), or an unmixing technique.

[0034] The input and target images can be split into a training and testing set (i.e., machine learning convention). With the pairs of images in the training set, there can be multiple ways to learn the cLUT. One example is as follows: for each different RGB value in the input image set [r, g, b], find out the pixel values of the target images, and then average the target image pixel values [r_mean_tgt, g_mean_tgt, b_mean_tgt], thus establishing the look-up relationship between the two: input of [r, g, b] mapped to [r mean tgt, g mean tgt, b mean tgt]. The resulting cLUT encodes transformation mappings that enable accurate synthetic singleplex generation.

[0035] In some instances, evaluation of the developed cLUTs may be performed using the testing set and/or additional unseen images from the same type of staining as the input images. Evaluation may comprise both qualitative and quantitative assessments. Qualitative assessment may involve visual comparison of the cLUT-transformed outputs versus those produced by alternative transformation methods, such as linear deconvolution. Quantitative assessment may include metric-based comparisons using image similarity or error metrics, such as Structural Similarity Index Measure (SSIM), pixel-wise mean squared error (MSE), Delta-E, and similar metrics.

[0036] Once constructed, a cLUT 106 can support transformation of previously unseen multiplex images 102 into one or more synthetic singleplex images 108a-n, each corresponding to a specific biomarker or chromogen. The synthetic image generation is performed by the detection module 104 and may be executed on conventional computing hardware, facilitating integration with digital pathology platforms, visualization tools, and clinical decision-support systems. These transformations do not require physical re-staining, sectioning, or complex inference models.

[0037] In certain embodiments, construction of a cLUT 106 may involve converting source image data into standardized color spaces, such as sRGB, using scanner-specific calibration profiles (e.g., ICC profiles). Multiplex and singleplex image pixels may be encoded as RGB vectors to ensure color-space consistency. Within the detection module 104, a multiplex RGB vector from image 102 may be used as an index into the cLUT 106, returning an RGB value representative of a target biomarker’s synthetic singleplex visualization.

[0038] The cLUT 106 may be configured as a multidimensional data structure containing color mappings, interpolation coefficients, normalization values, or additional transformation metadata. Depending on fidelity requirements, the cLUT may be densely or sparsely populated, with interpolation methods — such as linear, bilinear, trilinear, or tetrahedral — used to estimate intermediate values.

[0039] In some embodiments, the cLUT mapping need not operate at the resolution of individual R, G, B values. Instead, binned RGB ranges may be employed to reduce complexity. For example, input pixel values may be grouped into bins such that R, G, and B values are discretized in increments (e.g., every four values), whereby all pixels within the range R = 0-3, G = 0-3, B = 0-3 are mapped to a common RGB output value.

[0040] A cLUT can be initialized to include or have values close to or equal to 255 to represent glass slide (non-tissue regions in a whole slide image) and fill in the pixel mappings during cLUT generation phase. However, due to the fact that pixel values in one triplex or duplex assay usually covers a small subspace in the color space, when the generated cLUT is deployed, unseen pixels of the same assay may sometimes distribute outside of the color range covered by training pixels. In such case, there may be no accurate mapping for these out of distribution pixels, leading to visual artifacts. Thus, known color vectors (principle colors) from the same assay can be used to initialize the cLUT and/or to generate a range of colors to approximate possible color regimes of the multiplex images and corresponding synthetic images. For example, for a duplex, PDL1-Tamra/CK7-Dabsyl, one can identify color vectors of Tamra, Dabysl and Hematoxylin, digitally combine a range of stain intensities of Tamra and Hematoxylin to generate synthetic PDL1 pixels, and generate synthetic CK7 pixels and Synthetic duplex pixels in a similar matter.

[0041] In this way, digitally blended colors are obtained that can be used to initialize a cLUT before filling in actual data. Since the range of intensities used for generating the synthetic cLUT an be tuned, distribution visual artifacts can be reduced or avoided. In addition, interpolation based post-processing may be performed to improve the extent to which transitions from the mapped pixels are smooth in the color space and/or to avoid the case where the initialized pixel mapping deviate from the mapping obtained from real data. For example, in some cases, the cLUT index of 1000 may be a synthetic mapping, but index of 1001 may be a real mapping; if the differences between the target pixels are considerably large, one can interpolate the colors to reduce such gaps.

[0042] Once a cLUT 106 is available, it may be used by the synthetic singleplex image detection module 104 to perform real-time transformation of multiplex input image 102. Each pixel is mapped through the cLUT to generate output values corresponding to a selected biomarker. In some embodiments, the module 104 can generate multiple singleplex images 108a-n concurrently from a single multiplex input, with each output image reflecting a distinct chromogen channel. These synthetic images can be rendered locally or displayed through networked platforms or pathology viewers.

[0043] Although a cLUT typically defines a deterministic mapping for a given input value, variations or stochasticity may be introduced by applying multiple cLUTs for the same input image. For instance, the detection module 104 may apply several distinct cLUTs either sequentially or in parallel, in a predetermined or randomized order, to produce a variety of synthetic outputs for comparative analysis or ensemble-based downstream tasks.

[0044] The mapping in the cLUTs can be established by at least the following three methods: (1) manually identify target colors to map to for each pixel in the input image and generate a corresponding cLUT; (2) semi-manually identify the target colors to map to with, for examples a specific set of manually defined mapping rules; (3) automatically identify the target colors to map with, for example, by learning the mapping from data (e.g. a set of specific images).

[0045] The techniques disclosed herein offer advantages such as low computational complexity, low latency, and scalability across large pathology datasets. Unlike model-based inference systems, which require runtime execution of trained models, the table-driven architecture — centered on fixed cLUTs 106 and detection module 104 — provides consistent performance and visual fidelity across various diagnostic use cases.

[0046] Transformation errors in this pipeline may arise from various sources, including spectral overlap in RGB sensors, chromogen interactions, or inconsistencies in tissue preparation and digitization. For example, RGB scanners such as the Ventana DP 200 may exhibit channel cross-sensitivity, complicating stain separation and transformation accuracy.

[0047] Additional challenges may stem from staining chemistry variability or sample heterogeneity. Digitization processes — especially conversions from high-bit-depth OD imagery to lower bit-depth RGB formats — can introduce quantization artifacts. Controlled blending studies suggest that transformation accuracy is highest with higher bit-depth representations (e.g., 32-bit RGB), whereas lower bit-depth formats may degrade visual fidelity.

[0048] Despite such limitations, cLUTs 106 derived from representative training data can generalize across tissue types and scanner configurations. Advanced sampling techniques, such as adaptive RGB point selection and color normalization, may be employed during cLUT construction to enhance robustness.

[0049] Multiple cLUTs 106 may be employed concurrently or sequentially within the detection module 104, with each table optimized for a specific biomarker or chromogen. Applying these cLUTs to a single multiplex image 102 allows the system to generate multiple corresponding synthetic outputs 108a-n, which may then be used for analysis, archival, or diagnostic synthesis.

[0050] This approach enables generation of synthetic singleplex imagery without consuming additional tissue or requiring multiple staining cycles. As shown in FIG. 1, the process uses a single multiplex image 102 and one or more cLUTs 106 to efficiently yield biomarker-specific outputs 108a-n. This is particularly useful in scenarios with limited biological material or high assay demand.

[0051] The synthetic singleplex image detection functionality represented in module 104 may be integrated into existing digital pathology tools. Supported environments may include commercial software (e.g., HALO by Indica Labs), browser-based viewers (e.g., uPathX, dPath), and computational toolkits (e.g., Python or MATLAB). While specific implementations may vary, the underlying transformation method remains consistent and platform-agnostic.

[0052] In some embodiments, the developed cLUTs can be deployed to software such as, for example, whole-slide image viewers (e.g., for real-time and on-the-fly color transformation) and/or for real time stain unmixing or stain recoloring. Such software may have user interfaces to provide one or multiple mapping options for users to choose from, where the user interface may or may not have other functionalities besides cLUT based color mapping. Such software may be integrated into a larger scope analysis pipeline, where the cLUT based color mapping, e.g. stain unmixing, can be in one or multiple of the steps in an analysis pipeline. Such software may be web or non-web-based and may be developed for different operating systems and hardware setups.

[0053] Certain embodiments disclosed herein facilitate efficient, deterministic transformation of multiplex-stained whole-slide images 102 into clinically useful synthetic singleplex outputs 108a-n. This is achieved via a modular architecture incorporating synthetic image detection 104 and transformation logic using one or more cLUTs 106. The approach is adaptable to a wide range of hardware and software configurations and supports evolving clinical and research needs.

[0001] FIG. 2 shows an exemplary network of synthetic singleplex image generation in accordance with some aspects of the present disclosure. The multiplex image 102 may be generated by an image generation system 202. A fixation/embedding system 204 fixes and/or embeds a tissue sample (e.g., a liquid fixing agent, such as formaldehyde solution) and/or an embedding substance (e.g., a historical wax, such as paraffin wax and/or one or more resins, such as styrene or polyethylene). The tissue sample may be fixed by exposing it to a fixating agent for a predefined period of time (e.g., at least 3 hours) and by then dehydrating the tissue sample (e.g., via exposure to an ethanol solution and/or a clearing intermediate agent). The embedding substance can infiltrate the tissue sample when it is in liquid state (e.g., when heated). A tissue slicer 206 then slices the fixed and/or embedded tissue sample (e.g., a sample of a tumor) to obtain a series of sections, with each section having a thickness of, for example, 4-5 microns. Such sectioning can be performed by first chilling the sample and then slicing the sample in a warm water bath. The tissue can be sliced using (for example) a vibratome or compresstome. [0002] Because the tissue sections and the cells within them are virtually transparent, preparation of the slides typically includes staining (e.g., automatically staining) the tissue sections to render relevant structures more visible. In some instances, the staining is performed manually. In some instances, the staining is performed semi-automatically or automatically using a staining system 208. The staining can include exposing an individual section of the tissue to one or more different stains (e.g., consecutively, or concurrently) to express different characteristics of the tissue. For example, each section may be exposed to a predefined volume of a staining agent for a predefined period of time. The staining agent can include (for example) an RNA probe, protein probe (e.g., nuclear-protein probe or cytoplasm-protein probe), an immunohistochemistry stain, a probe for a secreted substance, etc. In some instances, the staining agent is one that stains for KAPPA mRNA or LAMBDA mRNA.

[0003] One exemplary type of tissue staining is histochemical staining, which uses one or more chemical dyes (e.g., acidic dyes, basic dyes) to stain tissue structures. Histochemical staining may be used to indicate general aspects of tissue morphology and/or cell microanatomy (e.g., to distinguish cell nuclei from cytoplasm, to indicate lipid droplets, etc.). One example of a histochemical stain is hematoxylin and eosin (H&E). Other examples of histochemical stains include trichrome stains (e.g., Masson's Trichrome), Periodic Acid-Schiff (PAS), silver stains, and iron stains. The molecular weight of a histochemical staining reagent (e g., dye) is typically about 500 kilodaltons (kD) or less, although some histochemical staining reagents (e.g., Alcian Blue, phosphomolybdic acid (PMA)) may have molecular weights of up to two or three thousand kD. One case of a high-molecular-weight histochemical staining reagent is alpha-amylase (about 55 kD), which may be used to indicate glycogen.

[0004] Another type of tissue staining is immunohistochemistry (IHC, also called "immunostaining"), which uses a primary antibody that binds specifically to the target antigen of interest (biomarker). IHC may be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a label (e.g., a chromophore or fluorophore). In indirect IHC, the primary antibody is first bound to the target antigen, and then a secondary antibody that is conjugated with a label (e.g., a chromophore or fluorophore) is bound to the primary antibody. The molecular weights of IHC reagents are much higher than those of histochemical staining reagents, as the antibodies have molecular weights of about 150 kD or more.

[0054] In some instances, the staining includes using one, more or all of: Dabsyl PDL1, Tamra EGFR, Green cMET. The staining may additionally or alternatively include staining that includes or is associated with Hematoxylin & Eosin staining, singleplex or multiple immunohistochemistry, dark field fluorescence, multi -spectral images, and the like.

[0001] The sections may then be individually mounted on corresponding slides, where an imaging system 210 may scan the entire area of each tissue section e.g., using a digital camera, sensor or scanner that moves systematically across the slide.

[0055] The multiplex image 102 can then be processed by the synthetic singleplex image detection module 104, which can pull data from the cLUT(s) 106 (generated in accordance with part or all of one or more techniques disclosed herein). For example, a query may be generated that corresponds to each of one or more pixels, each of one or more device profiles (e.g., corresponding to a device that captured the multiplex image), and/or each of one or more signals of interest (e.g., chromogen or dye) for a target output synthetic singleplex image. The one or more synthetic singleplex images 108a-n can be generated by the synthetic singleplex image detection module 104 based on a response to the query. The one or more synthetic singleplex images can be output, transmitted, stored and/or availed for further processing (e.g., to generate signal-specific statistics), user review, etc.

EXAMPLES [0056] The experiments described below describe an exemplary process of creating the cLUTs for synthetic singleplex images. Overall, the three following tasks were separately performed to generate synthetic singleplex images for pathologist and assay team review:

• Adaption of NMF methods, using constraints in color space;

• cycleGAN; and

• Customized GAN method.

[0057] Representative slide images were selected to encompass a range of biomarker expression levels, such as high-high-low and high-low-high combinations. For each biomarker, various slide types were made available in addition to the multiplex (triplex) slides, including duplex, singleplex, pure-stain, and hematoxylin-only slides. For most cases, hematoxylin and eosin (H&E) slides were also included. The chromogens employed in the assays were Dabsyl (yellow), Tamra (magenta), Green (green), Teal (teal), and Hematoxylin (blue).

[0058] A preliminary study comparing Green and Teal chromogens was conducted, and Green was selected as being more suitable for stain separation based on ease of unmixing.

[0059] For each slide, ten (10) fields of view (FOVs) were selected as representative of the relevant expression patterns. These high-resolution FOVs (typically less than 1 mm x 1 mm in size) were manually matched across all corresponding slide types, enabling a direct comparison between the unmixed triplex FOVs and adjacent-cut singleplex FOVs.

[0060] For visual evaluation, “International Color Consortium (ICC)-managed” images were generated, if needed, for visual evaluation, and for the construction of the cLUTs. These are images that were transformed from the digital slide scanner camera images (“raw images”) to images that represent on a standard computer monitor (as closely as possible) the slide’s colors as measured by a calibrated photospectrometer. Depending on how a particular microscope’s settings are adjusted, this may or may not be similar to the colors seen by the human eye in a microscope viewing of the same slide.

[0061] Results using NMF for duplex unmixing were pursued at great length, but ultimately proved to be inadequate for deconvolving four colors on the same pixel with only three input channels. Various methods were tried, using boundaries (constraints) to separate the pixels into categories in color space so that only three colors would be involved. In this way, part of the deconvolution challenge could be effectively solved with NMF.

[0062] However, there were inevitably areas on most of the images with high stain intensities for one or more biomarkers. In these areas, the pixels were often nearly gray in color. Using NMF to deconvolve those pixels yielded stain intensities that were quite frequently not realistic in their morphology. For example, in ER-PR-Her2, where PR was stained with purple, the Her2 membrane would sometimes be present in the PR stain intensity images. This was unacceptable, and short of a “membrane” or “nucleus” finding algorithm to help out, was unresolvable with NMF or any other purely mathematical approach. For that assay, the proximal cause was that the purple (for PR, a nuclear biomarker), if very intense, was usually faintly present in the cytoplasm and membrane. The reasons for this are out of scope for the present disclosure. Similar problems may occurred in other assay models.

[0063] Regarding the color space used for unmixing, during the present disclosure, it was performed with the digital slide scanner “raw” images, that is, the images captured by the digital slide scanner without applying the ICC color profile. This may have the disadvantage, in principle, of being device-dependent. At the current time, the ICC color profiles of all digital slide scanner instruments used in the present disclosure were identical, but this could change in the future. The advantage was that by using the raw images, the entire gamut of colors (400 nm to 750 nm), see FIG. 3, capturable by the digital slide scanner optics could be taken advantage of. A gamut is typically the range of colors that a device can display or print. A transform to standard Red Green Blue (sRGB space) can compress the RGB gamut and unacceptably reduce the amount of information available for determining stain intensities. Unfortunately, most commercially standard computer monitors are only capable of displaying a sRGB gamut.

[0064] This is illustrated in FIG. 4a, which illustrates the standard RGB chromaticity diagram. All of the colors in the semi-elliptical shape (the full RGB gamut) can be accessible with the digital slide scanner, at some level of sensitivity. The triangle indicates the limits imposed by sRGB.

[0065] To transform the full gamut to sRGB, two steps are typically required. The first is a transformation to a device independent profile connection space (PCS) using the ICC profile; in the case of the digital slide scanner used, the PCS is CIELAB color space, an industry wide standard. No compression or loss of data may be incurred in this step. However, to properly display the image on commercially standard computer monitors, the second step converts the CIELAB color space values to the sRGB gamut space. Colors outside the triangle in FIG. 4a are mapped to the nearest edge of the sRGB gamut space. As is evident from the figure, green and greenish colors commonly get highly compressed, as well as other colors.

[0066] A transform to one of the largest gamuts, Adobe Wide Gamut, is a compromise for over 20% of the full gamut color space, but might be an acceptable color space for unmixing. FIG. 4b shows that situation where Adobe Wide Gamut was used.

[0067] Color unmixing might also be conducted in CIELAB space. For CIELAB, the resulting stain intensities would be converted back to RGB and then to RGB OD space in order to be combined to create the synthetic singleplex images needed in the applications for the present disclosure. This approach may thus also be used to provide device-independent estimates of phenotype expression.

[0068] For the above reasons, all stain unmixing operations were carried out with digital slide scanner raw images. In order for the deliverables to a digital pathology platform that were created to be device-independent, the delivered cLUTs were based on images that were transformed from digital slide scanner raw color space to sRGB color space, using the appropriate digital slide scanner ICC profile (ICC managed).

[0069] These cLUTs delivered for use by the digital pathology platform included:

• cLUT to convert a duplex image (Dabysl-Tamra) to a Dabsyl singleplex image;

• cLUT to convert a duplex image (Dabysl-Tamra) to a Tamra singleplex image;

• cLUT to convert a triplex image (Dabysl-Tamra-Green) to a Dabsyl singleplex image;

• cLUT to convert a triplex image (Daniel-Tamara-Green) to a Tamra singleplex image;

• cLUT to convert a triplex image (Daniel-Tamara-Green) to a Green singleplex image.

[0070] Initially, to generate each of these specific cLUTs, hundreds of synthetic singleplex images were generated for each of the biomarkers from the digital slide scanner multiplex (duplex or triplex) image. The range of biomarker expression levels present in the cases available was somewhat limited; ideally a wide range of combinations of expression levels would be available for a more commercially viable effort in order to avoid incorrect colors being displayed by a digital pathology platform.

[0071] In practice, typically between a few hundred and several hundred pairs of multiplex- synthetic-singleplex images were used that were about 1000x1000 pixels (20x magnification, 0.50 microns/pixel). Each pair consisted of a duplex or triplex image and one of the synthetic singleplex images. The synthetic singleplex images were created using a variety of methods from the digital slide scanner raw images.

[0072] The type of method does not matter as long as the generated synthetic singleplex images can be a visual representation of the biomarker staining that can be equivalent to an adjacent cut singleplex image, as judged by a pathologist. It is worth repeating at this point that the entire process of color unmixing/deconvolution tends to be error-prone, and even a “strictly mathematical” method can still yield irretrievable errors in duplex unmixing due to the sources described earlier, and can be completely unusable for triplex images. Therefore, the only “reliable” ground truth can be an image from an adjacent cut slide, and tissue variations can - even then - easily cause surprisingly large differences in biomarker expression from slide to slide.

[0073] In addition, for triplex, there can be variations in synthetic singleplex images that can be expected from the nature of the generative adversarial networks used to produce them. Such errors can be constrained by the loss functions used in training the networks, but that does not ensure absolute accuracy. Therefore, it can be necessary to assemble a large input data set for evaluation of the synthetic singleplex images, and rely on pathologists to confirm that the clinical score for any given synthetic singleplex image is not significantly different from that of the adjacent cut singleplex image (unless of course there is an instrumental glitch of some kind).

[0074] FIGs. 5 and 6 show examples of a triplex-synthetic-singleplex pair in both the raw and sRGB color spaces. FIG. 5 illustrates, on the left, a digital slide scanner raw image (Dabsyl PDL1, Tamra EGFR, Green cMET) and, on the right, is the Tamra singleplex image generated from that image on the left. FIG. 6 illustrates the images in FIG. 5, transformed into sRGB color space. [0075] Before constructing the cLUT for a specific biomarker, the multiplex images and the synthetic singleplex images were transformed (corrected) to sRGB, using the digital slide scanner ICC profile (see FIG. 6).

[0076] For each pixel location, an RGB vector (array of three integers) in the multiplex image was identified, and a (generally different) RGB vector in the corresponding synthetic singleplex image was generated. Using the RGB vector of the multiplex image as the index to the cLUT, the value of cLUT at the index was assigned to the RGB vector in the synthetic image. With a large set of images covering a broad range of biomarker expressions, a reasonably complete cLUT was assembled.

[0077] FIG. 7 illustrates this concept. FIG. 7 illustrates how the triplex image pixels provide the index into the cLUT, and how the synthetic singleplex image pixels provide the value of the cLUT at that index.

[0078] FIG. 8 illustrates a cLUT indexing example. In other words, FIG. 8 illustrates a different way of showing the concept in FIG. 7, where the value of the RGB vector assigned to the index location in the cLUT can be determined by the RGB values of the synthetic singleplex image.

[0079] FIG. 9 illustrates an example of a cLUT table, showing the numeric values. FIG. 9 illustrates just a small portion of the numbers in one of the cLUTs developed in this work. In principle, the most accurate cLUT would result from using 256x256x256 bins in RGB space because the raw images are captured by a camera with 8-bit resolution per channel. However, utilization of such a large array for the digital slide scanner ICC profile can yield inadequate performance (i.e., scanning time that is too long), so the ICC profile is stored in the digital slide scanner scanned images as a 67x67x67 array. Therefore, the cLUT array size utilized for creating synthetic images was also tested with this coarser binning, specifically 64x64x64.

[0080] No visual differences could be observed between synthetic images created with a 256x256x256 cLUT and those created with a 64x64x64 cLUT.

[0081] Over the range of synthetic singleplex images available, there were small variations in the synthetic image RGB color vectors matched to any particular multiplex RGB bin. Hence, the median of the set of synthetic image color vectors was used as the value for that cLUT location. This represented averaging of the synthetic singleplex images, which can help remove some of the variations caused by the process of making them.

[0082] Several dozen cLUT generated image sets were visually compared to the test set of synthetic singleplex images and no significant differences were observed. Examples of such comparisons are shown in FIGs. 10-12

[0083] FIG. 10 illustrates a Dabsyl synthetic singleplex image from the test set on the left, with the cLUT output on the right (CD8(green)-Lag3(tamra)-PDLl (dabsyl)).

[0084] FIG. 11 illustrates a Tamra synthetic singleplex image from the test set on the left, with the cLUT output on the right (CD8(green)-Lag3(tamra)-PDLl (dabsyl)), Lag3 is faintly expressed in this case.

[0085] FIG. 12 illustrates a Green synthetic singleplex image from the test set on the left, with the cLUT output on the right (CD8(green)-Lag3(tamra)-PDLl (dabsyl)).

[0086] cLUTs of both the duplex to singleplex and triplex to singleplex type were then used to generate synthetic singleplex images automatically in display software of the digital pathology platform. To avoid artifacts, RGB index values were offset (by 0.5) because display software such as OpenGL requires an index pointing to the center of the RGB bin.

[0087] The cLUTs were created for transformation of multiplex (both duplex and triplex) immunohistochemistry (IHC) images to synthetic singleplex images. The synthetic images utilized for training were compared to adjacent cut singleplex images by pathologists, and the pathologists reported equivalent scoring outcomes. The cLUT-created synthetic images were checked against the synthetic images utilized for training, and found to be equivalent. The cLUTs were delivered to a digital pathology platform and enabled that platform to generate synthetic images in real time that were visually equivalent.

[0088] It is recognized that the process of generating synthetic singleplex images from multiplex images may not be error free: every resulting pixel is an estimate or approximation of the true biomarker expression. Therefore, once a detection kit is locked, and applied to a specific biomarker set and tissue type, it may be advantageous for pathologists to evaluate that the cLUTs generate images with scores equivalent to those of adjacent cut images. [0089] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods and/or part or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer-program product tangibly embodied in a non-transitory machine- readable storage medium, including instructions configured to cause one or more data processors to perform part or all of one or more methods and/or part or all of one or more processes disclosed herein.

[0090] The terms and expressions which have been employed are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention claimed. Thus, it should be understood that although the present invention as claimed has been specifically disclosed by embodiments and optional features, modification and variation of the concepts herein disclosed may be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this invention as defined by the appended claims.

[0091] The present description provides preferred exemplary embodiments only, and is not intended to limit the scope, applicability or configuration of the disclosure. Rather, the present description of the preferred exemplary embodiments will provide those skilled in the art with an enabling description for implementing various embodiments. It is understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0092] Specific details are given in the present description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Claims

CLAIMS What is claimed:

1. A method comprising: accessing a multiplex image comprising red, green, and blue (RGB) pixel values representing a tissue section stained with two or more biomarkers; accessing a color lookup table (cLUT) that maps RGB pixel values from the multiplex image to RGB pixel values of a synthetic singleplex image corresponding to a selected biomarker, the cLUT having been precomputed from a plurality of aligned multiplex and singleplex image pairs;, wherein the cLUT was generated by: accessing a plurality of aligned image pairs, each comprising a multiplex image and a synthetic singleplex image for a specific biomarker; computing, for each unique RGB value in the multiplex images, a representative RGB value based on corresponding pixels in the synthetic singleplex images; and storing the mapping from each multiplex RGB value to the computed representative RGB value in the cLUT; for each pixel in the multiplex image, determining an index based on its RGB value and retrieving from the cLUT a corresponding output RGB value; generating a synthetic singleplex image by assigning, to each pixel location, the corresponding output RGB value retrieved from the cLUT; and outputting the synthetic singleplex image for rendering or downstream analysis.

2. The method of claim 1, wherein computing the representative RGB value comprises computing a statistical measure selected from the group consisting of: a mean, a median, or a mode of RGB values from the corresponding pixels in the synthetic singleplex images.

3. The method of claim 1, wherein the RGB values of the multiplex image are binned into discrete intervals prior to computing the representative RGB values, such that each RGB bin maps to a common representative RGB value.

4. The method of claim 1, wherein the multiplex and synthetic singleplex images are transformed into a standardized color space prior to generating the cLUT, the standardized color space comprising sRGB obtained using an ICC color profde.

5. The method of claim 1, further comprising applying two or more distinct cLUTs to the same multiplex image, each cLUT corresponding to a different biomarker, to generate multiple synthetic singleplex images.

6. The method of claim 1, wherein evaluation of the synthetic singleplex image includes comparing the image to a reference image using one or more quantitative image quality metrics selected from the group consisting of: Structural Similarity Index Measure (SSIM), mean squared error (MSE), and Delta-E color difference.

7. The method of claim 1, wherein the cLUT is configured as a multidimensional array indexed by RGB values, and interpolation is used to compute output RGB values for input values that do not have an exact match in the cLUT.

8. A system comprising: one or more processors; one or more non-transitory computer-readable media storing instructions, which, when executed by the system, cause the system to perform a set of operations including: accessing a multiplex image comprising red, green, and blue (RGB) pixel values representing a tissue section stained with two or more biomarkers; accessing a color lookup table (cLUT) that maps RGB pixel values from the multiplex image to RGB pixel values of a synthetic singleplex image corresponding to a selected biomarker, the cLUT having been precomputed from a plurality of aligned multiplex and singleplex image pairs;, wherein the cLUT was generated by: accessing a plurality of aligned image pairs, each comprising a multiplex image and a synthetic singleplex image for a specific biomarker; computing, for each unique RGB value in the multiplex images, a representative RGB value based on corresponding pixels in the synthetic singleplex images; and storing the mapping from each multiplex RGB value to the computed representative RGB value in the cLUT; for each pixel in the multiplex image, determining an index based on its RGB value and retrieving from the cLUT a corresponding output RGB value; generating a synthetic singleplex image by assigning, to each pixel location, the corresponding output RGB value retrieved from the cLUT; and outputting the synthetic singleplex image for rendering or downstream analysis.

9. The system of claim 8, wherein computing the representative RGB value comprises computing a statistical measure selected from the group consisting of: a mean, a median, or a mode of RGB values from the corresponding pixels in the synthetic singleplex images.

10. The system of claim 8, wherein the RGB values of the multiplex image are binned into discrete intervals prior to computing the representative RGB values, such that each RGB bin maps to a common representative RGB value.

11. The system of claim 8, wherein the multiplex and synthetic singleplex images are transformed into a standardized color space prior to generating the cLUT, the standardized color space comprising sRGB obtained using an ICC color profde.

12. The system of claim 8, wherein the set of operations further includes applying two or more distinct cLUTs to the same multiplex image, each cLUT corresponding to a different biomarker, to generate multiple synthetic singleplex images.

13. The system of claim 8, wherein evaluation of the synthetic singleplex image includes comparing the image to a reference image using one or more quantitative image quality metrics selected from the group consisting of Structural Similarity Index Measure (SSIM), mean squared error (MSE), and Delta-E color difference.

14. The system of claim 8, wherein the cLUT is configured as a multidimensional array indexed by RGB values, and interpolation is used to compute output RGB values for input values that do not have an exact match in the cLUT.

15. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform a set of operations including: accessing a multiplex image comprising red, green, and blue (RGB) pixel values representing a tissue section stained with two or more biomarkers; accessing a color lookup table (cLUT) that maps RGB pixel values from the multiplex image to RGB pixel values of a synthetic singleplex image corresponding to a selected biomarker, the cLUT having been precomputed from a plurality of aligned multiplex and singleplex image pairs;, wherein the cLUT was generated by: accessing a plurality of aligned image pairs, each comprising a multiplex image and a synthetic singleplex image for a specific biomarker; computing, for each unique RGB value in the multiplex images, a representative RGB value based on corresponding pixels in the synthetic singleplex images; and storing the mapping from each multiplex RGB value to the computed representative RGB value in the cLUT; for each pixel in the multiplex image, determining an index based on its RGB value and retrieving from the cLUT a corresponding output RGB value; generating a synthetic singleplex image by assigning, to each pixel location, the corresponding output RGB value retrieved from the cLUT; and outputting the synthetic singleplex image for rendering or downstream analysis.

16. The computer-program product of claim 15, wherein computing the representative RGB value comprises computing a statistical measure selected from the group consisting of a mean, a median, or a mode of RGB values from the corresponding pixels in the synthetic singleplex images.

17. The computer-program product of claim 15, wherein the RGB values of the multiplex image are binned into discrete intervals prior to computing the representative RGB values, such that each RGB bin maps to a common representative RGB value.

18. The computer-program product of claim 15, wherein the multiplex and synthetic singleplex images are transformed into a standardized color space prior to generating the cLUT, the standardized color space comprising sRGB obtained using an ICC color profile.

19. The computer-program product of claim 15, wherein the set of operations further comprises applying two or more distinct cLUTs to the same multiplex image, each cLUT corresponding to a different biomarker, to generate multiple synthetic singleplex images.

20. The computer-program product of claim 15, wherein evaluation of the synthetic singleplex image includes comparing the image to a reference image using one or more quantitative image quality metrics selected from the group consisting of: Structural Similarity Index Measure (SSIM), mean squared error (MSE), and Delta-E color difference.