CA2948499A1

CA2948499A1 - System and method for classifying and segmenting microscopy images with deep multiple instance learning

Info

Publication number: CA2948499A1
Application number: CA2948499A
Authority: CA
Inventors: Oren Kraus; Brendan Frey; Jimmy Ba
Original assignee: University of Toronto
Current assignee: Dewpoint Therapeutics Inc
Priority date: 2016-11-16
Filing date: 2016-11-16
Publication date: 2018-05-16
Anticipated expiration: 2036-11-16
Also published as: CA2948499C

Abstract

Systems and methods that receive as input microscopy images, extract features, and apply layers of processing units to compute one or more set of cellular phenotype features, corresponding to cellular densities and/or fluorescence measured under different conditions. The system is a neural network architecture having a convolutional neural network followed by a multiple instance learning (MIL) pooling layer. The system does not necessarily require any segmentation steps or per cell labels as the convolutional neural network can be trained and tested directly on raw microscopy images in real-time. The system computes class specific feature maps for every phenotype variable using a fully convolutional neural network and uses multiple instance learning to aggregate across these class specific feature maps. The system produces predictions for one or more reference cellular phenotype variables based on microscopy images with populations of cells.

Description

2 WITH DEEP MULTIPLE INSTANCE LEARNING

3 TECHNICAL FIELD

4 [0001] The following relates generally to microscopy imaging and more specifically to the classification and segmentation of microscopy images utilizing deep learning with a multiple 6 instance learning pooling layer.

8 [0002] High-content screening (HCS) technologies that combine automated fluorescence 9 microscopy with high-throughput biotechnology have become powerful systems for studying cell biology and for drug screening. However, these systems can produce more than 105 images per 11 day, making their success dependent on automated image analysis.
Traditional analysis 12 pipelines heavily rely on hand-tuning the segmentation, feature extraction and classification 13 steps for each assay. Although comprehensive tools have become available, they are typically 14 optimized for mammalian cells and not directly applicable to model organisms such as yeast and Caenorhabditis elegans. Researchers studying these organisms often manually classify 16 cellular patterns by eye.
17 [0003] Recent advances in deep learning indicate that deep neural networks trained end-to-end 18 can learn powerful feature representations and outperform classifiers built on top of extracted 19 features. Although object recognition models, particularly convolutional networks, have been successfully trained using images with one or a few objects of interest at the center of the 21 image, microscopy images often contain hundreds of cells with a phenotype of interest, as well 22 as outliers.
23 [0004] Fully convolutional neural networks (FCNNs) have been applied to natural images for 24 segmentation tasks using ground truth pixel-level labels. These networks perform segmentation for each output category instead of producing a single prediction vector. For microscopy data, 26 convolutional sparse coding blocks have also been used to extract regions of interest from 27 spiking neurons and slices of cortical tissue without supervision. Other approaches utilize 28 FCNNs to perform segmentation using weak labels. However, while these techniques aim to 29 segment or localize regions of interest within full resolution images, they do not classify populations of objects in images of arbitrary size based on only training with weak labels. These 31 techniques suffer because dense pixel level ground truth labels are expensive to generate and 32 arbitrary, especially for niche datasets such as microscopy images.

1 [0005] Thus, there is a lack of automated cellular classification systems using full resolution 2 images. Applying deep neural networks to microscopy screens has been challenging due to the 3 lack of training data specific to cells; i.e., a lack of large datasets labeled at the single cell level.

[0006] In one aspect, a neural network architecture for classifying microscopy images 6 representing one or more cell classes is provided, the neural network architecture comprising: a 7 convolutional neural network (CNN) comprising: an input layer for inputting the microscopy 8 images; one or more hidden layers of processing nodes, each processing node comprising a 9 processor configured to apply an activation function and a weight to its inputs, a first of the hidden convolutional layers receiving an output of the input layer and each subsequent hidden 11 layer receiving an output of a prior hidden layer, each hidden layer comprising a convolutional 12 layer; and a hidden layer to generate one or more class specific feature maps for cellular 13 features of one or more cell classes present in the microscopy images;
and a global pooling 14 layer configured to receive the feature maps for cellular features and to apply a multiple instance learning pooling function to produce a prediction for each cell class present in the 16 microscopy images.
17 [0007] In another aspect, a method for classifying microscopy images representing one or more 18 cell classes using a neural network is provided, the method comprising:
applying a convolutional 19 neural network (CNN) to the microscopy images, the CNN comprising: an input layer for inputting the microscopy images; one or more hidden layers of processing nodes, each 21 processing node comprising a processor configured to apply an activation function and a weight 22 to its inputs, a first of the hidden convolutional layers receiving an output of the input layer and 23 each subsequent hidden layer receiving an output of a prior hidden layer, each hidden layer 24 comprising a convolutional layer; and a hidden layer to generate one or more class specific feature maps for cellular features of one or more cell classes present in the microscopy images;
26 and applying a global pooling layer to the feature maps for cellular features to apply a multiple 27 instance learning pooling function to produce a prediction for each cell class present in the 28 microscopy images.
29 [0008] These and other aspects are contemplated and described herein. It will be appreciated that the foregoing summary sets out representative aspects of convolutional neural networks 31 and microscopy imaging systems and methods for the classification and segmentation of 32 microscopy images utilizing deep multiple instance learning to assist skilled readers in 33 understanding the following detailed description.

2 [0009] The features of the invention will become more apparent in the following detailed 3 description in which reference is made to the appended drawings wherein:
4 [0010] Fig. 1 is a system for classifying and segmenting microscopy images;
[0011] Fig. 2 is an exemplary CNN and MIL pooling layer in accordance with the system for 6 classifying and segmenting microscopy images;
7 [0012] Fig. 3 shows MIL pooling functions with class specific feature map activations (P1) for a 8 drug screen data sample;
9 [0013] Fig. 4 shows class feature map probabilities for a test sample labeled as "5" overlaid onto the input image;
11 [0014] Fig. 5 shows a benchmarking dataset of MFC-7 breast cancer cells;
12 [0015] Fig. 6 shows a yeast protein localization screen; and 13 [0016] Fig. 7 shows localizing cells with Jacobian maps generated for an image with transient, 14 cell cycle dependent protein localizations.
DETAILED DESCRIPTION
16 [0017] Embodiments will now be described with reference to the figures.
For simplicity and 17 clarity of illustration, where considered appropriate, reference numerals may be repeated 18 among the Figures to indicate corresponding or analogous elements. In addition, numerous 19 specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the 21 embodiments described herein may be practised without these specific details. In other 22 instances, well-known methods, procedures and components have not been described in detail 23 so as not to obscure the embodiments described herein. Also, the description is not to be 24 considered as limiting the scope of the embodiments described herein.
[0018] Various terms used throughout the present description may be read and understood as 26 follows, unless the context indicates otherwise: "or" as used throughout is inclusive, as though 27 written "and/or"; singular articles and pronouns as used throughout include their plural forms, 28 and vice versa; similarly, gendered pronouns include their counterpart pronouns so that 29 pronouns should not be understood as limiting anything described herein to use, implementation, performance, etc. by a single gender; "exemplary" should be understood as 1 "illustrative" or "exemplifying" and not necessarily as "preferred" over other embodiments.
2 Further definitions for terms may be set out herein; these may apply to prior and subsequent 3 instances of those terms, as will be understood from a reading of the present description.
4 [0019] Any module, unit, component, server, computer, terminal, engine or device exemplified herein that executes instructions may include or otherwise have access to computer readable 6 media such as storage media, computer storage media, or data storage devices (removable 7 and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Computer 8 storage media may include volatile and non-volatile, removable and non-removable media 9 implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data.
Examples of computer 11 storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-12 ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, 13 magnetic disk storage or other magnetic storage devices, or any other medium which can be 14 used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the device or accessible or connectable 16 thereto. Further, unless the context clearly indicates otherwise, any processor or controller set 17 out herein may be implemented as a singular processor or as a plurality of processors. The 18 plurality of processors may be arrayed or distributed, and any processing function referred to 19 herein may be carried out by one or by a plurality of processors, even though a single processor may be exemplified. Any method, application or module herein described may be implemented 21 using computer readable/executable instructions that may be stored or otherwise held by such 22 computer readable media and executed by the one or more processors.
23 [0020] The following provides a system and method for classifying microscopy images with 24 deep multiple instance learning (MIL) without the prior need of segmentation. The system described herein is also capable of performing segmentation. This method described herein 26 allows the provided system to learn instance and bag level classifiers for full resolution 27 microscopy images without ever having to segment or label single cells.
28 [0021] In particular, the system comprises a convolutional neural network (CNN) having an 29 output linked to a pooling layer configured using MIL (alternatively described herein as a "convolutional MIL network"). The convolutional MIL network described herein uses MIL to 31 simultaneously classify and segment microscopy images with populations of cells. In an 32 embodiment, the CNN outputs class-specific feature maps representing the probabilities of the 33 classes for different locations in the input image and the MIL pooling layer is applied to these 1 feature maps. The system can be trained using only whole microscopy images with image level 2 labels, without requiring any segmentation steps. Processed images can be of arbitrary size and 3 contain varying number of cells. Individual cells can be classified by passing segmented cells 4 through the trained CNN or by mapping the probabilities in class specific feature maps back to the input space.
6 [0022] The systems and methods described herein relate, in part, to the problem of classifying 7 and segmenting microscopy images using only whole image level annotations. This problem 8 has implications in several industrial categories under the broad umbrellas of 'medicine' and 9 'imaging', including cellular microscopy, molecular diagnostics and pharmaceutical development.
11 [0023] MIL deals with problems for which labels only exist for sets of data points. In MIL, sets of 12 data points are typically referred to as bags and specific data points are referred to as 13 instances. A commonly used assumption for binary labels is that a bag is considered positive if 14 at least one instance within the bag is positive. Representative functions for mapping the instance space to the bag space include Noisy-OR, log-sum-exponential (LSE), generalized 16 mean (GM) and the integrated segmentation and recognition (ISR) model.
17 [0024] In an embodiment of the present system, the MIL pooling layer implements a pooling 18 function defined herein as "Noisy-AND". Unlike the aforementioned mapping functions, the 19 Noisy-AND pooling function is robust to outliers and large numbers of instances.
[0025] Referring now to Fig. 1, shown therein is a system 100 for classifying and segmenting 21 microscopy images, comprising a convolutional neural network 101, which is alternatively 22 referred to herein as a "convolutional network" or "CNN", an MIL pooling layer 109 and a 23 memory 106 communicatively linked to the CNN 101. The CNN comprises an input layer 103 24 that takes as input a set of microscopy images potentially depicting one or more cell classes and exhibiting cellular densities and florescence related to protein localization, one or more 26 hidden layers 105 for processing the images, and an output layer. 107 that produces feature 27 maps for every output category (i.e., cell class) to mimic a specific cell phenotype and/or 28 localized protein. In embodiments, the CNN may provide other outputs in addition to feature 29 maps. An MIL pooling layer 109 is applied to the feature maps to generate predictions of the cell classes present in the image.
31 [0026] In Fig. 1, an illustrated embodiment of the system 100 in which the CNN 101 has a 32 plurality of hidden layers 105 (i.e. deep) is shown. Each hidden layer 105 of the CNN 101

5 1 generally comprises a convolutional layer 111 followed optionally by a pooling layer 113. The 2 pooling layer 113 of a layer i will be connected to a convolutional layer 111 of a layer 4/. These 3 pooling layers 113 should not be confused with the MIL pooling layer 109 of the system 100 as 4 a whole.
[0027] Each of the convolutional layers 111 and pooling layers 113 comprise a plurality of

6 processing units. Each processing unit may be considered a processing "node" of the network

7 and one or more nodes may be implemented by processing hardware, such as a single or multi-

8 core processor and/or graphics processing unit(s) (GPU(s)). Further, it will be understood that

9 each processing unit may be considered to be associated with a hidden unit or an input unit of the neural network for a hidden layer or an input layer, respectively. The use of large (many 11 hidden variables) and deep (multiple hidden layers) neural networks may improve the predictive 12 performances of the CNN compared to other systems.
13 [0028] Each node is configured with an activation function (acting as a feature detector) and a 14 weighting. The activation functions are fixed for each of the processing nodes and the weighting is stored in the memory 106, which is linked to each such node. The weights are determined 16 during a training stage of the CNN 101 and stored in the memory 106.
17 [0029] In embodiments, inputs to the input layer 103 of the CNN 101 are microscopy images 18 that are associated or associable with microscopy information, such as cellular densities, size, 19 cellular division, features derived from fluorescence detection, and features providing extra information (e.g. ultrastructure, protein-protein interactions and cell cycle), while outputs at the 21 output layer 109 of the CNN 101 are feature maps. The MIL pooling layer 109 generates 22 predictions of cell classes present in the images based on the feature maps.
23 [0030] The memory 106 may comprise a database for storing activations and learned weights 24 for each feature detector, as well as for storing datasets of microscopy information and extra information and optionally for storing outputs from the CNN 101 or MIL pooling layer 109. The 26 microscopy information may provide a training set comprising training data. The training data 27 may, for example, be used for training the CNN 101 to generate feature maps, in which visually 28 assigning annotations from a known screen may be provided; specifically, optionally labelling 29 proteins that are annotated to localize to more than one sub-cellular compartment. The memory 106 may further store a validation set comprising validation data.
31 [0031] Generally, during the training stage, the CNN 101 learns optimized weights for each 32 processing unit. After learning, the optimized weight configuration can then be applied to test 1 data (and the validation data prior to utilizing the neural network for test data). Stochastic 2 gradient descent can be used to train feedforward neural networks. A
learning process 3 (backpropagation), involves for the most part matrix multiplications, which makes them suitable 4 for speed up using GPUs. Furthermore, the dropout technique may be utilized to prevent overfitting.
6 [0032] The system may further comprise a computing device 115 communicatively linked to the 7 convolutional MIL network for controlling operations carried out in the convolutional MIL
8 network. The computing device 115 may comprise further input and output devices, such as 9 input peripherals (such as a computer mouse or keyboard), and/or a display. Cellular feature maps representing the probabilities of the classes for different locations in the input image, 11 and/or predictions generated by the MIL pooling layer 109, may be visualized and displayed to a 12 user via the display.
13 [0033] Referring now to Fig. 2, an exemplary convolutional MIL network is shown. Assuming 14 that the total number of classes is Nais, for a full resolution image I, each class i may be treated as a separate binary classification problem with label t, c {0,1{.
Under this exemplary 16 MIL formulation, one is given a bag of N instances that are denoted as x = {x1,. = -,x,v} and 17 Xn E RD is the feature vector for each instance. The class labels t, are associated with the 18 entire bag instead of each instance. A binary instance classifier p(t, =11 x1) is used to generate 19 predictions across the instances in a bag. The instance predictions Ipd are combined through an aggregate function g(.), e.g. noisy-OR, to map the set of instance predictions to the 21 probability of the final bag label p(t,=11x1,= = -,x,v) . In an exemplary CNN, each activation in the 22 feature map is computed through the same set of filter weights convolved across the input 23 image. The pooling layers then combine activations of feature maps in convolutional layers. If 24 class specific feature maps are treated as bags of instances, the classical approaches in MIL
can be generalized to global pooling layers over these feature maps.
26 [0034] The MIL pooling layer in a convolutonal MIL network may be formulated as a global 27 pooling layer over a class specific feature map for class i referred to as the bag pi. Assume 28 that the class specific convolutional layer in a CNN computes a mapping directly from input 29 images to sets of binary instance predictions / --> {pio= ==p,N} . It first outputs the logit values zu in the feature map corresponding to instance j in the bag i. The feature level probability of 1 an instance j belonging to class i is defined as p, where põ ¨0-(z11) and o- is the sigmoid 2 function. The image level class prediction is obtained by applying the global pooling function 3 g() over all elements p,1. The global pooling function g() maps the instance space 4 probabilities to the bag space such that the bag level probability for class i is defined by P, = )= (1) 6 [0035] The global pooling function g() essentially combines the instance probabilities from 7 each class specific feature map p, into a single probability. This reduction allows training and 8 evaluation of the convolutonal MIL network on inputs of arbitrary size.
9 [0036] While the MIL pooling layer learns the relationship between instances of the same class, the co-occurrence statistics of instances from different classes within the bag could also be 11 informative for predicting the bag label. An extension of the convolutional MIL network is 12 provided to learn relationships between classes by adding an additional fully connected layer 13 117 following the MIL pooling layer. This layer 117 can either use softmax or signnoid activations 14 for either multi-class or multi-label problems, respectively. The softmax output from this layer 117 for each class i is defined as I. A joint cross entropy objective function is formulated at 16 both the MIL pooling layer and the additional fully connected layer defined by c 17.J=¨ (log p(t, I f:)+ log p(t, I
y,)). (2) 18 where Atilp) is the binary class prediction from the MIL layer, and where 19 At,' P,) = (1¨P,)(11') and p(t,1 y,) are either the binary or the multi-class prediction from the fully connected layer.
21 [0037] Prior MIL formulations are based on the assumption that at least one instance needs to 22 be positive for the bag level to be positive. However, due to heterogeneity within cellular 23 populations, imaging artifacts, and the large number of potential instances in an image, it cannot 24 be assumed that images with a negative label do not contain any instances of the specific phenotype. A more reasonable assumption is that bag labels are determined by a certain 26 proportion of instances being present.
27 [0038] In an embodiment, bag predictions are expressed as the geometric or arithmetic mean 28 of instances. This may address some of the challenges associated with imaging cellular 1 population and represent a generalized MIL problem. Prior generalizations for MIL are based on 2 the assumption that all instances collectively contribute to the bag label. However, for 3 microscopy images, it cannot be assumed that all bags require the same proportion of instances 4 to be positive.
[0039] In another embodiment, the use of several different global pooling functions g(=) in the 6 MIL pooling layer may be employed, where j indexes the instance within a bag. Previously 7 proposed global pooling functions for MIL have been designed as differentiable approximations 8 to the max function in order to satisfy the standard MIL assumption:
9 g({p1})=1¨n(1_pj) Noisy¨OR, g({13J})=1 ___ 1(11-E __ ) ISR, 1¨ 1¨ P, 11 g({p.,})=( __ Dr)11r Generalizedmean, 12 g({p,})¨ ¨1 log( le ') LSE.
r j 13 [0040] The inclusion of Noisy-OR and ISR can be sensitive to outliers and challenging to work 14 with microscopy datasets (as shown in Fig. 3). LSE and GM both have a parameter r that controls their sharpness. As r increases, the functions get closer to representing the max of the 16 instances. However, the present system utilizes a lowered r to allow more instances in the 17 feature maps to contribute to the pooled value.
18 [0041] Preferably, a pooling function defined herein as the Noisy-AND
pooling function is used.
19 In Noisy-AND, it may be assumed that a bag is positive if the number of positive instances in the bag surpasses a certain predefined threshold. The Noisy-AND pooling function is defined as:
¨b))¨o-(¨abi) 21 P,= g,({1)})¨ _______________ , (3) 22 where pi ¨ .

1 [0042] The Noisy-AND pooling function is designed to activate a bag level probability P, once 2 the mean of the instance level probabilities surpasses a certain threshold. This behaviour 3 mimics the logical AND function in the probabilistic domain. The parameters a and b, control 4 the shape of the activation function. b, is a set of parameters learned during training and represents an adaptable soft threshold for each class i. a is a fixed parameter that controls the 6 slope of the activation function. The terms a(-ab,) and o-(a(1-b,)) are included to normalized 7 P, to [0,1] for b, in [OA and a > O. Fig. 3 shows plots of relevant pooling functions. The top 8 exemplary graph shows pooling function activations by ratio of feature map activated (7). The 9 bottom exemplary graph shows activation functions learned by Noisy-AND
alc, (nand_a = 10.0) for different classes of an exemplary benchmarking dataset of breast cancer cells. For all of the 11 bag level evaluations used in this example, the Noisy-AND pooling function performs best. The 12 Noisy-AND pooling function accommodates variability related to different phenotypes by 13 learning an adaptive threshold for every class.
14 [0043] In another embodiment, the convolutional MIL network is used to localize regions of the full resolution input images that are responsible for activating the class specific feature maps.
16 This extension may be particularly useful for researchers conducting HCS
experiments who are 17 interested in obtaining statistics from single cell measurements of their screens. The pre-18 softmax activations of specific output nodes are back-propagated through a classification 19 network to generate Jacobian maps with respect to specific class predictions. The following general recursive non-linear back-propagation process is defined for computing a backward 21 activation a for each layer, analogous to the forward propagation:
-(1-1)8z -(') -co 22 a )- ,f( a (4) az(1-1) -r 23 where f (x)= max(0,x),a,, = p = pi, , 24 am is the hidden activations in layer 1, and z(i) is pre-nonlinearity activations 26 [0044] To start, the non-linear back-propagation (al; ) from the MIL
layer using its sigmoidal 27 activations for the class i specific feature maps {AI is multiplied by the pooling activation for 1 each class P, = põ . Applying the ReLU activation function to the partial derivatives during back 2 propagation generates Jacobian maps that are sharper and more localized to relevant objects in 3 the input. To generate segmentation masks, the sum of the Jacobian maps is thresholded along 4 the input channels. To improve the localization of cellular regions loopy belief propagation may be employed in an MRF to de-noise the thresholded Jacobian maps.
6 [0045] The CNN is designed such that an input the size of a typical cropped single cell 7 produces output feature maps of size 1x1. The same network can be convolved across larger 8 images of arbitrary size to produce output feature maps representing probabilities of target 9 labels for different locations in the input image. Training such a CNN
end-to-end allows the CNN
to work on vastly different datasets.
11 [0046] An exemplary embodiment is now described. In validation tests, the following CNN was 12 trained using two exemplary datasets while keeping the architecture and number of parameters 13 constant.
14 [0047] The basic convolutional MIL network architecture includes the following layers:
ave_poo10_3x3, conv1_3x3x32, conv2_3x3_64, pooh _3x3, conv3_5x5_64, poo12_3x3, 16 conv4_3x3_128, poo13_3x3, conv5_3x3_128, poo14_3x3, conv6_1x1_1000, conv7_1x1_ Naas, , 17 MIL_pool, FC_Na,,,, (as shown in Fig. 2). However, it will be appreciated that the convolutional 18 MIL network architecture may not be limited to this architecture.
19 [0048] A global pooling function g(=) is used as the activation function in the MIL_pool layer.
g(-) transforms the output feature maps z, into a vector with a single prediction P, for each 21 class i. In an exemplary embodiment, all of the above-mentioned pooling functions are defined 22 for binary categories and may be used in a multi-label setting (where each output category has 23 a separate binary target). In another embodiment, an additional fully connected output layer 24 may be added to the MIL_pool layer in order to learn relations between different categories.
Exemplary activations include softmax activation and signnoidal activation. In this example, both 26 exemplary MIL activations are trained with a learning rate of 10-3 using the Adam optimization 27 algorithm. Slightly smaller crops of the original images may be extracted to account for 28 variability in image sizes within the screens. The images are normalized by subtracting the 29 mean and dividing by the standard deviation of each channel in the training sets. During training, random patches are cropped from the full resolution images and random rotations and 31 reflections to the patches applied. The ReLU activation for the convolutional layers may be used 1 and, as an example, 20% dropout to the pooling layers and 50% dropout to layer conv6 may be 2 applied. In the following example data sets, the CNNs may be trained within 1-2 days on a Tesla 3 K80 GPU using 9 Gb of memory with a batch size of 16.
4 [0049] Following training, an image of any size can be passed through the convolutional MIL
network. This can be useful for classifying individual cropped objects or image patches. One 6 could use a separate segmentation algorithm (such as Otsu thresholding, mixture of Gaussians, 7 region growing, graphical models, etc.) to identify object locations, crop bounding boxes around 8 them, and pass them through the convolutional MIL network in order to produce single cell 9 predictions. Alternatively, the cellular regions can be identified by back propagating errors through the network to the input space, as earlier described.
11 [0050] Referring now to Fig. 4, shown therein is an example image representing a first dataset 12 used for validating the effectiveness of the present system. The convolutional MIL network is 13 used to classify images based on the presence of a threshold number of handwritten digits.
14 Four example feature maps are shown for feature map activations P, (for = 4, 5, 6 or 7) for a test sample labeled with handwritten digits including "0" and "5" overlaid onto the input image.
16 Each image in the dataset contains 100 digits cluttered on a black background of 512 x 512 17 pixels. The dataset may contain nine categories (digits E {1,2,..,9D and zeros may be used as 18 distractors. To simulate the conditions in cell culture microscopy, among the 100 digits, x 19 samples may be chosen from a single category and the remaining 100- x samples are zeros. x is fixed for each category and is equal to 10 times the digit value of the chosen category, as 21 shown in Fig. 4. For example, an image with label "5" contains 50 fours and 50 zeros. In the 22 exemplary embodiment, 50 images were used per category for training and

10 images per 23 category for testing.
24 [0051] In the example conducted, the CNNs trained on the cluttered hand written digits achieved 0% test error across all classes. These error rates were achieved despite the fact 26 images labeled as one actually contain 90 zeros and only 10 ones. The reason the 27 convolutional MIL network does not confuse zeros for ones in these samples is because zeros 28 also appear in images labeled with other categories, hence the convolutional MIL network is 29 capable of determining that the zeros constitute distractions. Another important element is that since there are only 50 training samples per digit, the CNN only sees 500 distinct ones during 31 training. The classic MNIST training dataset contains 6,000 cropped and centered samples per 32 category. The provided superior test performance with fewer training samples using the MIL
33 formulation is the result of the convolutional MIL network predictions being based on 1 aggregating over multiple instances. The convolutional MIL network may ignore samples that 2 are difficult to classify but still rely on easier instances to predict the overall image correctly.
3 Because different sampling rates for each digit category may be utilized, this exemplary 4 embodiment also shows that the convolutional MIL pooling layers are robust to different frequencies of the label class being present in the input image. In the specific image analyzed in 6 Fig. 4, class specific feature map activations (131) is shown for a test sample labeled as "5"
7 overlaid onto the input image. The convolutional MIL network successfully classifies almost all 8 the "5"s in the image and is not sensitive to the background or distractors (i.e. zeros).
9 [0052] Referring now to Fig. 6, a genonne wide screen of protein localization in yeast is shown.
The screen contains images of 4,144 yeast strains from the yeast GFP
collection representing

11 71% of the yeast proteome. The images contain 2 channels, with fluorescent markers for the

12 cytoplasm and a protein from the GFP collection at a resolution of 1010x1335. This exemplary

13 embodiment sampled 6% of the screen and used 2200 whole microscopy images for training

14 and 280 for testing. The whole images of strains were characterized into 17 localization classes based on visually assigned localization annotations from a previous screen.
These labels 16 include proteins that were annotated to localize to more than one sub-cellular compartment.
17 [0053] Table 1 provides the yeast dataset results on whole images. The results include the 18 accuracy and mean classifier accuracy across 17 classes for a subset of 998 proteins annotated 19 to localize to one sub-cellular compartment and the mean average precision for the all the proteins analyzed from the screen (2592), including proteins that localize to multiple 21 compartments. The "Huh" column indicates agreement with manually assigned protein 22 localizations. The "Single loc acc" and "single loc mean acc" columns indicate the accuracy and 23 mean accuracy, respectively, across all classes for a subset of proteins that localize to a single 24 compartment. The "full image" column indicates mean average precision on a full resolution image test set.

Mean average prec. Classification Model full image Huh single single loc loc acc. mean acc.
Chong et al. (2015) 0.703 0.935 0.808 Noisy-AND (lc 0.921 0.815 0.942 0.821 Noisy-AND a7.5 0.920 0.846 0.963 0.834 Noisy-AND ay, 0.950 0.883 0.953 0.876 LSE ri 0.925 0.817 0.945 0.828 LSE r25 0.925 0.829 0.953 0.859 LSE 1'5 0.933 0.861 0.960 0.832 GM 7-1 (avg. pooling) 0.915 0.822 0.938 0.862 GM r2. s 0.888 0.837 0.922 0.778 GM 1'5 0.405 0.390 0.506 0.323 max pooling 0.125 0.133 0.346 0.083 2 [0054] In addition to the performance on full resolution images, yeast dataset results on 3 segmented cells is provided in Table 2.
4 [0055] From a previous analysis pipeline using CellProfiler, the center of mass coordinates of segmented cells may be extracted and these coordinates used to crop single cells (for example, 6 crop size of 64 x 64) from the full resolution images. The dataset reflected in the results of Table 7 2 were annotated according to the labels from the full resolution images and likely includes 8 mislabelled samples. Also included is performance on 6,300 manually labelled segmented cells 9 used to train the SVM classifiers described in Chong,Y.T. et al. (2015) Yeast proteome dynamics from single cell imaging and automated analysis. Cell, 161, 1413-1424. For these 11 predictions we use the output from the MIL_pool layer.
12 [0056] Table 2 compares performance of a traditional CNN trained on the segmented cells with 13 noisy, whole image level labels on a dataset of manually labeled segmented cells. As an 14 additional baseline, a traditional CNN trained on the manually labeled cells achieved a test accuracy of 89.8%.
16 [0057] This dataset may be annotated according to the labels from the full resolution images 17 and may include mislabelled samples. For these predictions the output from the MIL pooling 18 layer may be utilized.

Mean average precision Model Segmented cells Segmented cells with noisy labels with manual labels CNN trained on 0.855 0.742 segmented cells with noisy labels Noisy-AND as 0.701 0.750 Noisy-AND a7,5 0.725 0.757 Noisy-AND alo 0.701 0.738 LSE ri 0.717 0.763 LSE r2 5 0.715 0.762 LSE r5 0.674 0.728 GM r1 (avg. pooling) 0.705 0.741 GM r2 0.629 0.691 GM rs 0.255 0.258 max pooling 0.111 0.070 2 [0058] Referring now to Fig. 5, a breast cancer screen is shown. More specifically, shown 3 therein is a benchmarking dataset of MFC-7 breast cancer cells available from the Broad 4 Bioimage Benchmark Collection (image set BBBCO21v1). The images contain 3 channels with fluorescent markers for DNA, actin filaments, and ,6 -tubulin at a resolution of 1024x1280.
6 Within this exemplary dataset, 103 treatments (compounds at active concentrations) have 7 known effects on cells based on visual inspection and prior literature and can be classified into 8 12 distinct categories referred to as mechanism of action (MOA). This exemplary embodiment 9 sampled 15% of images from these 103 treatments to train and validate the CNN. The same proportion of the data was used to train the best architecture reported in Ljosa,V. et al. (2013) 11 Comparison of methods for image-based profiling of cellular morphological responses to small-12 molecule treatment. J. Biomol. Screen., 18, 1321-1329. In total, 300 whole microscopy images 13 were used during training and 40 for testing. Evaluation of all the images in the screen is 14 provided and reports on the predicted treatment accuracy across the treatments.
[0059] Table 3 provides the breast cancer dataset results on whole images. The "full image"
16 column indicates accuracy on a full resolution image test set. The "treatment" column indicates 17 accuracy predicting treatment MOA by taking the median prediction over three experimental 18 replicates of the screen. For these predictions the output from the last layer of the network may 19 be used.

Model full image treatment Ljosa et al. (2013) 0.94 Noisy-AND as 0.915 0.957 Noisy-AND a7.5 0.915 0.957 Noisy-AND al() 0.958 0.971 LSE ri 0.915 0.943 LSE r25 0.888 0.871 LSE r 5 0.940 0.957 GM r1 (average pooling) 0.924 0.943 GM r25 0.924 0.957 GM rs 0.651 0.686 max pooling 0.452 0.429 2 [0060] Referring now to Fig. 7, a set of cell localizations is shown. The trained CNN described 3 herein can be used to locate regions with cells without additional training. Segmentation maps 4 may be generated to identify cellular regions in the input by back-propagating activations as described above with relation to the input space as shown in Fig. 7. We refer to gradients with 6 respect to the input as Jacobian maps. To evaluate the segmentation method the mean 7 intersection over union (IU) between the calculated maps and segmentation maps generated 8 optionally using the global Otsu thresholding module in CellProfiler. A
mean IU of 81.2% was 9 achieved using this method. The mask pairs with low IU were mostly incorrect using Otsu thresholding. The CNN may generate class specific segmentation maps by back-propagating 11 individual class specific feature maps while setting the rest of the feature maps to zero.
12 Specifically, Fig. 7 shows the Jacobian maps generated for an image with transient, cell cycle 13 dependent protein localizations.
14 [0061] For all of the bag level evaluation shown above, we see that the Noisy-AND models perform best, which follows from the pooling functions plotted in Fig. 3.
Setting the scaling 16 factors (a, r) to lower values make the pooling functions approach mean of the feature maps, 17 while for higher values the functions approach the max function. Since different phenotype 18 categories may have vastly different densities of cells neither extreme suits all classes. The 19 Noisy-AND pooling function accommodates this variability by learning an adaptive threshold for every class, as shown in Fig. 3.
21 [0062] Although the foregoing has been described with reference to certain specific 22 embodiments, various modifications thereto will be apparent to those skilled in the art without 1 departing from the spirit and scope of the invention as outlined in the appended claims. The 2 entire disclosures of all references recited above are incorporated herein by reference.

Claims

We claim:

1. A neural network architecture for classifying microscopy images representing one or more cell classes, the neural network architecture comprising:
a convolutional neural network (CNN) comprising:
an input layer for inputting the microscopy images;
one or more hidden layers of processing nodes, each processing node comprising a processor configured to apply an activation function and a weight to its inputs, a first of the hidden convolutional layers receiving an output of the input layer and each subsequent hidden layer receiving an output of a prior hidden layer, each hidden layer comprising a convolutional layer; and a hidden layer to generate one or more class specific feature maps for cellular features of one or more cell classes present in the microscopy images; and a global pooling layer configured to receive the feature maps for cellular features and to apply a multiple instance learning pooling function to produce a prediction for each cell class present in the microscopy images.

2. The neural network architecture of claim 1, wherein the CNN is trained using a training set comprising only whole microscopy images with image level labels.

3. The neural network architecture of claim 1, wherein the class specific feature maps represent probabilities of the cell classes for various locations in the microscopy image.

4. The neural network architecture of claim 1, wherein the CNN is configured to generate class' specific feature maps and the global pooling layer is configured to produce class predictions for microscopy images of arbitrary size irrespective of images with which the CNN is trained.

5. The neural network architecture of claim 1, wherein the CNN is configured to generate the class specific feature maps for microscopy images having any number of present cell classes irrespective of images with which the CNN is trained.

6. The neural network architecture of claim 1, wherein the pooling function maps instance space probabilities to bag space to define a bag level probability.

7. The neural network architecture of claim 6, further comprising a fully connected layer linked to the output of the global pooling layer to learn relationships between instances of a same cell class and co-occurrence information among different cell classes.

8. The neural network architecture of claim 6, wherein the bag level probability is utilized to produce a bag level prediction.

9. The neural network architecture of claim 8, wherein the bag level prediction is expressed as a mean of instance proportions among the cell classes.

10. The neural network architecture of claim 6, wherein the pooling function defines, for a particular image, one of the cell classes as positive if the number of observed instances of the respective cell class surpasses a predefined threshold.

11. The neural network architecture of claim 8, wherein the fully connected layer applies a softmax activation for multi-class learning.

12. The neural network architecture of claim 8, wherein the fully connected layer applies a sigmoid activation for multi-label learning.

13. The neural network architecture of claim 1, wherein the CNN is configured to generate Jacobian maps for specific cell class predictions by back-propagating through the CNN for a particular image.

14. The neural network architecture of claim 1, wherein the CNN is further configured to performing segmentation in the microscopy images.

15. The neural network architecture of claim 14, wherein the CNN performs classification and segmentation simultaneously.

16. A method for classifying microscopy images representing one or more cell classes using a neural network, the method comprising:
applying a convolutional neural network (CNN) to the microscopy images, the CNN
comprising:
an input layer for inputting the microscopy images;
one or more hidden layers of processing nodes, each processing node comprising a processor configured to apply an activation function and a weight to its inputs, a first of the hidden convolutional layers receiving an output of the input layer and each subsequent hidden layer receiving an output of a prior hidden layer, each hidden layer comprising a convolutional layer; and a hidden layer to generate one or more class specific feature maps for cellular features of one or more cell classes present in the microscopy images; and applying a global pooling layer to the feature maps for cellular features to apply a multiple instance learning pooling function to produce a prediction for each cell class present in the microscopy images.

17. The method of claim 16, wherein the CNN is trained using a training set comprising only whole microscopy images with image level labels.

18. The method of claim 16, wherein the class specific feature maps represent probabilities of the cell classes for various locations in the microscopy image.

19. The method of claim 16, wherein the CNN is configured to generate class specific feature maps and the global pooling layer is configured to produce class predictions for microscopy images of arbitrary size irrespective of images with which the CNN
is trained.

20. The method of claim 16, wherein the CNN is configured to generate the class specific feature maps for microscopy images having any number of present cell classes irrespective of images with which the CNN is trained.

21. The method of claim 16, wherein the pooling function maps instance space probabilities to bag space to define a bag level probability.

22. The method of claim 21, further comprising applying a fully connected layer to the output of the global pooling layer to learn relationships between instances of a same cell class and co-occurrence information among different cell classes.

23. The method of claim 21, wherein the bag level probability is utilized to produce a bag level prediction.

24. The method of claim 23, wherein the bag level prediction is expressed as a mean of instance proportions among the cell classes.

25. The method of claim 21, wherein the pooling function defines, for a particular image, one of the cell classes as positive if the number of observed instances of the respective cell class surpasses a predefined threshold.

26. The method of claim 23, wherein the fully connected layer applies a softmax activation for multi-class learning.

27. The method of claim 23, wherein the fully connected layer applies a sigmoid activation for multi-label learning.

28. The method of claim 16, wherein the CNN is configured to generate Jacobian maps for specific cell class predictions by back-propagating through the CNN for a particular image.

29. The method of claim 16, wherein the CNN is further configured to performing segmentation in the microscopy images.

30. The method of claim 29, wherein the CNN performs classification and segmentation simultaneously.