WO2021228894A1

WO2021228894A1 - Image analysis method in microscopy

Info

Publication number: WO2021228894A1
Application number: PCT/EP2021/062539
Authority: WO
Inventors: Manuel AMTHOR; Daniel Haase
Original assignee: Carl Zeiss Microscopy Gmbh
Priority date: 2020-05-14
Filing date: 2021-05-11
Publication date: 2021-11-18
Also published as: DE102020206088A1; CN115552462A

Abstract

The invention relates to an image analysis method in microscopy, in which method a first image (IA) of a specimen is recorded with a first image recording type and the image data is stored, wherein an image (IA→B) of the specimen with a second image recording type is simulated using the image data. The method also records a second image with the second image recording type and stores same. The method compares the simulated image of the second image recording type and the second recorded image to find differences which are captured, analysed and evaluated.

Description

Image evaluation methods in microscopy

The invention relates to an image evaluation method in microscopy according to the preamble of the independent claim.

Image evaluation methods are used in microscopy, among other things, to find areas of interest and / or unusual areas and differences in microscope images.

For example, on the basis of differences, faulty sample preparations can be recognized or an atypical reaction (anomalies) of certain sample areas can be recorded. Approaches such as Novelty Detection or Autoencoder are known, for example.

It is also known from the prior art that, on the basis of image data of one type of image recording, an image of the sample can be simulated in another type of image recording. For example, Ounkomol et al. (2018; Nature methods 11: 917) a simulation of a 3D fluorescence image based on a transmitted light image or the prediction of an immunofluorescence image based on an electron microscope image. The quality of the simulation results is assessed on the basis of the degree of correspondence between a simulated image and an actual image of the other image recording type.

The invention is based on the object of proposing a possibility by means of which deviations from structures of a sample and their images can be recognized.

The object is achieved in a first alternative with an image evaluation method in microscopy, in which a first image of a sample is recorded with a first type of image recording and the image values are stored. Using the image values, an image of the sample or the same sample area is predicted in a second type of image recording. In addition, a second image of the sample or of the same sample area is recorded and stored with the second type of image recording.

The method is characterized in that the predicted (henceforth also: simulated) image of the second image recording type and the second image recorded with the second image recording type are compared with one another in order to find differences between them. The differences found are recorded, analyzed and assessed.

The first image recorded with the first type of image recording serves as an output data record for generating a predicted or simulated image. In connection with the invention, the terms of prediction or simulation are understood to be synonymous in the sense of an image or a virtual projection using a mathematical function. The order in which the first or second image is captured is not decisive. The functions, rules or mathematical and / or logical models on which the prediction or simulation is based are also summarized under the term simulation models.

It was recognized that differences between the simulated second image and the second image can be recognized better or even at all if similarities and differences between at least two types of image recording are exploited and taken into account. Differences can be based, for example, on the noise that usually occurs in image and signal processing. Artifacts of the sample treatment, for example the sample preparation, the sample coloring or the sample marking, also represent differences and are also referred to as anomalies. Anomalies are, for example, incorrectly colored cells or cell structures. Differences (abnormalities) can also arise from cells or cell constituents have assumed or not assumed color in an undesired or unexpected manner. Furthermore, a dye that is used can exhibit unexpected behavior, as can be the case, for example, due to the bleaching behavior of a fluorescent dye.

Differences can also be caused by structurally conspicuous sample areas, e.g. B. deformed cell areas may be conditional. If the presence or absence of a known structure is viewed as a difference, this can be recognized and recorded by means of the method according to the invention. As a result, the method according to the invention advantageously differs from known methods such as, for example, auto-encoders or one-class classification.

Differences can be of particular interest to the user. A region of the sample in which differences occur can therefore also be identified and identified as a so-called ROI (region of interest). In the context of this description, anomalies and ROI are simply referred to as differences.

The first and the second type of image recording can, for example, be different contrasting methods and / or different channels of a contrasting method. Different channels can be fluorescence channels, for example, by means of which different structures can be imaged. For example, an image of a DAPI fluorescence channel (DAPI is the abbreviation for a marker for AT-rich region of DNA) can be generated as a simulated second image based on a first image of a fluorescence channel showing fluorescence-labeled microtubules.

The types of image recording from which the simulation is based, or which are a result of a simulation, can be, for example, the contrasting methods brightfield, darkfield, DIC (differential interference contrast) and fluorescence methods. In the latter, structures of the sample are specifically provided with fluorescent markers and their fluorescent light is detected.

The image data of the first and second types of image recording can be, for example, 2D data, 3D data and / or time series data.

In a further possible embodiment of the method, a (first) simulated image can be generated and provided on the basis of the first image. In addition, based on the image of the second type of image recording, a (second) simulated image can be generated and provided. Starting from the second image of the second image recording type, a simulated image can be generated that corresponds to the first image of the first image recording type. The first simulated image and the second simulated image can be mapped onto the image of the second image recording type or onto the image of the first image recording type. In this embodiment of the method according to the invention, similar to a cross-correlation, analyzes are carried out starting from the first image of the first image recording type and starting from the second image of the second image recording type.

A second alternative of the method according to the invention advantageously manages without the acquisition of the image of the second image recording type. Starting from an image of the first type of image recording, a simulated image of the second type of image recording is predicted. This in turn is used as an output for predicting a simulated image of the first type of image recording. In simple terms, predictions are made in a loop. The image of the first type of image recording serving as the output data set is compared with the simulated image of the first type of image recording. The approach of this refinement is that the simulation models used cannot predict occurring anomalies, so that when comparing the image of the first image recording type with the simulated image of the first image recording type There are differences that can be recorded and evaluated. Results of the comparison can also be used to assess the suitability of the simulation model.

The image data of the images in the at least two image recording types can also already be held as stored data and can be called up and processed on request. The same applies to a simulated image based on the image of the first image recording type and optionally for the simulated image of the second image recording type. The image data of the images in the at least two types of image recording can be used, for example, to be improved, reconstructed or converted into a different contrast, optionally by means of machine learning.

For the prediction or simulation of the images, context information, for example on the type of sample, the sample color, the contrast method, etc., can be included in further refinements of the method.

For the analysis and the step of ascertaining differences, in addition to the image of the second image recording type and the simulated image, the image of the first image recording type can optionally also be taken into account.

In further refinements of the image evaluation method, more than two types of image recording can also be used. In this way, the probability of finding differences, in particular of anomalies and / or areas of interest, is advantageously increased or the presence of specific anomalies can be checked through a targeted selection of the types of image recording used.

One of the advantages of the method according to the invention is that the sample can be changed between the recordings of the images of the different types of image recording. It is possible for the sample to be colored differently, for example chemically, in the meantime. One example can be such a staining with flematoxylin-eosin (HE staining).

If differences are found in the course of the comparison, these can be assigned to categories that were previously defined in one embodiment of the method. Such categories are, for example, simulation errors; simulated structures, however, not recorded in the second image; and / or the presence / absence of at least one previously defined structure in the second image. At least one category can also contain those differences that are classified as noise artifacts, for example, and should no longer be taken into account in the further analyzes.

In a further embodiment of the method, a difference image can be generated from the simulated image of the second type of image recording and the second image recorded with the second type of image recording and shown on a display, e.g. B. a monitor, are displayed. This difference image can be evaluated by a user with regard to differences that occur.

However, an automated analysis of the differences found and / or a categorization of differences found by means of image analysis, in particular by means of machine learning, is advantageous.

The prediction or simulation of the simulated image can also take place using machine learning methods. For this purpose, image-to-image images known from the prior art can be used, for example. Such an image is created on the basis of a number N of corresponding image pairs

learned both types of image acquisition. Here, an image of a first type of image recording and l _B denotes an image of a second type of image recording. An image simulated on the basis of an image is also referred to below with the index. The trained model for the image-to-image mapping can then usually mimic the correspondences between the two types of image recording in the training data very well, so the prediction> B is very similar to the true image IB. Cases that deviate greatly from this training data can generally no longer be well covered by the model, and consequently lead to a discrepancy between the simulated image> B and the recorded image IB, which is ultimately the basis of the method according to the invention presented here forms.

As already stated, differences, in particular anomalies, can be recognized automatically by means of image analysis. For example, simple metrics between the simulated image> B and the second image 1 _B can be quantified on the basis of the difference, the amount, at least one correlation, the variance, the application of threshold values, etc. Alternatively, machine learning can be used, for example to use metrics for image evaluation.

In machine learning, a distinction must be made between methods of supervised learning and unsupervised learning. In the case of monitored learning, learning is carried out on the basis of a learning data set consisting of true images, (predicted) simulated images and corresponding annotations with regard to existing differences, in order to recognize these and other differences. Classification models and / or segmentation models can be used. Differences can also be recognized using image data or by calculation.

For example, when a classification model is used, it is possible to predict the occurrence of an anomaly / a difference in an image (yes / no decision). Predictions can also be made about the type of difference (e.g. no difference; type 1; type 2; ...).

If a segmentation model is used, it is possible, for example, to segment the image into areas without differences (“normal”) and into areas with differences (“abnormal”). Alternatively, the image can be segmented into areas which are assigned to predetermined categories of differences / anomaly types (e.g. no difference; type 1; type 2; ...).

Differences can be recognized as such or even with a finer granularity (type 1; type 2; ...) if the image data (detection) are consulted and used for the evaluation.

A difference can be found computationally, for example by calculating regressions, and, if necessary, further specified. For example, the proportion of cells that are affected by an anomaly or the image coordinates of the differences can be used.

In unsupervised learning, for example, clustering algorithms or "classic" anomaly detection, for example by an autoencoder, which uses both the predicted and the true image as input, can be used.

For the machine-learning-based method, the data of the simulated images> B and the data of the second images 1 _{B can} serve as input values. Optionally, the data of the first images as well as manually or automatically determined context information such as the type of sample and information about the contrast methods (type, dyes used, wavelengths, etc.) and / or information about the recording conditions (lens, immersion liquid, etc.) can be entered.

The method according to the invention can additionally or alternatively be used for the detection, analysis and evaluation of differences found on the basis of the comparison of the simulated image ( _{^ B} ) of the second image recording type with the image (l _B ) of the second image recording type for an evaluation of the quality of the simulation model. In particular, a suitability of the simulation model for the prediction of a certain type of image recording and / or a suitability for the imaging of a certain type of sample by checking whether predetermined quality criteria are met.

In order to enable a user of the method according to the invention to interact with the process sequence, a currently achieved training state of the simulation model used to predict the respective simulated image subjected to the test can be determined as a function of a result of the test for compliance with the quality criteria. This determined training status is advantageously shown on a display. Associated with this is preferably a query for the input or selection of at least one control command, by means of which, for example, further iterations of the training sequence of the simulation model in question can be triggered if the current training status is not assessed to be sufficient. In further refinements, the determined current training status can be displayed to the user and, in the event of an insufficient training status, training data of the current sample can be automatically recorded again and the simulation model retrained.

The information made available to the user can also give indications of artifacts to be expected in the respective simulated images, in particular if the training state of the simulation model is classified as unsuitable or only conditionally suitable.

Such an interaction with a user allows him to train models himself. In addition, due to the interaction presented, users without proven expertise in the field of machine learning can also deal with corresponding systems and are protected from incorrect results due to insufficiently trained simulation models.

In order to be able to carry out the image evaluation method according to the invention, a microscope is sufficient that allows image recordings with at least two types of contrast or with at least two channels of one type of contrast. In addition, an evaluation unit, for example in the form of a computer, is required which is configured to carry out the image evaluation method according to the invention. Optionally, a display (screen, monitor, display) can be present on which, for example, the difference image is displayed. In addition, there may be an option for entering data and commands (interface, keyboard, etc.).

The described invention relates to a method which can automatically recognize anomalies or areas of interest in images of a sample. With the method according to the invention, in comparison to previously known anomaly detection methods (e.g. autoencoder, one-class classification), for example, errors in the staining of the sample can also be recognized. Finding and optionally classifying differences such as anomalies and / or areas of interest is a technical challenge that will become even more relevant in the future due to the ever increasing amounts of data.

A great advantage of the method according to the invention is that the simulated image data is compared with actually measured target data (so-called "inference"). However, this comparison makes the evaluation of the respective model quality more precise and reliable. In addition, it is independent of the evaluation In contrast to the method according to the invention, in methods according to the prior art, input data or output data of the model are evaluated directly (commonly referred to as “validation”). It is assessed whether these match the input or output distribution.

The finding of differences according to the method according to the invention takes place on the basis of new and previously unknown image data (inference) by using a previously finally trained model in a workflow. This also means that image data, in which potential differences are to be discovered in the course of the inference by means of the method according to the invention, must not have previously been used to train the simulation model. In contrast to this, in a validation, as in the prior art, the performance of a simulation model is assessed on the basis of data from the training distribution.

The invention is explained below by way of example with the aid of two figures. the

1 shows a microscope and a schematic sequence of a first embodiment of the image evaluation method according to the invention; and

2 shows a microscope and a schematic sequence of a second embodiment of the image evaluation method according to the invention.

A first image of a first type of image, for example in brightfield, is captured by means of a microscope 1, which allows image recordings in at least two types of contrast or additionally or alternatively in at least two channels of one type of contrast. The microscope 1 is connected to an evaluation unit 2 in the form of a computer and a monitor as a display 3. The evaluation unit 2 additionally has a data memory (not shown) and is configured in such a way that, starting from the first image, it _{simulates an image ^ B} by means of a simulation model as if the simulated image I _{A ^ B were} with a second type of image recording, for example with a second type of contrast such as B. as a fluorescent image would have been recorded. The data of the simulated image I _{A ^ B} are stored. In Fig. 1 it is indicated that the structures with little puncturing present in the first image are no longer present _{in the simulated image I A ^ B.} The decision criteria that lead to the omission of these structures can be based on a machine learning process (machine learning).

In addition, a second image _B l is the sample in the second Bildaufnahmeart (eg., Fluorescence image) detected and stored. The simulated image I _{A ^ B} and the second image l _B are then compared with one another, for example by means of the correspondingly configured evaluation unit 2, in order to find differences between them. On the basis of the comparison, a difference _{image I Diff is} generated, for example, in which any differences found are visualized.

The _{difference between the simulated image I A ^ B} and the second image l _B , shown as an example in the difference image _{l Diff,} is a structure provided with network-like hatching in the lower right part of the difference _{image l Diff.} This structure was predicted for the simulated image _{^ B} , can but in the second image l _B actually cannot be confirmed.

The structure embodying the difference cannot be detected with the second type of image recording, for example due to insufficient sample preparation. It could also be a structure type (shown with dotted lines in the image) of the sample which cannot be distinguished from other structure types (also shown with dotted lines in the image) with the first type of image recording. The difference can be categorized manually or - preferably - automatically using existing and retrievable stored prior information and / or using the characteristics of the types of image recording.

The further handling of the found difference can take place depending on its categorization. For example, the difference in the further image evaluation can be ignored, weighted into the further evaluation or explicitly selected and treated as an area of interest (ROI). A second alternative of the method according to the invention comprises the prediction of a simulated image I _{A ^ B of} the second type of image recording by means of a simulation model based on the image of the first type of image recording. The simulated image I _{A ^ B of} the second type of image recording is in turn used in a further step as the basis for predicting a simulated image (IA ^ B)> IA of the first type of image recording. The simulated image (IA ^ B)> IA of the first type of image recording thus obtained is compared with the image I _{A of} the first type of image recording. Existing differences can be recorded and analyzed from the comparison and conclusions can be drawn about the suitability of the simulation model used for predicting the simulated image I _{A ^ B of} the second type of image recording and the simulated image ( _^ B)> IA of the first type of image recording. Compliance with or failure to meet certain quality criteria in advance can serve as a decision-making basis from a comparison. This refinement of the method does not require any actual acquisition of the second image 1 _B , but advantageously uses its virtual equivalent.

Reference number

1 microscope

2 evaluation unit

3 display

IA first image (first type of image recording) l _B second image (second type of image recording)

IA-> B simulated image loi _ff difference image

Claims

1. Image evaluation method in microscopy in which a first image () of a sample is recorded with a first type of image recording and the image values are stored; _{a simulated image (I A ^ B} ) of the sample in a second type of image recording is predicted on the basis of the image values by means of a simulation _{model, and either in a first alternative a second image (l B} ) is recorded and stored with the second type of image recording, the simulated image ( I _{A ^ B} ) of the second type of image recording and the second image (l _B ) recorded with the second type of image recording are compared with one another in order to find differences between them, and differences found are recorded, analyzed and assessed .; or in a second alternative, based on an image () of the first image recording type, a simulated image ( _{^ B} ) of the second image recording type is predicted, based on the simulated image ( _{^ B} ) of the second image recording type, a simulated image (^ B)> IA of the first Image recording type is predicted, the image () of the first image recording type is compared with the simulated image (IA ^ B)> IA of the first image recording type in order to find differences between these, and differences found are recorded, analyzed and evaluated.

2. The method according to claim 1, characterized in that image data of the images (IB) in the at least two image recording types and / or image data of the simulated image (I _{A ^ B} ) of the second image recording type and / or the simulated image (IA ^ B) > IA of the first type of image recording are held as stored data and can be called up and processed on request.

3. The method according to any one of the preceding claims, characterized in that the differences found are assigned to previously defined categories.

4. The method according to claim 3, characterized in that the categories simulation errors, simulated, but in the first alternative in the second image (l _B ) or in the second alternative in the image () of the first image recording type not recorded structures and / or the presence at least a previously defined structure in the second image (l _B ) or in the first image () are set.

5. The method according to any one of the preceding claims, characterized in that different contrasting methods and / or different channels of a contrasting method are used as image recording types.

6. The method according to any one of the preceding claims, characterized in that the first simulated image ( _{^ B} ) with the image (l _B ) of the second image recording type or the second simulated image (I _{B ^ A} ) with the image () of the first image recording type or the first image () is compared with the simulated image (I _{A ^ B} )> I _{A of} the first image recording type, taking into account a selected metric.

7. The method according to any one of the preceding claims, characterized in that an analysis of the differences and / or a categorization of found differences takes place by means of machine learning.

8. The method according to any one of the preceding claims, characterized in that the prediction of the first simulated image (I _{A ^ B} ) of the second type of image recording and / or of the second simulated image (I _{B ^ A} ) of the first type of image recording or of the simulated image (I. _{A ^ B} )> I _{A of} the first type of image recording takes place by means of machine learning.

9. The method according to claim 7, characterized in that an analysis and / or categorization by means of at least one method from a group comprising classification, segmentation, detection and regression as methods of monitored machine learning and cluster analysis and classic anomaly detection as methods of unsupervised learning is performed.

10. The method according to claim 1, characterized in that in addition or as an alternative to the detection, analysis and evaluation of differences found on the basis of the comparison of the simulated image (I _{A ^ B} ) of the second image recording type with the image (l _B ) of the second image recording type or using the Comparison of the simulated image (I _{A ^ B} )> I _{A of} the first type of image recording with the image () of the first type of image recording an evaluation of the quality of the simulation model is carried out by checking whether predetermined quality criteria are met.