EP3608828A1 - Method for providing an evaluating means for at least one optical application system of a microscopic application technology - Google Patents

Abstract

The invention relates to a method for providing an evaluation means (60) for at least one optical application system (5) of a microscope-based application technology, the following steps being carried out, in particular in each case by an optical training system (4): - Performing an input detection (101) of at least one Sample (2) according to the application technology in order to obtain at least one input record (110) of the sample (2) on the basis of the input detection (101), - performing a target detection (102) of the sample (2) according to a training technology in order to use the target detection ( 102) to obtain at least one target record (112) of the sample (2), the training technology differing from the application technology at least in that additional information (115) about the sample (2) is provided, - teaching (130) the evaluation means ( 60) at least based on the input record (110) and the target record (112) to receive learning information (200) from the evaluation means (60) that different sample positions are automatically recorded during the input detection (101) and / or during the target detection (102), so that in particular the learning information (200) for a continuous sample relative shift to determine the additional information (115) is taught.

Description

The present invention relates to a method for providing an evaluation means for at least one optical application system of a microscope-based application technology. The invention further relates to a system, a computer program and a computer-readable medium.

It is known from the prior art that a so-called ground truth for training learning methods from the area of machine learning is obtained by complex methods. In particular, a segmentation of samples to determine the ground truth can be very complex and often only done by manual intervention. For example, data from various data sources often has to be brought together and processed in a complex manner.

It is therefore an object of the present invention to at least partially remedy the disadvantages described above. In particular, it is the object of the present invention to propose an improved possibility of using teaching methods in microscopic systems.

The above object is achieved by a method with the features of claim 1, a system with the features of claim 12, a computer program with the features of claim 13, a computer program with the features of claim 14 and by a computer-readable medium with the features of 15. Further features and details of the invention emerge from the respective subclaims, the description and the drawings. Features and details that are described in connection with the method according to the invention apply, of course, also in connection with the system according to the invention, the computer programs according to the invention and the computer-readable medium according to the invention, and vice versa, so that there is always a reciprocal reference to the individual aspects of the invention is or can be taken.

The object is achieved in particular by a method for providing (at least) one evaluation means for at least one optical application system of a microscope-based application technology.

• Durchführen einer Eingabeerfassung wenigstens einer Probe gemäß der Anwendungstechnologie, um anhand der Eingabeerfassung wenigstens eine Eingabeaufzeichnung der (wenigstens einen) Probe zu erhalten,
• Durchführen einer Zielerfassung der (wenigstens einen) Probe gemäß einer Trainingstechnologie, um anhand der Zielerfassung wenigstens eine Zielaufzeichnung der (wenigstens einen) Probe zu erhalten, wobei vorzugsweise die Trainingstechnologie sich von der Anwendungstechnologie zumindest dadurch unterscheidet, dass eine Zusatzinformation über die (wenigstens eine) Probe durch die Trainingstechnologie bereitgestellt wird, und vorzugsweise aus der Zielaufzeichnung ermittelt wird,
• Anlernen des Auswertungsmittels zumindest anhand der Eingabeaufzeichnung und der Zielaufzeichnung, um wenigstens eine Anlerninformation des Auswertungsmittels zu erhalten.

It is particularly provided that the following steps are carried out, preferably in each case by an optical detection system, in particular a training system:

• Performing an input acquisition of at least one sample according to the application technology in order to obtain at least one input record of the (at least one) sample based on the input acquisition,
• Performing a target acquisition of the (at least one) sample according to a training technology in order to obtain at least one target record of the (at least one) sample based on the target acquisition, wherein the training technology preferably differs from the application technology at least in that additional information about the (at least one) Sample by the training technology is provided and is preferably determined from the target recording,
• Teaching the evaluation means at least on the basis of the input recording and the target recording, in order to obtain at least learning information from the evaluation means.

The preceding steps can be carried out repeatedly in any order or in the order specified and, if necessary, individually. The steps are preferably carried out completely or predominantly automatically, particularly preferably without manual intervention.

It can be provided that the training technology differs from the application technology at least in that additional information about the (at least one) sample is provided by the training technology (i.e. is determined or is available). This means that in the application system, which provides the application technology in an identical or similar manner to the training system, the additional information is not determined on the basis of the application technology, but is possibly only determined at least approximately on the basis of the evaluation means. The application technology may or may not be able to determine the additional information. For example. the determination of the additional information by the application technology is more time-consuming than by the training technology, so that the additional information in the method according to the invention is not provided by the application technology.

The application technology in the training system z. B. (unlike the application system) can also be implemented in that it is synthetically emulated and / or provided by a neural network. This also makes it possible to flexibly use different application technologies such. B. software-assisted in the training system, for various implementations of teaching.

It can be advantageous in a method according to the invention if different sample positions are automatically detected during the input detection and / or during the target detection, so that in particular the learning information for a continuous sample relative displacement is learned to determine the additional information becomes. The different sample positions can e.g. B. can be obtained by an automated stepwise and / or continuous sample relative displacement (also: relative sample movement). The learning to determine the additional information is particularly suitable for determining the learning information in such a way that the evaluation means taught in this way at least approximately determines the additional information in a subsequent evaluation by the application system. For this purpose, for example, methods of machine learning are used for teaching and / or acquisition in accordance with the application technology by the application system to determine input data for the evaluation.

A sample relative displacement (in particular executed as a sample movement and / or detection system movement to change the sample position) can be understood as a displacement of the sample relative to the detection system (e.g. to the training and / or application system, in particular to a detection optics and / or detection device) become. This can e.g. B. caused by the fact that the detection device and / or the detection optics is moved relative to the sample, or vice versa. Accordingly, a relative movement of the sample can also be understood as a movement in which either the sample moves in relation to the detection system, or vice versa. Of course, both the sample and the detection system can also be moved. It is particularly important that the sample area that is recorded changes. The use of a motorized sample table for direct movement of the sample to achieve the sample relative displacement has z. B. the advantage that a large number of samples or objects of the sample can be acquired over a wide range and less mass has to be moved if necessary for a light sample. The movement of a part of the detection system, such as the detection optics or the detection device, has the advantage that the sample itself is not exposed to any accelerations and is therefore particularly advantageous in the case of a detection device of low mass. Both can be done manually, if necessary, to reduce the cost of motorization to avoid.

In particular, the acquisition information for a continuous sample relative displacement, in particular sample displacement or sample movement, for determining the Additional information learned. This means that the teaching takes place in such a way (e.g. by using the sample relative shift and / or evaluation of the different sample positions) that the evaluation means taught in this way leads to a (subsequent and / or renewed) continuous sample relative shift (or, preferably, at least one other sample) can be used to at least approximately determine the additional information. This continuous sample relative shift takes place, for example, in an application system for at least one further sample after the teaching has been completed and / or the taught-in evaluation means is provided for the application system. The continuous sample relative displacement is to be understood as a detection method in which a sample is moved continuously, so that the detection, such as movement blur, can be influenced by the movement, in particular depending on the movement speed and the detection speed. Advantageously, the continuous sample relative displacement is carried out in the application system for samples which are designed differently from the at least one sample recorded in the training system, but may be of the same design. Alternatively, it can also be possible that only a step-wise sample relative shift or no sample relative shift is carried out in the application system. In this case, there is nevertheless an advantage due to the (stepwise or continuous) sample relative displacement in the method according to the invention or in the training system, since this enables a large number of training data to be determined in a short time.

It may also be possible that the sample position is changed and / or a sample relative shift takes place before each input and target acquisition. For example. there is first a first input acquisition and target acquisition at a first position of the sample, then a second input acquisition and target acquisition at a second position of the sample, etc. Only after all input and target acquisitions have been carried out at the various sample positions can the step of teaching be carried out, if necessary become.

The teaching takes place z. B. in that the learning information is determined for the evaluation means. This can advantageously be done using methods of machine learning, in particular deep learning. Accordingly, the learning information can be a classifier or a model or a weighting for the evaluation means. It can it is therefore sufficient to provide the learned evaluation means for the application system that the learning information is transmitted to the application system by a data exchange. This is possible in particular if the application system already has an evaluation means or a computer program similar to it, which can use the learning information and is therefore also parameterized to a “learned” evaluation means on the basis of the transmitted learning information.

An optical detection system, such as the application and / or training system, can provide sample detection by an optical enlargement method for cell and / or sample display and / or by microscopy and / or by flow cytometry. As an example, the detection system can be designed as a microscope system for transmitted light microscopy. In particular, the application and / or training system can also only provide microscopy and / or no flow cytometry.

A sample can be acquired by using the acquisition system to determine at least one item of information on the sample. The detection information can e.g. B. be a training system-side input recording or (possibly of the same type) application system-side acquisition information. This can then be evaluated by means of a previously learned evaluation using an evaluation means (e.g. using methods of machine learning and / or deep learning methods and / or artificial intelligence and / or neural networks), in particular in an application system, in order to provide result information about the To determine sample. Depending on the training (training) of an evaluation means for evaluation (if sufficient training data is used), the result information can provide at least approximately additional information about the sample, which would otherwise only be accessible with considerably complex methods. The result information can thus be designed as additional information about the sample or estimate or predict such additional information. For example. the evaluation can also provide an evaluation method such as a sample specific assay.

The acquisition information is, for example, an image recording of the sample. The image data of the image recording can be used particularly advantageously in combination with the described evaluation, in particular if the evaluation uses methods of supervised learning and large annotated training data are available for teaching. The training data can e.g. B. input records and target records, such as acquisition information from microscopy and / or flow cytometry. In particular, the training data can be generated automatically (possibly completely) without the need for manual intervention for annotation or the like. This opens up the possibility of automating the evaluation means (e.g. an artificial neural network) and thus using large amounts of training data to save time. The evaluation means can also include a system of neural networks.

For training purposes, it may be necessary that acquisition information according to an application technology (hereinafter briefly: application data, in particular an input record) and acquisition information according to a training technology (hereinafter briefly: expert data, in particular a target record) are available. The application data can also come from a different source than the expert data. It is also conceivable that the application and expert data have information about the same sample and / or the same sample area. In particular, there is information overlap in both. The application data can have such acquisition information as is also acquired in an application system. The expert data may not be able to be recorded in the application system, or only in a lower quality and / or more slowly. It is also conceivable that the expert data cannot be recorded, or not in the same way, in the application system, but possibly only in a training system. Both systems can be designed as acquisition systems, such as microscopes. The application data include, for example, a stack of transmission images. The expert data, however, can e.g. B. fluorescence images of the sample. By means of teaching, the evaluation means can possibly be taught to augment additional information from the expert data in the application data. In other words, by evaluating the application data without knowing the expert data, the result information can be determined, which then at least approximately Has information of expert data. This can also be possible if the application system captures this application data using the application technology without using the training technology. The application technology is e.g. B. a transmitted light microscopy and / or a multi-colored microscopy, in particular living and / or unstained cells of the sample, and / or the like. The training technology is e.g. B. fluorescence microscopy and / or segmentation of the sample, in particular fixed and / or stained cells, and / or the like. Both the expert data and the application data can advantageously be recorded automatically (for example when teaching in by the training system) without manual intervention. In contrast to the application system, the training system can use both the application and the training technology to record the acquisition information. It can be decisive that the recording of the expert and / or application data takes place at a sufficient speed to determine a sufficient amount of data for teaching. The training can be based on the expert data (accordingly as ground truth) and application data. B. take place in such a way that learning information is determined by means of machine learning. This teaching information has z. B. a classifier and / or a model and / or a weighting and / or a parameterization of the evaluation means, which can thus represent the result of teaching. The learning information can then be used by the evaluation means (such as a computer program) in application systems in order to carry out the learned evaluation on the basis of the application data determined by the application system and / or to determine the result information. For this purpose, the evaluation means is carried out, for example, by an evaluation device, such as a computer.

In particular, the described method is not restricted to (classic) transmitted light microscopy as application technology or training technology. For example, the application technology or training technology can also be designed as a flow cytometry. For cell analysis in flow cytometry, it can be provided that samples in a liquid are guided past an optical device consisting of detectors (e.g. sensors of a detection device) and / or illumination in order in this way to obtain sample information (e.g. e.g. the input record and / or the acquisition information). An advantage of this continuous relative displacement between the sample, or cells of the sample, and the optical detection device is the comparatively high sample throughput due to the possibility of running the illumination and / or the recording by the detector without significant technical disadvantages. When implemented in a microscopic setup for adherent cells, there is the advantage of lower acceleration forces, which plays a role in particular in the careful observation of living cells and the avoidance of vibrations of the sample liquid. Because bpw. in transmission lighting methods, the latter can lead to irregular lighting changes due to surface waves.
It may be possible that the method according to the invention is limited to microscopy, preferably transmitted light microscopy, as application technology and / or training technology. In other words, the application technology and / or training technology can only relate to microscopy technologies and z. B. not on flow cytometry.

Conventionally, it is complex to obtain structural information and / or images and / or analyzes derived from such a continuously moving sample. In particular, the sample throughput is considerably reduced in spite of the stronger parallelization by the detection device, possibly using several sensors for detection. Here, the advantage can be achieved according to the invention that the training for the evaluation means reduces the effort for this determination, for. B. motion blur is reduced by means of a neural network. Further neural networks of the evaluation means and / or further evaluation means can also be trained and used in the evaluation to determine structural information and / or images and / or analyzes derived therefrom as additional information

It may also be possible for the sample or the cells of the sample for the recording of images (input recordings and / or acquisition information) or the sample analysis (based on the input recordings and / or acquisition information) to be continuously shifted relative to the detection device and analyzed in the meantime by one Get speed gain. In addition to flow cytometry, this can also be provided when using microscopy as application and / or training technology. For example, flash lighting is used as one Illumination of the sample used, and / or a linear sensor used as at least one sensor of the detection device. In microscopy, such a method using camera sensors can be complex since the movement of the sample has to be synchronized with the flash light illumination and the camera recording, in particular at high speeds. According to the invention, it can be an advantage here if an evaluation means is taught to determine the additional information, which is provided by target recordings with the more extensive lighting situation, from detection information determined by less extensive lighting situations (e.g. without flash light illumination and / or fewer sensors). This means that the sample can be illuminated continuously as well as recorded non-synchronized, which can greatly reduce the demands on the technology and allow higher camera speeds to be achieved.
In addition, the continuous movement has the advantage that a fixed lighting pattern, for example a stripe pattern, can produce an increase in resolution in the direction of movement in the sense of structured lighting without the need for a further device for moving the lighting pattern.

It is also conceivable that the teaching is carried out, in particular, automatically on a sample type with a specific color and / or a specific recording method, in particular for the development of sample-specific assays and analyzes. However, the use of conventional dedicated algorithms can be problematic. Machine learning methods, on the other hand, can be able to solve such problems, such as motion blur, more robustly than dedicated algorithms with appropriately extensive ground truth. Recordings of the sample taken at a high frame rate can serve as expert data in order to train the evaluation means (like a neural network) in such a way that it can calculate the motion blur better than conventional methods from a video with a lower frame rate than application data.
The method can be carried out regardless of the advantages of calculating the motion blur by teaching but also on the basis of the motion speeds using classic methods of motion blur correction without the advantages of automated Learning by means of ground truth and increased speed by continuously recording on the evaluation systems.

The invention described here can be used to provide a system and / or a method which, for a specific sample type, allows both automated teaching (in particular automated training of the image or analysis function) using training technology, if appropriate also by means of in silico labeling, in order to preferably in a further type of recording (application technology) to allow analysis of samples at high speed for a specific question on the sample type. The latter recordings or analyzes can also be carried out on technically simpler application systems which do not support the more complex training technology for training.

The so-called "In Silico Labeling" method is described in the publication " In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images ", Christiansen, Eric M. et al. Cell, Volume 173, Issue 3, 792-803 disclosed. These are fluorescent markings carried out by computer processing ("in silico"). It is conceivable for the input and / or target recording to have fluorescence markings carried out in this way in a method according to the invention.

This may also have the advantage that microscopic image analyzes using methods that are conventionally technically difficult to implement can be mapped onto optical (in particular camera-based) methods, which can be technically simpler. The evaluation means (such as a neural network or a system of neural networks) can in particular be designed so that it can be trained on expert data obtained through a training system, but can then be used on technically simpler application systems. As a result of the special training methods, it is also possible for the evaluation means to carry out corresponding analyzes on a manual microscope by manually shifting the sample, whereby, in contrast to conventional methods, the data can also be robustly evaluated during the continuous shift.

For the evaluation and / or teaching, it is also conceivable that external neural networks are used to deliver the expert data.

The sample can advantageously be designed as a cell sample, in particular with adherent cells on tissue sections or in so-called microtiter plates (well plates). According to the invention, these can be analyzed at a very high speed and / or without mechanical problems, which can result from step-by-step execution with mechanical vibrations or from the inaccuracies in the direction and speed of movement when the sensor or the sample (s) are moved manually.

Furthermore, it is optionally provided that the input acquisition differs from the target acquisition at least in that the input recording is influenced by an in particular relative movement (relative movement, for example, relative to the acquisition system) of the sample during the input acquisition, so that the learning information for at least a reduction this influence and in addition for the determination of the additional information is specific. The training for reducing this influence can take place, for example, by the evaluation means having a neural network or the like, which uses the input record (which has the influence) as application data and the target record (which does not have the influence) as expert data or ground Truth is being trained. This results in an adaptation of the learning information (for example, as an adaptation of the neural network), which is specific for the reduction of this influence. In order to additionally enable the additional information to be determined by the learning information, a further neural network or the like of the evaluation means can optionally be trained. For this, e.g. B. the further neural network or the like can be trained with at least one additional input record (without the additional information) as application data and at least one further target record (with the additional information) as expert data or ground truth. All the networks trained in this way can be linked to one another during the evaluation in order to carry out the evaluation cascaded. The adaptation of these neural networks can be provided in the evaluation information as one or more data records in order to be able to provide them in a simple manner.

It can preferably be provided within the scope of the invention that an automated step-by-step or continuous sample relative displacement, in particular relative movement of the sample, for which the learning information is specifically taught, is carried out during the input acquisition and / or the target acquisition. By introducing the relative movement into the application data, teaching can thus be carried out in a reliable manner in order to take the movement into account in the evaluation. The learning information is therefore specific because it takes the movement into account when evaluating the application system.

In a further possibility it can be provided that in the input acquisition and in the target acquisition a (possibly completely) automated step-by-step or continuous, in particular relative, movement of the sample is carried out, in particular for a large number of objects of the sample and / or for to determine a large number of samples in each case at least one input and target recording as training data for teaching, wherein preferably a ground truth or expert data for teaching is or are determined automatically on the basis of the target recording, in particular by reference processing, preferably by automatic segmentation. The reference processing can e.g. B. by using a threshold method (thresholding) in the target recording to reliably and / or in an automated manner z. B. to perform a masking (segmentation) of cells.

Furthermore, it is optionally provided that the target recording and / or the input recording each have recordings of the sample at the different sample positions, preferably during the input detection and / or during the target detection during a motorized, automated continuous or step-wise, in particular relative, displacement of the sample Position data are determined via the sample positions in order to assign the recordings to the sample positions on the basis of the position data. In this way it is possible to implement a particularly fast and in particular continuous mode for acquiring the sample.

If the input and / or target acquisition and / or the application system acquisition takes place in continuous mode, it is conceivable that one or more Input recordings and / or target recordings and / or acquisition information are recorded at least partially in parallel and / or simultaneously and / or sequentially in the at least one sample during an, in particular relative, movement of the at least one sample. For this purpose, e.g. B. different channels of the optical detection system can be used. The various input and / or target recordings and / or acquisition information can also be records according to different technology or application and / or training technology. For example. transmission and fluorescence images can be determined at least partially at the same time. Furthermore, the acquisitions for the respective input and / or target recordings and / or acquisition information can be carried out repeatedly. In order to combine these repeated acquisitions again for a corresponding recording and / or acquisition information, the current position data of the at least one sample, for example, are simultaneously acquired during the (relative) movement. This allows z. B. It may be possible to determine a large amount of training data for teaching at high speed. It may also be possible for the input acquisition and the target acquisition to be carried out as an acquisition on the training system side by an optical training system, the learning information preferably being specific for the determination of the additional information on the basis of an application system-side acquisition information, which is determined by a subsequent application system acquisition (application system input acquisition) by an optical application system. With this detection on the application system side, a (particularly manual) continuous sample relative displacement can also be carried out. Alternatively or additionally, the application system can be designed to provide the application technology, and the training system can additionally (in addition to providing the application technology) be configured to provide the training technology. This enables a reliable and quick determination of training data for teaching.

It may further be possible that the evaluation means is designed as a computer program based on machine learning and / or artificial intelligence and / or deep learning, and preferably has one or more artificial neural networks, in particular according to a training hierarchy, preferably a first of the networks for Determination of the additional information is learned and at least a second one of the networks has been taught in to compensate for the continuous sample relative displacement. This enables extensive sample analyzes to be carried out in continuous mode. The training hierarchy relates in particular to an evaluation by the networks in cascade form. To teach the network to compensate for the continuous sample shift, it may also be possible to use ground truth from stuck visual fields from a step-by-step or image generated using a flash lamp without motion blur. In the case of a z movement or a movement in several spatial directions, this ground truth would possibly have to be expanded by corresponding recordings from the z planes.

Alternatively, a classic method can be used here instead of the second network to compensate for the continuous relative sample shift within the system. For example. a conversion could take place here using the known movement speeds.

It is also possible for neural networks to be taught in parallel. For this purpose, for example, a first of the networks is taught in parallel to a second of the networks. As an alternative or in addition, a neural network can be learned in particular in accordance with a "multitask learning" in such a way that it provides a number of result information items from the (application system side) information (e.g. at least one additional item of information and / or further information such as information about a Compensation for a continuous sample relative displacement).

In a further possibility, it can be provided that when teaching, several neural networks of the evaluation means for respective processing of the input recording are taught in, and in particular the evaluation means is trained by the learning information to carry out at least two sample analyzes, each of which one of the neural networks of the specific purpose Evaluation means is assigned. In other words, the networks can be taught to carry out the corresponding sample analyzes. Such sample analyzes are, for example, segmentation and / or substance detection in the sample or the like.

It is also conceivable that the training technology and / or the application technology are each designed as microscopy technologies, in particular for transmitted light microscopy, preferably that the training system and / or an application system are each designed as microscopes, preferably for flow cytometry.

• bei der Eingabeerfassung die Probe in einem ersten Aufnahmemodus und/oder mit einer ersten Beleuchtungsintensität relativ zu dem Trainingssystem, insbesondere einer Erfassungsoptik (oder auch Abbildungsoptik), über mehr als ein gesamtes Detektionsfeld der Detektionsvorrichtung kontinuierlich oder schrittweise verschoben wird, um die Eingabeaufzeichnung eines Bereichs der Probe zu erhalten,
• bei der Zielerfassung zumindest teilweise der gleiche Bereich der Probe in einem zweiten Aufnahmemodus und/oder mit einer zweiten Beleuchtungsintensität erfasst wird, um die Zielaufzeichnung zu erhalten, wobei sich vorzugsweise der zweite Aufnahmemodus und/oder die zweite Beleuchtungsintensität von dem ersten Aufnahmemodus bzw. der ersten Beleuchtungsintensität unterscheidet.
• bei dem Anlernen aus der Zielaufzeichnung ein Ground Truth bestimmt wird und aus der Eingabeaufzeichnung Anwendungsdaten (Targetdaten) bestimmt werden, welche zum Anlernen wenigstens eines ersten neuronalen Netzes (des Auswertungsmittels) verwendet werden, um die Anlerninformation zu bestimmen,
• optional: Anlernen (Trainieren) wenigstens eines weiteren neuronalen Netzes (des Auswertungsmittels) mittels des ersten neuronalen Netzes, um die Anlerninformation zu bestimmen, welche für ein neuronales Netzsystem sämtlicher trainierter neuronaler Netze (des Auswertungsmittels) spezifisch ist,
• Anwenden des Auswertungsmittels mit der Anlerninformation bei einem Anwendungssystem zur anwendungssystemseitigen Erfassung in dem ersten Aufnahmemodus und/oder mit der ersten Beleuchtungsintensität, wobei die Erfassung insbesondere dazu dient eine anwendungssystemseitige Erfassungsinformation zu erhalten, wobei hierzu vorzugsweise wenigstens eine Probe kontinuierlich über mehr als ein gesamtes Detektionsfeld einer weiteren Detektionsvorrichtung des Anwendungssystems verschoben wird.

It is also advantageous if at least one detection device and at least one illuminating means (such as at least one light source) of a training system are used for input and / or target detection (each or together) within the scope of the invention, wherein

• during the input detection, the sample is shifted continuously or step by step over more than an entire detection field of the detection device in a first recording mode and / or with a first illumination intensity relative to the training system, in particular detection optics (or also imaging optics), in order to record an area of the input To get sample
• in the case of target detection, the same area of the sample is recorded at least partially in a second recording mode and / or with a second lighting intensity in order to obtain the target recording, the second recording mode and / or the second lighting intensity preferably being different from the first recording mode or the first Illumination intensity differs.
• when teaching from the target recording, a ground truth is determined and from the input recording application data (target data) are determined which are used for teaching at least one first neural network (of the evaluation means) in order to determine the teaching information,
• optional: teaching (training) at least one further neural network (of the evaluation means) by means of the first neural network in order to determine the learning information which is specific for a neural network system of all trained neural networks (of the evaluation means),
• Use of the evaluation means with the learning information in an application system for detection on the application system side in the first recording mode and / or with the first illumination intensity, the detection being used in particular to obtain information on the application system side, preferably at least one for this purpose Sample is continuously moved over more than an entire detection field of a further detection device of the application system.

The acquisition mode can relate, for example, to a continuous or step-by-step or manual mode of the sample relative displacement. The recording mode and / or the lighting intensity can also be implemented in accordance with a training or application technology.

For example, it can be provided that the method steps of a method according to the invention for input and target acquisition, and in particular also for the application system acquisition, of various samples and / or objects of the sample are carried out completely automatically, so that any manual intervention can be dispensed with here. This enables the sample to be recorded particularly quickly.

A tracking system for the continuous motorized tracking of the position of the sample, in particular the relative optical z position between the sample and the detection device (in particular detection sensor) can also be used in the continuous recording. An advantageous embodiment is such. B. a method that has been trained for this purpose, which, from the training of z-stacks (several sample recordings in the z-direction, that is to say for different depths of the sample) and the ground truth of the corresponding focused positions within the z-stacks, form a motorized z-drive of the objective so that during a continuous displacement the sample is always kept at an optical distance from the detection system that is as constant as possible.

In a further possibility it can be provided that different depths of the sample are recorded during the input and target acquisition, and in particular also during the application system acquisition, so that the input recording and / or target recording a stack (z-stack) of recordings of the sample ( for different sample positions in the axial z direction). Just like a lateral sample relative displacement (in the x and / or y direction), a continuous sample relative displacement can also be provided orthogonally to this in the z direction.

A particularly advantageous embodiment of acquisition information on the application system side is an optical design on the recording side of the acquisition system, which allows simultaneous imaging of different z-planes (i.e. depth planes of the sample) on a sensor (the detection device). This could be done, for example, by a sensor tilted to the optical solder. As a result of the continuous movement, the sample can then be detected in different z-planes as well as carried out at different points in the optics with different optical properties, with the different sample sections detected thereby preferably experiencing the same optical detection conditions as a result of the shift in the z-planes. Another advantage of this version is the reduced requirements for focus tracking of the system in the z direction. The sensor can e.g. B. can be designed as a monochrome camera sensor, a polarization-resolving camera sensor or as an RGB camera sensor, in the latter cases a polarization-coded or color-coded illumination can provide additional information about the sample.

Furthermore, it is possible within the scope of the invention that different depths of the sample are detected during the input and / or target detection and / or during the application system-side detection (of the application system), so that preferably the input recording and / or the target recording and / or a ( Application system side) detection information of the application system side detection has a stack with recordings of the sample for axial sample positions.

As an alternative or in addition, it is possible for the different depths of the sample to be detected by, in particular through the application system (and / or the training system), sample areas of the different depths (depth areas) on different areas of a sensor of a detection device of the application and / or training system. Different positions of the sample in the z direction (depth direction of the sample) can thus be imaged on the sensor at the same time. In order to record further positions of the sample, the sample relative shift can also take place in particular as a continuous shift, while further recordings are carried out. In order to enable the simultaneous imaging of the different depth ranges, the sensor can preferably, in particular with respect to an optical axis in the application system (or training system) a plumb line of the optical axis. The mapping can also be made possible by adapting an imaging optics of the application system (or training system) accordingly. In particular, the sensor is designed as an area sensor and / or camera sensor.

It is also conceivable that the input acquisition and / or target acquisition and / or further target acquisitions and / or further input acquisitions for teaching through different channels of the training system are carried out simultaneously or at different times for different sample positions.

• adhärente Zellen auf Gewebeschnitten,
• Proben auf einer Mikrotiterplatte,
• gefärbte Zellen,
• ungefärbte Zellen.

It may also be possible that the evaluation means is adapted by the learning information to analyze at least one of the following samples:

• adherent cells on tissue sections,
• Samples on a microtiter plate,
• colored cells,
• unstained cells.

• eine Detektion, insbesondere Segmentierung oder Lokalisierung von Objekten und/oder Zellen der Probe, insbesondere eine Segmentierungsmaske,
• eine Schätzung der Klassifizierung oder Regression von Pixel oder Objekten oder Probenbereichen der Probe,
• ein zytometrisches oder histologisches Analyseergebnis (z. B. für den Ground Truth, gewonnen mit etablierten Methoden der Probenanalyse bspw. High Content Analyse),
• eine Phaseninformation (z. B. für den Ground Truth aus einer komplementären, etablierten Methode zur Phasenmessung),
• eine dreidimensionale Information über die räumliche Struktur der Probe (z. B. ist der Ground Truth eine z-Stapel-Aufnahme),
• wenigstens eine Fluoreszenzeigenschaft der Probe (z. B. ist der Ground Truth wenigstens ein Fluoreszenzbild der Probe),
• ein erweiterter Schärfetiefenbereich einer Probenaufzeichnung (z. B. ist der Ground Truth ein entsprechendes Bild mit erweitertem Schärfetiefenbereich der Probe).

A further advantage within the scope of the invention can be achieved if the additional information (or result information) and / or a ground truth for teaching an evaluation means is one of the following information about the sample:

• detection, in particular segmentation or localization, of objects and / or cells of the sample, in particular a segmentation mask,
• an estimate of the classification or regression of pixels or objects or sample areas of the sample,
• a cytometric or histological analysis result (e.g. for ground truth, obtained with established methods of sample analysis, e.g. high content analysis),
• phase information (e.g. for ground truth from a complementary, established method for phase measurement),
• three-dimensional information about the spatial structure of the sample (e.g. the ground truth is a z-stack image),
• at least one fluorescence property of the sample (e.g. the ground truth is at least one fluorescence image of the sample),
• an extended depth of field of a sample recording (e.g. Ground Truth is a corresponding image with an extended depth of field of the sample).

• zumindest eine Detektionsvorrichtung zum Durchführen einer Eingabeerfassung wenigstens einer Probe gemäß der Anwendungstechnologie, um anhand der Eingabeerfassung wenigstens eine insbesondere digitale Eingabeaufzeichnung der Probe zu erhalten,
• die oder wenigstens eine weitere Detektionsvorrichtung zum Durchführen einer Zielerfassung der Probe gemäß einer Trainingstechnologie, um anhand der Zielerfassung wenigstens eine, insbesondere digitale, Zielaufzeichnung der Probe zu erhalten, wobei vorzugsweise die Trainingstechnologie sich von der Anwendungstechnologie zumindest dadurch unterscheidet, dass die Trainingstechnologie zur Bereitstellung einer Zusatzinformation über die Probe ausgeführt ist,
• eine Auswertevorrichtung zum Anlernen des Auswertungsmittels zumindest anhand der Eingabeaufzeichnung und der Zielaufzeichnung, um eine Anlerninformation des Auswertungsmittels zu erhalten,
• eine Probenbewegungsvorrichtung, wie ein motorisierter Probentisch, zur (lateralen und/oder axialen) relativen oder direkten Verschiebung der Probe, um bei der Eingabeerfassung und/oder bei der Zielerfassung verschiedene Probenpositionen automatisiert zu erfassen, sodass insbesondere die Anlerninformation für eine kontinuierliche Probenrelativverschiebung (insbesondere bei dem Anwendungssystem) zur Ermittlung der Zusatzinformation angelernt wird.

The invention also relates to a system for providing an evaluation means for at least one optical application system of a microscope-based application technology, in particular according to a method according to the invention, comprising:

• at least one detection device for performing an input detection of at least one sample according to the application technology in order to obtain at least one, in particular digital input recording of the sample based on the input detection,
• the or at least one further detection device for performing target acquisition of the sample according to a training technology, in order to obtain at least one, in particular digital, target record of the sample based on the target acquisition, wherein the training technology preferably differs from the application technology at least in that the training technology is used to provide a Additional information about the sample is executed,
• an evaluation device for teaching the evaluation means at least on the basis of the input recording and the target recording in order to obtain learning information from the evaluation means,
• a sample moving device, such as a motorized sample table, for (lateral and / or axial) relative or direct displacement of the sample in order to automatically record different sample positions during input acquisition and / or target acquisition, so that in particular the learning information for a continuous sample relative displacement (especially with the application system) is learned to determine the additional information.

The system according to the invention thus brings with it the same advantages as have been described in detail with reference to a method according to the invention.

It is possible that the training technology differs from the application technology with regard to the detection device and / or detection technology and / or lighting device and / or lighting technology used.

The invention also relates to a computer program, preferably a computer program product, comprising commands which, when the computer program is executed by a computer (such as an evaluation device), cause it to determine additional information about a sample on the basis of at least one input recording of the sample and on the basis of teaching information. In particular, it is provided that the input recording is specific for an input acquisition of the sample according to an application technology and / or the learning information is determined according to a method according to the invention. The computer program according to the invention thus brings with it the same advantages as have been described in detail with reference to a method according to the invention. In addition, the computer program can be suitable for being stored in a non-volatile manner in a data memory of an evaluation device, preferably a system according to the invention. The computer program can e.g. B. be produced by a method according to the invention.

A computer program, preferably a computer program product, in particular evaluation means, is also protected, produced by a method according to the invention, so that the computer program has the learning information.

Also protected is a computer program, preferably a computer program product, which has commands which, when the computer program is executed by a computer (such as an evaluation device), cause the computer to carry out the steps of a method according to the invention. In addition, the computer program can be suitable for being stored in a non-volatile manner in a data memory of an evaluation device, preferably a system according to the invention.

Each of the computer programs can e.g. B. can be read by a processor of an evaluation device from the data memory, and then by the processor be carried out. For example. For example, each of the computer programs can be suitable for controlling a sample movement device for relative sample displacement, and in particular for capturing and / or storing the position data of the sample. Each of the computer programs and / or a computer-readable medium with the computer program can be designed to be non-transitory.

The invention also relates to a computer-readable medium. It is provided here that one of the computer programs according to the invention is stored on the computer-readable medium. The computer-readable medium according to the invention thus brings with it the same advantages as have been described in detail with reference to a computer program according to the invention. The computer-readable medium can, for. B. as a hard disk and / or non-volatile data storage and / or flash memory and / or firmware and / or downloadable program or the like.

Fig. 1

eine schematische Darstellung zur Visualisierung eines erfindungsgemäßen Verfahrens,

Fig. 2

eine weitere schematische Darstellung zur Visualisierung eines erfindungsgemäßen Verfahrens,

Fig. 3

eine weitere schematische Darstellung zur Visualisierung eines erfindungsgemäßen Verfahrens,

Fig. 4

eine schematische Darstellung eines erfindungsgemäßen Systems.

Further advantages, features and details of the invention emerge from the following description, in which exemplary embodiments of the invention are described in detail with reference to the drawings. The features mentioned in the claims and in the description can each be essential to the invention individually or in any combination. Show it:

Fig. 1

1 shows a schematic illustration for visualizing a method according to the invention,

Fig. 2

another schematic representation for the visualization of a method according to the invention,

Fig. 3

another schematic representation for the visualization of a method according to the invention,

Fig. 4

is a schematic representation of a system according to the invention.

In the following figures, the same reference numerals are used for the same technical features of different exemplary embodiments.

Figure 1 visualizes an inventive method for providing an evaluation means 60 for at least one optical application system 5 of a microscope-based application technology. The application technology is e.g. B. a microscopy technique such as transmitted light microscopy, incident light microscopy, flow cytometry or the like.

In order to carry out the provision of the evaluation means 60, this can first be trained by an optical training system 4. For this purpose, according to a first method step, an input acquisition 101 of at least one sample 2 is carried out in accordance with the application technology in order to obtain at least one input record 110 of the sample 2 based on the input acquisition 101. The microscopy technology used in the training system 4 thus essentially corresponds to the application technology of the application system 5 or is at least similar, ie. H. the sample acquisition for both systems 4, 5 has common characteristics or properties relevant for the sample evaluation. Furthermore, multiple samples 2 can always be acquired or multiple objects of a single sample 2 can be acquired.

Subsequently, according to a second method step, target acquisition 102 of sample 2 is carried out according to a training technology in order to obtain at least one target record 112 of sample 2 based on target acquisition 102. The training technology can differ from the application technology at least in that additional information 115 about the sample 2 is provided.

The input record 110 and target record 112 can also be understood as training data (or also training data). The input record 110 and target record 112 can also each have at least one piece of image information about the sample 2, possibly also about a plurality of samples 2 and / or different and possibly overlapping regions of the samples 2.

A teaching 130 of the evaluation means 60 is provided as a third method step, which is carried out at least on the basis of the input recording 110 and the target recording 112 in order to obtain learning information 200 of the evaluation means 60. The evaluation means 60 taught in this way can be adapted to determine 200 output data from input data on the basis of the learned learning information. The input data are, for example, designed in accordance with (at least approximately) the input record 110 and / or the output data are designed in accordance with (at least approximately) the additional information 115 and, in contrast to this, are recorded in the application system 5. The application system 5 can also be used spatially separated and / or independently from the training system 4. Thus, distributing 140 may be necessary as an intermediate step, e.g. B. via an in Figure 4 The cloud system 8 shown. The distribution 140 preferably includes data exchange and / or non-volatile storage of the learning information 200 in a database of the cloud system 8.

It can be advantageous according to the invention that different sample positions, possibly also a plurality of samples 2, are recorded automatically in the input detection 101 and / or in the target detection 102. This makes it possible to provide a particularly fast and automated acquisition of the training data for the teaching, so that the teaching can be carried out particularly reliably and without complex manual intervention using a large amount of data.

A further advantage can result from the fact that the learning information 200 for a continuous sample relative displacement of at least one sample 2 is learned to determine the additional information 115 by detecting the different sample positions. This makes it possible that when the learned evaluation means 60 is later used, a continuous sample relative displacement can be used in the case of an application system-side acquisition 101 'in order to determine an application system-side acquisition information 110' on the basis of the application system-side acquisition 101 '. The application system-side acquisition 101 'corresponds methodologically completely or approximately to the input acquisition 101 at least to the extent that an application technology is also used here. Possibly. can both in the input detection 101 and in the application system side detection 101 '- or only in the application system side detection 101' continuous sample relative shift can be used. In any case, according to this advantageous further development of the invention, the evaluation means 60 is adapted to be able to use the acquisition system information 110 ′ as input data in order to at least approximately determine the additional information 115 on the basis of this input data and on the basis of the learning information 200 by means of an evaluation 120 as output data. The learning information 200 can be learned to take into account the special features of the continuous sample relative displacement. In this way, particularly fast and effective sample recording can be carried out on the application system side.

It may be possible that the special features of the continuous sample relative shift are taken into account when teaching 130 (and thus, for example, in addition to the evaluation 120 to provide the additional information 115, the evaluation 120 of these special features is also learned). For this purpose, the input detection 101 can at least differ from the target detection 102 in that the input recording 110 is influenced in the input detection 101 by a continuous or, in particular, relative movement of the sample 2. As a result, the learning information 200 can be specific (learned) for at least a reduction in this influence and additionally for the determination of the additional information 115. This can affect, for example, a reduction in motion blur. In other words, a completely automated step-by-step or continuous (relative) movement of the sample 2 can be carried out for the input detection 101 and / or for the target detection 102, for which the learning information 200 is specifically learned.

A particularly rapid sample acquisition can result if, in the case of the input acquisition 101 and / or the target acquisition 102, a fully automated step-by-step or continuous, in particular relative movement of the sample 2 is carried out in a motorized manner in order for a plurality of objects of the sample 2 and / or for a A large number of samples 2 each determine at least one input and target record 110, 112 as training data for teaching 130. A ground truth for teaching 130 can then be determined automatically on the basis of the target recording 112, in particular by reference processing, preferably by automatic segmentation. Furthermore, the target record 112 and / or the input record 110 can each have recordings of the sample 2 at the different sample positions. During the motorized, automated, continuous or stepwise (relative) displacement of the sample 2, position data about the sample positions can be determined in the input detection 101 and / or in the target detection 102 in order to assign the recordings to the sample positions on the basis of the position data. This can also include, for example, "stitching" the sample images recorded in the process.

As in Figure 2 and 3 Shown in more detail, the evaluation means 60 can have one or more artificial neural networks NN1, NN2, NN3, NN4 according to a training hierarchy, preferably at least a first NN1 of the networks having been trained to determine the additional information 115 and at least a second NN2, NN3 of the networks is trained to compensate for the continuous sample relative displacement. In this way, when teaching 130, several neural networks of the evaluation means 60 can be taught hierarchically for respective processing of the input record 110. In particular, the evaluation means 60 has been trained by the learning information 200 to carry out at least two sample analyzes, to which one of the specific neural networks of the evaluation means 60 is assigned. Each of the neural networks can be defined on the basis of a respective learning information 200.

Figure 3 shows schematically another exemplary training approach of the invention. In this case, the training (evaluation) of the evaluation means 60 is carried out in that cells or cell compartments of the sample 2 are marked with a fluorescence stain specific for a sample analysis. This coloring can be used as the basis for target acquisition 102 of this sample 2 according to the training technology in order to determine the target recording 112 therefrom. It is also conceivable that further processing, such as automated segmentation (for example by thresholding the fluorescence signals), is carried out on the target recording 112 in order to use the resulting segmentation information 112 ′ (such as segmented areas of sample 2) as ground truth , So z. B. only the masks of certain cell areas or cells are used as ground truth, in order to increase the specificity of the evaluation means 60 with respect to the desired sample analysis. Subsequently, a first part of the evaluation means NN1, in particular a first neural network NN1, can be taught. In particular, NN1 is automatically trained accordingly in a marker-free imaging method, such as transmission imaging, as application technology to find correlating areas in the application system-side acquisition information 110 ′. For this purpose, a first input detection 101a is carried out with the application technology in order to obtain a first input recording 110a.

In a further step, additional external neural networks of other types can optionally be used in order to increase the specificity of the cell analysis with regard to other sample properties.

Advantageously, with a first input acquisition 101a and / or a second input acquisition 101b and / or a third input acquisition 101c and / or with a target acquisition 102, an automated movement of the sample 2 relative to the training system 4 can take place. In the second and third input acquisition 101b, 101c, this movement can also be carried out continuously. B. gradually. Other movement patterns are also conceivable (such as random movements or a movement of the sample on a disk in order to avoid acceleration peaks due to a spiral continuous movement). is a first input record 110a and a third input record 110c is a record according to a first technique of an application technology, e.g. B. transmission images. Preferably, a second input record 110b is implemented according to a second technique of an application or training technology, e.g. B. fluorescence microscopy. The target recording 112 is optionally also a (possibly different) fluorescence recording or possibly has additional information. The target recording 112 or a further processing 112 ′ based thereon can be used as ground truth for training a first part of the evaluation means, in particular the neural network NN1. Furthermore, the target recording 112 can be used as ground truth for training a second part of the evaluation means (in particular a neural network) NN2, in particular using the second input recording 110b. In addition, the first input record 110a can serve as ground truth for training a third part of the evaluation means (in particular a neural network) NN3 using the third input record 110c. NN2 and NN3 are thus trained to take continuous movement into account.

A preferred further step in the training is shown below in order to allow speeds of sample acquisition above the critical frame rate in the manually moved or motorized continuous mode of the sample relative displacement. For this purpose, the motion blur can be calculated out by means of the evaluation by the evaluation means 60. For example. For this purpose, the first input acquisition 101a is first carried out classically step by step and, if necessary, the resulting images are stitched in order to thereby generate an input record 110a. This can be used to train the evaluation means 60 in such a way that the continuously recorded data of the second input recording 110b are freed from the motion blur.

In Figure 3 It is shown that on the basis of the evaluation by this evaluation means 60 to reduce the motion blur, a further evaluation means 60 can possibly be trained in such a way that it determines the additional information 115 directly from acquisition information 110 ′ on the application system side.

This application system-side acquisition information 110 ′ can be carried out by an application system-side acquisition 101 ′ in the continuous mode. The application system-side acquisition information 110 ′ can also have first acquisition information 110'a and a second acquisition information 110'b, which were acquired in accordance with techniques of application technology. A first technique is, for example, fluorescence microscopy (possibly different from a technically more complex fluorescence microscopy of training technology and / or without segmentation information or other information about the sample) and a second technique is transmission microscopy. According to the trained parts of the evaluation means NN1, NN2, NN3, the additional information 115 can then be determined at least approximately, in particular with a simultaneous reduction in the motion blur. Alternatively, a further fourth part of the evaluation means (in particular the neural network) NN4 can be trained in such a way that it directly determines the desired additional information 115 at least approximately.

Possible combinations of application technology techniques and / or possible combinations of application and training technology are e.g. B. Transmission and Fluorescence, multicolored z. B. RGB images and confocal fluorescence images, stacks of multi-colored images and three-dimensional data from multiphoton images, the latter by z. B. also train three-dimensional data and obtain acquisition system information 110 ′ from it. Furthermore, when using a continuous mode, continuous (in particular relative) movements in the z direction of the sample 2 can also be trained in order to manually pass through the z direction by z. B. to train the detection device 40 and / or an imaging optics 20 of the application system 5. Likewise, randomized continuous (relative) movements in the z direction and mutually orthogonal xy directions (lateral to the sample) could be trained in order to later use in manual mode (i.e. non-motorized manual effect of a (relative) movement of the sample 2 relative to Application system 5 or vice versa) to obtain a higher specificity. From this, e.g. For example, an additional indicator for the specificity in manual mode can be developed automatically in order to generate feedback for a user for the manual (relative) movement.

In Figure 4 an exemplary system for providing an evaluation means 60 for at least one optical application system 5 of a microscope-based application technology is shown schematically. The system is preferably designed as an optical detection system, in particular training system 4. The system can have at least one detection device 40 for performing an input detection 101 of at least one sample 2 according to the application technology in order to obtain at least one input record 110 of the sample 2 based on the input detection 101. In addition, the or a further detection device 40 can be designed to carry out target acquisition 102 of the sample 2 according to a training technology. This is used to obtain at least one target record 112 of sample 2 based on target acquisition 102. The training technology can at least differ from the application technology in that the training technology is designed to provide additional information 115 about the sample 2, such as fluorescence information or segmentation or classification of the sample 2.

The at least one input record 110 and / or the at least one target record 112 can also be determined (sequentially or at least partially simultaneously) via a plurality of channels 35 of the system. These use z. B. a beam splitter, not shown, and / or a dichroic filter or the like to transmit the light changed by a sample 2 to different sensors of the detection device 40 (or also to a further detection device). The forwarding takes place, for example, as a function of at least one property of the light, such as a light color and / or polarization and / or the like. In this way, e.g. B. a transmission channel for the input recording 110 and a fluorescence channel for the target recording 112 can be used. For lighting z. B. one or more light sources 30 are used as lighting means 30, for example to generate laser light or broadband light, and / or an imaging optics 20 corresponding to the generated light z. B. affect the light color or polarization for the forwarding.

Furthermore, an evaluation device 50 for teaching 130 the evaluation means 60 can be provided, at least on the basis of the input record 110 and the target record 112, in order to obtain learning information 200 of the evaluation means 60.

In addition, a sample movement device 45 of the system for (relative or direct) displacement of the sample 2 can be used to automatically detect different sample positions during the input detection 101 and / or during the target detection 102, so that in particular the learning information 200 for a continuous sample relative displacement for determination the additional information 115 is learned. The evaluation means 60 is designed, for example, as a computer program which has commands which, when the computer program is executed by an evaluation device 50 such as a computer 50, cause the additional information about the sample 2 on the basis of the input record 110 of the sample 2 and on the basis of the learning information 200 to investigate. The input record 110 can be specific for the input acquisition 101 of the sample 2 according to the application technology and the learning information 200 can be determined according to a method according to the invention. The evaluation device 50 can have a processor 51 for executing the evaluation means 60, which processor is connected to a data memory 52, such as a computer-readable medium (e.g. a hard disk and / or a non-volatile memory and / or a flash memory and / or the like).

To provide the learned evaluation means 60, the learning information 200 can then be transmitted to a cloud system 8 or a local data storage device or the like. The learning information 200 can then be distributed to one or more application systems 5 in order to carry out an application system-side detection 101 ′ there with the learned evaluation means 60 (i.e. using the learning information 200).

The above explanation of the embodiments describes the present invention exclusively in the context of examples. Of course, individual features of the embodiments, if technically meaningful, can be freely combined with one another without departing from the scope of the present invention.

LIST OF REFERENCE NUMBERS

2

sample

4

Training system, training microscope

5

application system

8th

Cloud system, central computer system for cloud computing

20

Imaging optics, acquisition optics

30

Light source, illuminant

35

channels

40

Detection device, camera

45

Sample moving device, motorized sample table

50

Evaluation device, computer

51

processor

52

data storage

60

evaluating means

101

Input entry, training system side entry

101

first input entry

101b

second input entry

101c

third input entry

102

Target acquisition, reference acquisition

110

input recording

112

Target Record, Ground Truth

115

Additional information, result information

120

evaluation

130

teaching

140

To distribute

200

Teaching information, transfer information, trained model

101 '

application system side acquisition

110 '

application system-side acquisition information

110'a

first acquisition information

110'b

second acquisition information

112 '

processed target record, segmentation result

NN1

first network

NN2

second network

NN3

third network

NN4

fourth network

Claims (15)

Method for providing an evaluation means (60) for at least one optical application system (5) of a microscope-based application technology,
the following steps being carried out, in particular in each case using an optical training system (4): Performing an input acquisition (101) of at least one sample (2) according to the application technology in order to obtain at least one input record (110) of the sample (2) based on the input acquisition (101), - Performing a target acquisition (102) of the sample (2) according to a training technology in order to obtain at least one target record (112) of the sample (2) based on the target acquisition (102), the training technology differing from the application technology at least in that a Additional information (115) about the sample (2) is provided, - Teaching (130) the evaluation means (60) at least on the basis of the input record (110) and the target record (112) in order to obtain teaching information (200) of the evaluation means (60), whereby different sample positions are automatically recorded during the input detection (101) and / or during the target detection (102), so that in particular the learning information (200) for a continuous sample relative displacement is learned to determine the additional information (115). Method according to claim 1,
characterized,
that the input detection (101) differs from the target detection (102) at least in that in the input detection (101) the input recording (110) is influenced by a relative movement of the sample (2), so that the learning information (200) for at least one Reduction of this influence and is also specific for the determination of the additional information (115). The method of claim 1 or 2,
characterized,
that during the input acquisition (101) and / or the target acquisition (102) a completely automated step-by-step or continuous sample relative displacement, in particular relative movement of the sample (2) is carried out, for which the learning information (200) is specifically taught. Method according to one of the preceding claims,
characterized,
that in the input acquisition (101) and in the target acquisition (102) a fully automated step-by-step or continuous relative movement of the sample (2) is carried out in a motorized manner in order for a large number of objects of the sample (2) and / or for a large number of samples (2) in each case to determine at least one input and target record (110, 112) as training data for teaching (130), a ground truth for teaching (130) being determined automatically on the basis of the target recording (112), in particular by reference processing, preferably by automatic segmentation. Method according to one of the preceding claims,
characterized,
that the target record (112) and / or the input record (110) each have recordings of the sample (2) at the different sample positions, the input capture (101) and / or the target capture (102) during a motorized automated continuous or stepwise relative displacement of the sample (2) position data on the sample positions are determined in order to assign the records to the sample positions on the basis of the position data. Method according to one of the preceding claims,
characterized,
that the input acquisition (101) and the target acquisition (102) are each carried out as an acquisition (101, 102) on the training system side by an optical training system (4), the learning information (200) for determining the additional information (115) on the basis of an application system-side acquisition information (110 ' ) is specific, which is determined by a subsequent application system-side input acquisition (101 ') by an optical application system (5), in which a manual continuous sample relative displacement takes place, the application system (5) being designed to provide the application technology, and the training system ( 4) in addition to providing the training technology. Method according to one of the preceding claims,
characterized,
that the evaluation means (60) is designed as a computer program based on machine learning and / or artificial intelligence and / or deep learning, and preferably has one or more artificial neural networks (NN1, NN2, NN3, NN4) according to a training hierarchy, preferably one first (NN1) of the networks for the determination of the additional information (115) has been taught in and at least a second (NN2, NN3) of the networks for the compensation of the continuous sample relative displacement has been taught in. Method according to one of the preceding claims,
characterized,
that during the teaching (130) a plurality of neural networks of the evaluation means (60) for respective processing of the input record (110) are learned hierarchically, and in particular the evaluation means (60) is taught by the learning information (200) to carry out at least two sample analyzes, each of which one of the specific neural networks of the evaluation means (60) is assigned. Method according to one of the preceding claims,
characterized,
that at least one detection device (40) and at least one lighting means (30) of a training system (4) are used for input and target detection (101, 102), wherein - In the input detection (101), the sample (2) in a first recording mode and / or with a first illumination intensity relative to the training system, in particular a detection optics (20), continuously or step by step over more than an entire detection field of the detection device (40) is relatively shifted in order to obtain the input record (110) of a region of the sample (2), the target area (102) is at least partially the same area of the sample (2) in a second recording mode and / or with a second illumination intensity in order to obtain the target record (112), - during teaching (130), a ground truth (112) is determined from the target record (112) and target data are determined from the input record (110), which are used for teaching (130) at least one first neural network in order to obtain the teach-in information ( 200) to determine - optional: teaching (130) at least one further neural network by means of the first neural network in order to determine the learning information (200) which is specific for a neural network system of all trained neural networks, - Using the evaluation means (60) with the learning information (200) in an application system (5) for application-side detection (101 ') in the first recording mode and / or with the first illumination intensity in order to obtain application-side detection information (110'), wherein for this purpose, at least one sample (2) is continuously shifted at least relatively over more than an entire detection field of a further detection device (40). Method according to one of the preceding claims,
characterized,
that the method steps of the method for input and target acquisition (101, 102), and in particular also for acquisition (101 ') on the application system side, of different samples (2) and / or objects of the sample (2) are carried out completely automatically, so that any manual intervention is avoided. Method according to one of the preceding claims,
characterized,
that different depths of the sample (2) are recorded during the input and / or target acquisition (101, 102) and / or during the application system acquisition (101 '), so that the input record (110) and / or the target record (112) and / or acquisition information (110 ') of the application system-side acquisition (101') has a stack with recordings of the sample (2) for axial sample positions, the different depths of the sample (2) preferably being acquired in that, in particular by the application system (5), sample areas of the different depths are mapped to different areas of a sensor of a detection device (40), the sensor preferably being arranged inclined with respect to an optical axis in the application system (5),
and or
that the input acquisition (101) and / or target acquisition (102) and / or further target acquisitions (102) for teaching (130) through different channels (35) of the training system (4) are carried out simultaneously or at different times for different sample positions. System for providing an evaluation means (60) for at least one optical application system (5) of a microscope-based application technology, in particular according to a method according to one of claims 1 to 11, comprising: at least one detection device (40) for performing an input detection (101) of at least one sample (2) according to the application technology in order to obtain at least one input record (110) of the sample (2) based on the input detection (101), - The or at least one further detection device (40) for performing target acquisition (102) of the sample (2) according to a training technology in order to obtain at least one target record (112) of the sample (2) based on the target acquisition (102), the training technology differs from the application technology at least in that the training technology is designed to provide additional information (115) about the sample (2), - an evaluation device (50) for teaching (130) the evaluation means (60) at least on the basis of the input record (110) and the target record (112) in order to obtain teaching information (200) of the evaluation means (60), - A sample movement device (45) for the relative displacement of the sample (2) in order to automatically detect different sample positions during the input detection (101) and / or the target detection (102), so that in particular the learning information (200) for a continuous sample relative displacement for the determination the additional information (115) is learned. Computer program, comprising commands which cause the computer (50) to execute the computer program, on the basis of at least one input record (110) of the sample (2) and on the basis of teaching information (200) additional information (115) about a sample (2) to be determined, the input recording (110) being specific for an input acquisition (101) of the sample (2) according to an application technology and the learning information (200) being determined according to one of claims 1 to 11. A computer program comprising commands which, when the computer program is executed by a computer (50), cause the computer (50) to carry out the steps of a method according to one of claims 1 to 11. Computer-readable medium on which the computer program according to claim 13 or 14 is stored.

Images