DE102022201258A1 - Process for separating objects - Google Patents

Abstract

The invention relates to a computer-implemented method for identifying objects, in particular cover slips, on an image of a sample carrier, in particular a specimen slide, of a microscope system, having the following work steps: • Reading in the image, in particular as image data; • Segmenting at least one region on the sample carrier, in which the objects are arranged on the sample carrier, on the basis of the image, in particular the image data, by means of a first image recognition model, the first image recognition model being based on a first machine learning model, carrier data which characterize at least one property of the at least one region being generated; • reading in auxiliary data, the auxiliary data characterizing at least one property of the objects;• separating objects in the at least one region by means of an image processing method on the basis of the carrier data and the auxiliary data; and• outputting information related to the separated objects.

Description

The invention relates to a method for identifying objects, in particular cover slips, on an image of a sample carrier, in particular a specimen slide, of a microscope system, the image being read in and at least one region on the specimen carrier, in which the objects are arranged on the specimen carrier, on which Based on the image is segmented by means of a first image recognition model and carrier data, which characterize at least one property of the at least one region, are generated.

In classic microscopes, a user looks at a sample carrier through an eyepiece while sitting or standing. He can interact directly with the sample in such a way that on the one hand he can obtain a cursory overview of the field of view of the objective, in particular of the sample carrier, the position of cover glasses and samples, and on the other hand the sample carrier with the sample either directly or can be shifted laterally using an adjustable sample stage in order to bring other areas of the sample carrier into the field of view of the objective. The user of the microscope can remain in his place and only has to move his head minimally, so that the classic microscopes are highly ergonomic in this respect.

Today's microscope systems, on the other hand, allow so-called flash stacks to be recorded along an observation direction and a three-dimensional image of a sample to be reconstructed from them. For this purpose, images are generated using detectors. Cameras that are equipped with appropriate area sensors, in particular CD chips, or so-called photomultipliers are preferably used as detectors.

With these new types of microscope systems, the workplace has therefore shifted away from the microscope stand and thus away from the sample to the computer or the screen of such a system. However, the work station in front of the microscope stand is often still used and also required to prepare or set up the sample carrier or the sample for an analysis.

• • den Probenträger in das Sichtfeld des Objektivs zu bringen;
• • eine Region auf dem Probenträger auszuwählen, in welchem eine Probe angeordnet ist;
• • diese anzufahren; und
• • schließlich das Mikroskop auf den Probenträger oder die Probe zu fokussieren.

This includes the work steps:

• • bring the sample carrier into the field of view of the objective;
• • select a region on the sample carrier in which a sample is arranged;
• • drive to them; and
• • finally to focus the microscope on the sample carrier or the sample.

The workload when using modern, complex microscope systems is therefore often associated with two work areas in which different work steps take place and which are spatially separated from one another. On the one hand the microscope stand with eyepiece for direct observation, and on the other hand the screen of a computer.

The document DE 10 2017 111 718 A1 relates to a method for generating and analyzing an overview contrast image of a sample carrier and/or samples arranged on a sample carrier, in which a sample carrier arranged at least partially in the focus of a detection optics is illuminated in transmitted light with a two-dimensional, array-shaped illumination pattern, with at least two overview raw images with different illuminations of the Sample carrier are detected and, depending on information to be extracted from the overview contrast image, a calculation algorithm is selected, by means of which the at least two overview raw images are calculated to form the overview contrast image and, depending on information to be extracted from the overview contrast image, an image evaluation algorithm is selected, by means of which information from the overview contrast image be extracted.

It is an object of the invention to provide a method and system by means of which objects on an image of a sample carrier can be identified in an automated manner.

This object is solved by the teaching of the independent claims. Advantageous developments are claimed in the dependent claims.

• • Einlesen des Bilds, insbesondere als Bilddaten;
• • Segmentieren wenigstens einer Region auf dem Probenträger, in welcher die Objekte auf dem Probenträger angeordnet sind, auf der Grundlage des Bilds, insbesondere der Bilddaten, mittels eines ersten Bilderkennungsmodells, wobei das erste Bilderkennungsmodell auf einem ersten Maschinenlernmodell, insbesondere einem ersten künstlichen neuronalen Netz, beruht, wobei Trägerdaten, welche wenigstens eine Eigenschaft der wenigstens einen Region charakterisieren, erzeugt werden;
• • Einlesen von Hilfsdaten, wobei die Hilfsdaten wenigstens eine Eigenschaft der Objekte charakterisieren;
• • Separieren von Objekten in der wenigstens einen Region mittels eines Bildverarbeitungsverfahrens auf der Grundlage der Trägerdaten und der Hilfsdaten; und
• • Ausgeben von Informationen in Bezug auf die separierten Objekte.

A first aspect of the invention relates to a computer-implemented method for identifying objects, in particular cover slips, on an image of a sample carrier, in particular a specimen slide, of a microscope system, having the following work steps:

• • Reading in the image, in particular as image data;
• • segmenting at least one region on the sample carrier, in which the objects are arranged on the sample carrier, based on the image, in particular the image data, using a first image recognition model, the first image recognition model being based on a first machine learning model, in particular a first artificial neural network, is based, wherein carrier data characterizing at least one property of the at least one region is generated;
• • Reading in auxiliary data, the auxiliary data characterizing at least one property of the objects;
• • separating objects in the at least one region by means of an image processing method based on the carrier data and the auxiliary data; and
• • Outputting information related to the separated objects.

A second aspect of the invention relates to a method for training an artificial neural network for identifying objects, the artificial neural network having two paths, a first path being an image recognition model for segmenting objects, in particular cover glasses, on an overview image of a sample carrier of a microscope system, forms and a second path forms an image recognition model for detecting properties of the objects.

The second aspect also relates to a method for training two artificial neural networks for identifying objects, the first artificial neural network forming an image recognition model for segmenting objects, in particular cover glasses, on an overview image of a sample carrier of a microscope system, and the second artificial neural network forms an image recognition model for detecting properties of the objects.

• • Erzeugen eines Datenstroms, insbesondere eines Bildstroms einer Vielzahl von Übersichtsbildern, welche Probenträger eines Mikroskopsystems abbilden, auf welchem Objekte angeordnet sind;
• • Einlesen von Ground Truth-Trägerdaten, welche wenigstens eine Eigenschaft wenigstens einen Region charakterisieren, in welcher die Objekte auf dem Probenträger angeordnet sind; und
• • Bereitstellen des Datenstroms mit den dazugehörigen jeweiligen Ground Truth-Trägerdaten an das künstliche neuronale Netz oder die künstlichen neuronalen Netze.

The procedure has the following work steps:

• • Generating a data stream, in particular an image stream of a multiplicity of overview images which depict sample carriers of a microscope system on which objects are arranged;
• • Reading in ground truth carrier data which characterize at least one property of at least one region in which the objects are arranged on the sample carrier; and
• • Providing the data stream with the associated respective ground truth carrier data to the artificial neural network or artificial neural networks.

In this way, an artificial neural network or two artificial neural networks can be generated, which can be used in a method for identifying objects.

The methods can preferably be carried out by a computer, in particular a system with various means, preferably automated.

• • eine erste Schnittstelle zum Einlesen des Bilds, insbesondere als Bilddaten, und von Hilfsdaten, wobei die Hilfsdaten wenigstens eine Eigenschaft der Objekte charakterisieren;
• • Mittel zum Segmentieren wenigstens einer Region auf dem Probenträger, in welcher die Objekte auf dem Probenträger angeordnet sind, auf der Grundlage des Bilds, insbesondere der Bilddaten, mittels eines ersten Bilderkennungsmodells, wobei das erste Bilderkennungsmodell auf einem ersten künstlichen neuronalen Netz beruht, wobei Trägerdaten, welche wenigstens eine Eigenschaft der wenigstens einen Region charakterisieren, erzeugt werden;

• • Mittel zum Separieren von Objekten in der wenigstens einen Region mittels eines Bildverarbeitungsverfahrens auf der Grundlage der Trägerdaten und der Hilfsdaten; und
• • eine zweite Schnittstelle zum Ausgeben von Informationen in Bezug auf die separierten Objekte.

A third aspect of the invention relates to a system for identifying objects, in particular cover glasses, on an image of a sample carrier of a microscope system, having:

• • a first interface for reading in the image, in particular as image data, and auxiliary data, the auxiliary data characterizing at least one property of the objects;
• • Means for segmenting at least one region on the sample carrier, in which the objects are arranged on the sample carrier, on the basis of the image, in particular the image data, by means of a first image recognition model, the first image recognition model being based on a first artificial neural network, with carrier data , which characterize at least one property of the at least one region, are generated;

• • means for separating objects in the at least one region by means of an image processing method based on the carrier data and the auxiliary data; and
• • a second interface for outputting information related to the separated objects.

A fourth aspect of the invention relates to a microscope system with such a system for identifying objects. Further aspects relate to a computer program and a computer-readable medium for identifying objects.

According to the invention, the carrier data preferably correspond to a segmentation mask.

• Klasse 1: Objekt
• Klasse 2: Hintergrund oder Probenträger
• Klasse 3: Rand der Objekte.

The segmentation mask preferably has the following classes:

• Class 1: Object
• Class 2: background or sample carrier
• Class 3: Edge of objects.

A segmentation mask within the meaning of the invention preferably indicates which class a specific pixel of an image belongs to. In this case, the segmentation can be considered as a classification task, because every single pixel of a predefined class has to be identified.

An overview image within the meaning of the invention is preferably an image of a sample carrier, in which preferably the entire sample carrier, particularly preferably more than half of the sample carrier gers and most preferably at least one third of the sample carrier are visible.

A segmentation within the meaning of the invention is in particular a semantic image segmentation. The segmentation preferably has a simultaneous clustering of an image and classification of the image segments created by the clustering into a fixed number of classes. Artificial neural networks are preferably used for this purpose. The segmentation problem is preferably formulated as a classification problem for each individual pixel of the image. Preferably, after a training phase of the artificial neural network, novel input data which the artificial neural network has not processed during the training phase can be segmented semantically.

A separation within the meaning of the invention is preferably an identification of individual objects in a region in which several objects are located.

Reading in within the meaning of the invention preferably means capturing data that are provided. More preferably, reading in means providing and acquiring the data.

An output within the meaning of the invention is in particular a provision at a data interface or a user interface.

A means within the meaning of the invention can be configured as hardware and/or software and in particular a processing unit (CPU) and/or a or have several program(s) or program module(s). The CPU can be designed to process commands that are implemented as a program stored in a memory system, to detect input signals from a data bus and/or to emit output signals to a data bus. A storage system can have one or more, in particular different, storage media, in particular optical, magnetic, solid-state and/or other non-volatile media. The program may be arranged to embody or be capable of performing the methods described herein such that the CPU can perform the steps of such methods.

The invention is based on the approach of using an image processing method for analyzing the overview image in addition to segmenting an overview image of a sample carrier by means of an image recognition model, which method takes auxiliary information into account. In this way, a region can first be identified on a sample carrier which is different from the sample carrier which is arranged in the background and in which there is therefore a high probability that objects are arranged.

In a second step, individual objects, if present, can be separated in this region using auxiliary data that characterize at least one property of the objects.

By separating objects, their position, their number, their shape and/or their orientation, etc. can be determined, for example.

This information can in turn be used for navigating on the sample carrier or on a sample. The microscope system can preferably move to the position of a sample automatically on the basis of the information from the method according to the invention.

Furthermore, however, the information can also be used within the framework of an automated examination, for example a high-throughput examination of samples.

In an advantageous embodiment of the method, the image processing method includes determining the at least one region, with individual regions being merged into one region or splitting up into a plurality of regions on the basis of their shape and orientation.

In this way, objects that each form a region can be identified.

In a further advantageous embodiment of the method, the image processing method includes determining parameters of a model of the objects on the basis of the carrier data and the auxiliary data by means of a fitting calculation.

In a further advantageous embodiment of the method, the image processing method includes an optimization of an evaluation function, with carrier data and the auxiliary data entering into the evaluation function, with the evaluation function preferably being designed as a correction calculation. By applying a best fit calculation and/or an evaluation function, objects whose properties, such as number and shape, are known can be separated by fitting the model.

In a further advantageous embodiment of the method, the image processing method includes a region growing on the basis of Carrier data and the auxiliary data, with homogeneous image elements of the region being merged into objects.

In a region growing according to the invention, starting from an identified center or focus of an object as the starting area, this starting area is compared using a criterion, in particular the difference of a gray value to the neighboring cells. If the criterion applies, the neighboring cell is added to the range. This process is preferably repeated recursively by examining the neighboring areas of the newly added areas as well. If no more areas can be added, the object is identified.

The objects can be identified particularly easily using a region-growing method.

In a further advantageous embodiment of the method, the image processing method includes identifying edges around the objects on the basis of the carrier data and the auxiliary data.

In a further advantageous embodiment, reading in the auxiliary data includes determining at least one property of the objects using the first image recognition model or a second image recognition model, which is based on a second artificial neural network, on the basis of the overview image, with the auxiliary data relating to the at least one Property of the objects are generated. In this case, the auxiliary data is also generated using the first artificial neural network or the second artificial neural network on the basis of the overview image.

If the auxiliary data is also generated by the first artificial neural network, an artificial neural network with a plurality of so-called heads is preferably used. This is particularly advantageous when determining the number of objects, in particular the number of cover glasses.

In a further advantageous refinement of the method, the auxiliary data are preferably based on inputs by a user via a user interface. As a result, the user's knowledge can be taken into account.

A user interface within the meaning of the invention is preferably a keyboard, a device for voice input, a touch-sensitive display, etc.

• • Art des Mikroskopsystems;
• • Art, insbesondere Form, des Probenträgers;
• • Benutzerinformationen.

In a further advantageous embodiment of the method, the auxiliary data relate to at least one of the properties from the following list of properties:

• • type of microscope system;
• • Type, in particular shape, of the sample carrier;
• • User Information.

In a further advantageous embodiment of the method, reading in the auxiliary data includes determining at least one property of the objects using a second image recognition method, the auxiliary data being generated in relation to the at least one property of the objects.

A second image recognition method can be used to gain further insights into the properties of the objects.

• • Anzahl der Objekte;
• • Form der Objekte;
• • Größe der Objekte;
• • Rand der Objekte;
• • Ausrichtung der Objekte;
• • Mittelpunkte oder Schwerpunkte der Objekte.

In a further advantageous embodiment of the method, the auxiliary data relate to at least one of the properties of the objects from the following list of properties:

• • number of objects;
• • shape of objects;
• • size of objects;
• • edge of objects;
• • Alignment of objects;
• • Centers or centroids of the objects.

• • Anzahl der Regionen;
• • Form der wenigstens einen Region;
• • Größe der wenigstens einen Region;
• • Rand der wenigstens einen Region;
• • Ausrichtung der wenigstens einen Region.

In a further advantageous embodiment of the method, the carrier data relate to at least one of the properties of the at least one region from the following list of properties:

• • number of regions;
• • shape of the at least one region;
• • size of the at least one region;
• • edge of the at least one region;
• • Alignment of the at least one region.

In a further advantageous embodiment of the method, the first artificial neural network is a multi-layer artificial neural network, in particular a convolutional artificial neural network, and has at least one processing section (backbone) and at least two paths (heads). a first path generates the bearer data and a second path generates the auxiliary data.

The auxiliary data and the carrier data can be generated particularly efficiently by an artificial neural network with a plurality of paths.

• • Labeln der Objekte in dem Übersichtsbild auf der Grundlage der Informationen in Bezug auf die Objekte, wobei ein gelabeltes Übersichtsbild, insbesondere gelabelte Bilddaten, erzeugt wird bzw. werden; und
• • Ausgeben des gelabelten Übersichtsbilds, insbesondere der gelabelten Bilddaten.

In a further advantageous embodiment, the method also has the following work steps:

• • labeling the objects in the overview image based on the information relating to the objects, wherein a labeled overview image, in particular labeled image data, is/are generated; and
• • Output of the labeled overview image, in particular the labeled image data.

In this way, objects can be identified directly using the image data.

In a further advantageous embodiment, the method also has the work step of controlling the microscope system on the basis of the information.

As a result, the microscope system, in particular the microscope itself, can be operated in an automated manner and, for example, a sample can be approached.

The aforementioned features and advantages in relation to the first aspect of the invention apply correspondingly to the second, third and fourth as well as further aspects of the invention and vice versa.

• 1a: ein Beispiel eines Übersichtsbilds eines Probenträgers;
• 1b: ein Ausführungsbeispiel eines Übersichtsbilds mit einer segmentierten Region;
• 1c: ein Ausführungsbeispiel von Hilfsdaten in Bezug auf Objekte auf einem Probenträger;
• 1d: ein Ausführungsbeispiel von separierten Objekten auf einem Probenträger;
• 2: ein Ausführungsbeispiel eines Verfahrens zum Identifizieren von Objekten auf einem Übersichtsbild eines Probenträgers;
• 3: ein Ausführungsbeispiel eines Mikroskopsystems mit einem System zum Identifizieren von Objekten; und
• 4: ein Ausführungsbeispiel eines Verfahrens zum Trainieren eines künstlichen neuronalen Netzes für ein Bilderkennungsmodell zum Trennen von Objekten auf einem Übersichtsbild eines Probenträgers

Further features and advantages result from the description with reference to the figures. They show at least partially schematically:

• 1a : an example of an overview image of a sample carrier;
• 1b : an embodiment of an overview image with a segmented region;
• 1c : an embodiment of auxiliary data related to objects on a sample carrier;
• 1d : an embodiment of separated objects on a sample carrier;
• 2 : an embodiment of a method for identifying objects on an overview image of a sample carrier;
• 3 : an embodiment of a microscope system with a system for identifying objects; and
• 4 1: an embodiment of a method for training an artificial neural network for an image recognition model for separating objects on an overview image of a sample carrier

A method 100 for identifying objects is described in relation to 1 and 2 explained below.

The objects identified in this exemplary embodiment are cover glasses 11, 12, 13, which are arranged on a sample carrier 10, in particular an object carrier.

The method is based on an overview image of a sample carrier 10, which can preferably be recorded by an overview camera 8 (both not shown) attached to a microscope 7, in particular an overview camera 8 attached to the microscope stand.

In a first work step 101, the image data of an overview image are read.

An example of such an overview is shown in 1a shown.

In a second work step 102, at least one region 14 on the sample carrier 10 is segmented on the basis of the overview image. An image recognition model is used for this, which is formed by a trained first artificial neural network.

Preferably, the artificial neural network has multiple layers and is preferably a convolutional artificial neural network.

The image recognition model can be used to generate carrier data which characterize at least one property of the at least one region.

Such properties are preferably a number of regions 14, a shape of the at least one region 14, a size of the at least one region 14, an edge of the at least one region 14 and/or an orientation of the at least one region 14.

The use of an artificial neural network enables the image recognition model to cover the region 14 on the sample carrier 10 with cover glasses 11, 12, 13, as in 1a shown, based on various properties or in the interaction of features of the cover glasses 11, 12, 13 and the sample carrier 10 can recognize. Such features can be, for example, the reflectivity, the color, artefacts, etc.

The segmentation according to the invention is preferably a semantic image segmentation. In this case, individual pixels or areas of the overview image are preferably assigned to the individual classes in a classification problem. The classes are preferably specified for this.

• Klasse 1: Deckglas
• Klasse 2: Hintergrund
• Klasse 3: Deckglasrand.

In the present case, such classes can be, for example:

• Class 1: coverslip
• Class 2: Background
• Class 3: coverslip edge.

A region 14 segmented on a sample carrier 10 is in 1b shown.

The carrier data include the position of region 14, the alignment of region 14, the edge 15a, 15b, 15c of region 14, and the area of region 14.

In a third work step 103, auxiliary data is read in, with the auxiliary data characterizing at least one property of the objects.

Depending on the type of auxiliary data, these can be read in in different ways.

In this way, the auxiliary data can preferably be determined from the overview image. The image recognition model, which is also used to determine the carrier data, is preferably used for this. In this case, the artificial neural network on which the image recognition model is based also preferably has a processing section (backbone) and at least two paths (heads). In this case, a first path generates the carrier data and a second path generates the auxiliary data.

As an alternative to this, the auxiliary data are generated using a further artificial neural network, which forms a second image recognition model. This second artificial neural network is also trained and can determine the properties of the cover glasses 11, 12, 13 in this way.

Alternatively or additionally, properties of the objects can be determined using a second image processing method.

Auxiliary data generated in this way can be the following properties of the cover glasses 11, 12, 13: number of cover glasses 11, 12, 13, shape of the cover glasses 11, 12, 13, size of the cover glasses 11, 12, 13, edge 15a, 15b, 15c of the cover glasses 11, 12, 13, alignment of the cover glasses 11, 12, 13, centers M1, M2, M3 or focal points of the cover glasses 11, 12, 13.

Alternatively or additionally, auxiliary data are provided by reading in a user's input via a user interface 6 (not shown) or by reading in an attribute, for example a system identifier, a specific user, identifiable via his login, or a classification of the sample carrier.

Corresponding properties or context information can be, for example, the type of microscope system 1, the type, in particular the shape of the sample carrier 10 or also user information. Alternatively, this information can also be used to select a suitable model, in particular the first model, which carries out the segmentation, or to flow in as further input data when training the image recognition model.

In 1c For example, the focal points M1, M2, M3 of the cover glasses 11, 12, 13 are shown. These focal points M1, M2, M3 can be determined, for example, by means of the image recognition model, by means of which the carrier data were generated, or by means of the further image recognition model, preferably by means of semantic segmentation.

Another example of the properties of the cover glasses 11, 12, 13 can be their number, three in the illustrated case, and their shape, rectangular in the illustrated case. Such information can also be read in by means of semantic segmentation or via input from a user.

In a fourth work step 104, the cover glasses 11, 12, 13 in the at least one region 14 are processed using an image processing method based on the carrier data or the properties of the at least one region 14 and the auxiliary data or the properties of the cover glasses 11, 12, 13 separated.

The first image processing method preferably includes a merging of individual regions 14 based on their shape and orientation into one region or a splitting of an individual region 14 into a plurality of regions 11, 12, 13, as in 1d shown. In 1d the split regions 11, 12, 13 each correspond to a cover glass. The shape and orientation of the cover glasses 11, 12, 13 to be separated are preferably used as auxiliary data in the image processing method.

Alternatively or additionally, the first image processing method includes determining parameters of a model of the cover glasses 11, 12, 13 on the basis of the carrier data, in particular the position of the region 14 with the cover glasses 11, 12, 13 and the auxiliary data, in particular the number and the shape the coverslips 11, 12, 13 by means of an off equal calculation. In the compensation calculation, a function is preferably fitted in such a way that it matches the determined carrier data and auxiliary data. In this regard, an optimization of an evaluation function can preferably also be used, with which the fitted function can be optimized.

Alternatively or additionally, what is known as a region-growing method can be used as the first image processing method. Here, on the basis of carrier data, in particular the position of a region 14 and the auxiliary data, in particular the number of central or focal points found, homogeneous image elements 11, 12, 13 of the segmented region 14 of the sample carrier 10 become areas which correspond to the cover glasses, merged.

Alternatively or additionally, the first image processing method includes identifying edges 15a, 15b, 15c around the cover glasses 11, 12, 13. Here, the individual cover glasses 11, 12, 13 are preferably separated on the basis of the carrier data, which form a segmentation mask. In this case, one class of the classification problem to be solved is the edge 15a, 15b, 15c of the cover glasses 11, 12, 13, which must be recognizable in the overview image of the sample carrier 10 for the image processing method.

In a fifth work step 105, information relating to the separated cover glasses 11, 12, 13 is output.

Such information can, for example, be image data relating to a processed overview image, for example as in 1d shown, in which the cover glasses 11, 12, 13 and the sample carrier 10 are visible. The position, number, shape, orientation, etc. of the cover glasses 11, 12, 13 can be additional information.

This information can preferably be used in a work step 106 for controlling the microscope system 1 .

In a seventh work step 107, the information generated can be used to label an overview image, with a labeled overview image, in particular labeled image data, being or being generated.

3 1 shows an exemplary embodiment of a microscope system 1 with a system 2 for identifying objects 11, 12, 13. The microscope system 1 preferably has a microscope 7, on the stand of which an overview camera 8 is attached.

Image data of an overview image taken by means of the overview camera 8 are read into the system 2 via a first interface 3 . Furthermore, auxiliary data, which characterize a property of the objects 11, 12, 13 to be identified, are read into the system 2, preferably via the first interface 3.

The system 2 is preferably a device for data processing.

The system 2 also preferably has means 4 for segmenting at least one region 14 on the sample carrier 10, and means 5 for separating objects 11, 12, 13 in the at least one region 14.

Finally, a second interface 6 is set up to output information relating to the separated objects 11, 12, 13.

4 shows an exemplary embodiment of a method 200 for training an artificial neural network for an image recognition model for separating objects 11, 12, 13 on an overview image of a sample carrier 10 of a microscope system 1.

In a first working step 201 of the method 200, a data stream is generated, in particular an image stream of a large number of overview images which depict the sample carrier 10 in a microscope 7 of a microscope system 1 on which objects 11, 12, 13 are arranged.

In a second work step 202, ground truth carrier data are read in, which indicate at least one property of at least one region 14 in which the objects 11, 12, 13 are arranged on the sample carrier 10.

In a third work step 203, the data stream is provided with the associated respective ground truth carrier data for the artificial neural network for training.

It is pointed out that the exemplary embodiments are only examples that are not intended to restrict the scope of protection of the application and the structure in any way. Rather, the above description gives the person skilled in the art a guideline for the implementation of at least one exemplary embodiment, with various changes, in particular with regard to the function and arrangement of the described components, being able to be made without departing from the scope of protection as it results from the claims and these equivalent combinations of features. In particular, the described methods and systems used to identify other objects such as coverslips on a sample carrier. Non-limiting examples are samples, artifacts, particles, etc.

Reference List

1

microscope system

2

system

3

interface

4

means for segmenting

5

means of reading

6

Second interface

10

sample carrier

11, 12, 13

coverslip

14

region

15a, 15b, 15c

edge of a coverslip

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102017111718 A1 [0007]

Claims (15)

Computer-implemented method (100) for identifying objects (11, 12, 13), in particular cover glasses, on an overview image of a sample carrier (10) of a microscope system (1), having the following work steps: reading (101) the overview image; Segmenting (102) at least one region (14) on the sample carrier (10), in which the objects (11, 12, 13) are arranged on the sample carrier (10), on the basis of the overview image, in particular the image data, by means of a first Image recognition model, the first image recognition model being based on a first machine learning model, in particular a first artificial neural network, carrier data which characterize at least one property of the at least one region (14) being generated; Reading in (103) auxiliary data, the auxiliary data characterizing at least one property of the objects (11, 12, 13); separating (104) objects (11, 12, 13) in the at least one region (14) by means of a first image processing method on the basis of the carrier data and the auxiliary data; and outputting (105) information related to the separated objects (11, 12, 13). Method (100) according to claim 1 , wherein the first image processing method comprises determining the at least one region (14), individual regions (14) being merged into one region as auxiliary data on the basis of their shape and orientation or being split into a plurality of regions (11, 12, 13). Method (100) according to claim 1 or 2 , wherein the first image processing method includes determining parameters of a model of the objects (11, 12, 13) on the basis of the carrier data and the auxiliary data, in particular the number of individual objects present in the overview image or their orientation, by means of a fitting calculation. Method (100) according to any one of Claims 1 until 3 , wherein the first image processing method comprises an optimization of an evaluation function, with the carrier data and the auxiliary data entering into the evaluation function, with the evaluation function preferably being designed as an adjustment calculation. Method (100) according to claim 1 or 2 , wherein the first image processing method comprises region growing based on the carrier data and the auxiliary data, in particular the centers and/or focal points of the objects, with homogeneous image elements (11, 12, 13) of the region (14) being merged into objects. A method (100) according to any one of the preceding claims, wherein the first image recognition model or the second image recognition model comprises identifying edges (15a, 15b, 15c) around the objects (11, 12, 13) based on the overview image. Method (100) according to one of the preceding claims, wherein the reading in of the auxiliary data involves determining at least one property of the objects (11, 12, 13) using the first image recognition model or a second image recognition model which is based on a second machine learning model, in particular a second artificial neural network. or a second path of the first machine learning model, based on the overview image, wherein the auxiliary data are generated in relation to the at least one property of the objects (11, 12, 13). Method (100) according to one of the preceding claims, wherein the reading in of the auxiliary data involves determining at least one property of the objects (11, 12, 13) by means of a second image processing method, the auxiliary data relating to the at least one property of the objects (11, 12, 13) are generated. Use of by means of a method (100) according to one of Claims 1 until 8th generated information for labeling (107) the overview image, wherein a labeled overview image, in particular labeled image data, is or are generated. Use of by means of a method (100) according to one of Claims 1 until 9 generated information for controlling (106) the microscope system (1). Computer-implemented method (200) for training an artificial neural network for identifying objects (11, 12, 13), the artificial neural network having two paths, a first path using an image recognition model for segmenting objects (11, 12, 13), in particular cover glasses, on an overview image of a sample carrier (10) of a microscope system (1), and a second path forms an image recognition model for detecting properties of the objects, having the following work steps: generating (201) a data stream, in particular an image stream of a large number of Overview images which depict sample carriers of a microscope system (1) on which objects (11, 12, 13) are arranged; Reading (202) ground truth carrier data, wel che at least one real property characterize at least one region (14), in which the objects (11, 12, 13) are arranged on the sample carrier (10), and ground truth auxiliary data, which at least one real property of the objects (11, 12, 13) characterize; and providing (203) the data stream with the corresponding respective ground truth carrier data and ground truth auxiliary data to the artificial neural network. Computer-implemented method (200) for training two artificial neural networks for identifying objects (11, 12, 13), the first artificial neural network having an image recognition model for segmenting objects (11, 12, 13), in particular cover glasses, on an overview image a sample carrier (10) of a microscope system (1), and the second artificial neural network forms an image recognition model for detecting properties of the objects, having the following work steps: Generating (201) a data stream, in particular an image stream of a multiplicity of overview images which depict sample carriers of a microscope system (1) on which objects (11, 12, 13) are arranged; Reading in (202) ground truth carrier data which characterize at least one real property of at least one region (14) in which the objects (11, 12, 13) are arranged on the sample carrier (10), and ground truth auxiliary data, which characterize at least one real property of the objects (11, 12, 13); and Providing (203) the data stream with the corresponding respective ground truth carrier data and ground truth auxiliary data to both artificial neural networks. System (2) for identifying objects (11, 12, 13), in particular cover glasses, on an overview image of a sample carrier (10) of a microscope system (1), having: a first interface (3) for reading in the overview image, in particular as image data, and auxiliary data, the auxiliary data characterizing at least one property of the objects (11, 12, 13); Means (4) for segmenting at least one region (14) on the sample carrier (10), in which the objects (11, 12, 13) are arranged on the sample carrier (10), on the basis of the overview image, in particular the image data, by means a first overview image recognition model, wherein the first overview image recognition model is based on a first machine learning model, in particular a first artificial neural network, wherein carrier data which characterize at least one property of the at least one region (14) is generated; means (5) for separating objects (11, 12, 13) in the at least one region (14) by means of an overview image processing method on the basis of the carrier data and the auxiliary data; and a second interface (6) for outputting information relating to the separated objects (11, 12, 13). Microscope system (1) with a system (2) for identifying objects (11, 12, 13). Claim 13 . Computer program or computer program product, wherein the computer program or computer program product contains instructions, in particular stored on a computer-readable and/or non-volatile storage medium, which, when they are carried out by a computer or a system (2). Claim 13 are executed, cause the computer or the system (2) to carry out the steps of a method (100) according to one of Claims 1 until 12 to execute.

Images