DE102020126549A1 - MICROSCOPY SYSTEM AND CALIBRATION CHECK PROCEDURE - Google Patents

Abstract

A microscopy system comprises an overview camera (9) for recording overview images (11, 12) of a sample location environment and a computing device (20) which is set up to evaluate the overview images (11, 12). The computing device (20) has calibration parameters (P) for interpreting image coordinates of the overview images (11, 12). The overview camera (9) records at least two overview images (11, 12) at different sample positions or sample table positions. A displacement image (31) for superimposing the overview images (11, 12; 21, 22) is then calculated, the overview images (11, 12; 21, 22) being converted to the same perspective with the aid of the calibration parameters (P). Based on a quality of agreement (Q) between the superimposed overview images (32), it is evaluated whether the calibration parameters (P) are valid. A calibration verification procedure is also described.

Description

TECHNICAL AREA

The present disclosure relates to a microscopy system and a method for calibration verification.

BACKGROUND

Automation plays an increasingly important role in modern microscopy systems. A sample to be examined is to be recorded, approached and examined in more detail by the microscopy system in a partially or fully automated manner. For this purpose, an overview camera records overview images of the sample and a sample environment. For example, a navigation map can be formed from an overview image, in which a user can select a location that can then be automatically approached with a motorized sample table. The overview image can also be used for automatic sample recognition, for example by recognizing the wells of a microtiter plate and optionally examining them more closely. An overview image can also be used for autofocusing, for example by estimating a sample height from the overview image or by determining a suitable position at which an autofocusing method is then carried out.

In order to be able to use an overview image in this way, a relationship between image coordinates of the overview image and spatial information must be known, for example through a pose of the overview camera (viewing angle and position) relative to a reference point on the microscope. For this purpose, the overview camera is calibrated. For example, an overview image of a calibration object with known dimensions can be recorded so that a relationship to image coordinates can be determined. Distortions of the overview camera can also be determined by such a calibration and corrected by calculation.

Accordingly, a generic microscopy system comprises an overview camera for recording overview images of a sample location environment and a computing device which is set up for evaluating the overview images, the computing device having calibration parameters for interpreting image coordinates of the overview images.

In U.S. 9,344,650 B2 and DE 10 2013 012 987 A1 was described by the applicant to carry out a calibration using a reference object in order to then process recorded images using calibration data. In DE 10 2017 109 698 A1 the applicant disclosed a microscope in which overview images from a calibrated overview camera are evaluated, for example in order to classify microscope components and to load information on recognized components. Another calibration method was developed by the applicant in DE 10 2013 222 295 A1 described for a microscope with a swivel stand to enable automatic focus and image center tracking. The evaluation of an overview image, for example to determine and automatically move to a sample position, was carried out by the applicant in DE 10 2013 006 994 A1 described. Further image evaluations for automatically executing workflows of a microscope are available from the applicant in DE 10 2019 114 117 described, for which purpose calibration and special samples are recognized in an overview image. Evaluations of overview images of a microscope are also available from the applicant in DE 10 2020 118 801 described. In this case, a distance of an overview camera from the sample plane is estimated from the overview image in particular, for which purpose reference markings in the field of view of the overview camera can be used. In DE 10 2020 101 191 the applicant has described an evaluation of overview images of a microscope, with a homograph being determined by which the overview image is converted into a different representation; the determination of the correct perspective homography can be done, among other things, by recording calibration patterns of known dimensions.

In DE 10 2018 133 188 A1 the applicant has described a microscope in which, among other things, two overview images are recorded for different sample stage positions. A location of a specific structure is determined in the two overview images. A pixel distance between the locations in the two overview images provides information about a height or a distance of the structure shown.

Ensuring proper calibration is made more difficult when a user can reconfigure, swap out, or reposition components. For example, limit switches of motorized tables can be reconfigured. Even if instructions indicate that a new calibration must be carried out after such measures, a user may overlook this information. Changes can also be caused unknowingly by impacts, loose connections or temperature changes. This means there is a risk that navigation will only take place with reduced accuracy and that collisions with damage will result ments of device components or the sample.

In order to recognize whether a calibration is necessary, the applicant in DE 10 2018 133 196 A1 presented a machine learning model that recognizes deviations from correct cases in overview images, for example using a neural network trained to detect anomalies. In this way, based on an overview image, an error message can be generated if maintenance is required. A more precise determination of whether or to what extent a calibration is still correct would be desirable.

SHORT VERSION

An object of the invention can be seen as specifying a microscopy system and a method for checking the calibration, which enable an evaluation that is as precise and reliable as possible as to whether or to what extent calibration parameters that are used to interpret an overview image are correct.

This object is achieved by the microscopy system having the features of claim 1 and by the method for checking the calibration having the features of claim 2 .

In the microscopy system of the type mentioned above, the computing device is set up according to the invention to control the overview camera for recording at least two overview images at different sample positions or sample table positions. The computing device is also set up to calculate a displacement image for superimposing the overview images, with the overview images being converted to the same perspective with the aid of the calibration parameters. Based on a quality of agreement between the superimposed overview images, the computing device assesses whether the calibration parameters are valid.

A method according to the invention for checking the calibration comprises: Obtaining at least two overview images of a microscope system at different sample positions or sample stage positions; Calculating a displacement image for superimposing the overview images, with the overview images being converted to the same perspective with the aid of calibration parameters; and assessing whether the calibration parameters are valid based on a goodness of fit between the overlaid overview images.

An example is explained for quicker understanding: The image contents of two overview images, which are recorded at different sample stage positions, should be able to be superimposed very precisely given ideal calibration parameters. If, for example, top views are calculated from the overview images with the help of the calibration parameters, an image content from an observed plane in the two top views should only be shifted relative to one another, but not distorted in terms of perspective. Therefore, with ideal calibration parameters, an image shift should be possible through which the image contents agree very well with one another and therefore have a high quality of agreement. However, if the image content in the two top views is distorted in relation to one another or scaled differently, the quality of agreement is lower. In this case, it can be concluded that only imprecise top views were calculated using the calibration parameters. As a result, poor accuracy or validity of the calibration parameters can be identified.

Optional designs

Advantageous variants of the microscopy system according to the invention and the method according to the invention are the subject matter of the dependent claims and are explained in the following description.

Perspective mapping using calibration parameters; Overview images as views

With the help of the calibration parameters, the overview images are converted to the same perspective. The perspective can be understood in particular as a viewing direction of the overview camera from a certain distance onto a plane, a structure or an object in the respective overview image. If a sample table is moved with a stationary overview camera, the shape and size of structures on the sample table change in general in overview images. By converting the overview images to the same perspective using the calibration parameters, it is possible to ensure that the shape and size of the structures in a plane are the same in the converted overview images, despite different sample stage positions.

The overview images can be converted to the same perspective as a homographic image, which maps or projects one level in space into another level. A homography can be described, for example, by a 3×3 matrix whose entries depend on a height (along the optical axis of the microscope) of a plane to be imaged, with this height being described via the calibration parameters.

In particular, a top view/top view can be calculated from each overview image. The top view corresponds to a vertical view of a specific plane. The level can, for example, correspond to a sample table surface, a top or bottom side of the sample carrier or a level of a holding frame. In plan view, a structure lying in that plane should be the same size and shape no matter where it is in the plane. If the sample table is moved in this plane, the shape and size of structures in this plane should therefore remain the same in top views. Only a shift in the structures in the overview images available as top views should occur. The prerequisite for this is that a position of the plane in relation to the overview camera has been correctly taken into account. This is done using the calibration parameters. Depending on the precision of the calibration parameters, calculated views deviate slightly from actual views, so that the shape and size of structures change slightly when the sample stage is moved. The evaluation of such deviations will be described in more detail later. The conversions of two overview images into top views can be mathematically identical if there is a movement between the recordings of the overview images within a plane that is mapped into the top view. The movement can be a displacement and/or a rotation. In principle, a movement can also take place obliquely or perpendicularly to a plane that is transferred to the top view. In these cases, the changed distance of the plane to the camera is also taken into account in the conversions to the top views with the help of the calibration parameters.

Overheads are also suitable, among other things, because optional calculations, such as the segmentation described in more detail later, can often be implemented more robustly or with less training data. However, a conversion to supervision is not mandatory. Any other pose that is the same for both images with regard to the structures to be evaluated can also be calculated with the aid of the calibration parameters. An identical pose denotes a converted identical position and orientation of the camera with respect to a structure in the overview images.

Optionally, further image transformations can also take place, for example a conversion to an isometric perspective.

segmentation

The overview images can optionally be segmented. This can be done in particular with a learned model (segmentation model) which was learned using training data with the aid of a learning algorithm. An output of the segmentation model is a segmentation mask, in particular a binary mask, in which a certain structure is characterized by a same pixel value, while another pixel value describes a background or indicates that the corresponding pixels do not belong to this structure. Which structures are characterized by the segmentation was given to the model by the training data. For example, the segmentation model can be trained to segment samples, sample carriers, sample carrier vessels or holding frames/holding frame clamps for holding a sample carrier. In principle, segmentation can also be carried out using a classic algorithm without a machine learning model. In general, a segmentation mask does not have to identify objects segmented by different pixel values, instead, e.g. the edges of segmented objects can be entered into an image or a table with coordinates of localized object shapes can serve as a segmentation mask.

The calculation of the segmentation can be done after the calculation of the same perspective. In particular, the overview images can first be transformed into top views and then segmented. If only top views are segmented, as an advantage, less training data is usually required to learn the segmentation. For example, holding frame clamps on the sample table always have the same shape in plan views, regardless of their position. Wells of a microtiter plate should have a common size and circular shape in plan views, while in other views they have different sizes and different degrees of ovality. If the segmentation is carried out before the registration (the calculation of the displacement map), this has the advantage that image content from other height levels does not have a disruptive effect on the calculation of the displacement map. For example, the segmentation can take place with respect to a top side of a (partially) transparent microtiter plate; Due to the transparency, components from other height levels are also included in the overview image and after the perspective conversion (e.g. into a top view) they are still disturbingly included, but this is removed by the segmentation.

In principle, however, segmentation can also take place before the overview images are converted to the same perspective. In this way, under certain circumstances, an impairment of the segmentation can be avoided, which would be possible in the event of an incorrect attempt to convert the overview images to the same perspective.

The segmentation can be a semantic segmentation in which a meaning is assigned to a segmentation area. For example, semantics can specify a "samples carrier", "clamping frame", "stage", or "background". Optionally, a segmentation model can also be designed for instance segmentation. Here, different objects of the same object type are distinguished from one another. Instance segmentation can be particularly advantageous in the case of objects of the same object type that are touching or overlapping, for example if the positions or distances of these objects are compared between the two overview images.

displacement mapping

The displacement mapping between the overview images can in particular be a displacement and/or rotation of an image content from one of the overview images relative to an image content of the other overview image. In principle, both overview images can also be shifted and/or rotated, for example in order to align an edge depicted in both overview images in a specified direction.

The displacement mapping can be calculated as that displacement and/or rotation that maximizes correspondence between image contents of the overview images. In particular, the overview images can be segmented and converted into top views. In this case, a displacement map that maximizes the correspondence can be calculated relatively easily from the segmented overview images. The maximization can be calculated iteratively or, for example, via a maximum of the convolution of the two images. In principle, however, the displacement mapping can also take place using the overview images, without calculating a segmentation.

Allowed limit values of the displacement and/or rotation can be specified for the calculation of the displacement mapping. The permitted limit values delimit a possible displacement range or rotation range of the images relative to one another and can be specified in particular as a function of the calibration parameters. In the case of an iterative calculation of the displacement mapping, alternatively or additionally, starting values of the iteration can also be specified as a function of the calibration parameters. In addition to the calibration parameters, the limit values and start values can also depend on data received which relate to the sample stage position or a sample stage movement carried out between the recordings of the overview images. If a sample table is moved between the recordings of two overview images, a displacement of the images can be estimated, for example, by the calibration parameters and a control command to the movable sample table, by which the sample table was moved between the recordings, or by a position value supplied by a sample table sensor. This estimated displacement can be used as a starting value for the iteration and/or for setting limit values for the displacement to be determined. If it is known that between the recordings of the overview images only a sample stage displacement has taken place, the displacement mapping can also be restricted to a displacement without rotation.

In principle, the displacement mapping can also be calculated with the aid of or solely from a known adjustment between the sample stage positions of the at least two overview images. In this variant, therefore, the image contents of the overview images do not necessarily have to be evaluated in order to define the displacement mapping. In other embodiments, on the other hand, the movement distance of a sample stage adjustment does not have to be known and also does not have to be estimated, since the overview images are shifted onto one another without the size of the corresponding image shift having to be taken into account for the further evaluation.

If the at least two overview images are first transformed into plan views using the calibration parameters, the displacement image can then be calculated as a linear displacement.

If a segmentation mask is first calculated from the at least two overview images, the displacement mapping can be calculated on the basis of the segmentation masks.

In further variants, the perspective conversion takes place with the aid of the calibration parameters together with or at the same time as the calculation of the displacement image. In this case, in particular, the unchanged overview images can be iteratively shifted or rotated relative to one another, with a perspective conversion of the shifted or rotated overview image, which is dependent on the shift/rotation, taking place for each of these shifts/rotations.

Entire overview image or only part of it

For easier linguistic understanding, various variants of the invention call an overview image, which can be understood to mean the entire overview image or just a part of it. In particular, the displacement mapping can also be determined solely from one or more image areas from the overview images. The quality of agreement can likewise be calculated either using the entire overview images or only using one or more image areas.

Remaining areas of the overview images can either be cut out or also included in the calculation; For example, the entire overview images can be shifted relative to one another in the displacement image, but the displacement image is defined solely on the basis of certain relevant areas and other image areas do not influence the definition of the displacement image. In particular, at least one relevant area can first be determined from the overview images; then the displacement map is calculated with respect to the at least one relevant area. The at least one relevant area can be determined, for example, using a machine learning model that is trained for segmentation or detection. For example, a segmentation model can be trained to distinguish holding frame brackets from the rest of the image content (=background). After such a segmentation, only a displacement or rotation of the holding frame clamps between the two overview images plays a role, while changes in the remaining image content have no effect on the determination of the displacement mapping. In the case of a detection, the image areas of one or more specified objects can be determined. For example, edges or corners of holding frame components or sample carriers can be searched for by a detection. Then only image areas around these objects are taken into account for determining the displacement mapping. For example, a shift between the overview images can be determined in such a way that in image sections around the corners of a holding frame a match between the overview images is maximum, while it is not taken into account whether in other image sections the match between the two overview images is only slight.

The displacement mapping and the quality of agreement can thus be calculated either for the entire image content or only for image areas, with the image areas either being fixed or determined using image content, e.g. using a detection model and/or a segmentation model.

Overview images for different sample or sample stage positions

A field of view of the at least two overview images covers a sample site environment. A sample location is intended to denote the location at which a sample is placed in the measurement mode. In the overview images, however, a sample does not have to be visible or present at all. For example, overview images can also be recorded from an empty sample carrier or solely from a sample table (without sample carrier).

The sample or the sample stage moves between two recorded overview images. In particular, a motorized adjustable sample table can be adjusted laterally, ie within the plane that is spanned by a sample table top. In principle, however, movements in other directions are also possible. A sample table or the sample itself can also be moved manually between the recordings of the overview images, in particular shifted laterally.

The design of the sample table is essentially irrelevant for the functioning of the described invention, which is why this can be understood very generally as a movable component via which an object to be examined can be moved. The object itself can lie on the sample table, be held in a holder or be held by clamps or a gripper arm, for example. The sample table itself can be visible in the overview image, or alternatively only the object or only an object holder of the sample table can be visible.

Any object to be examined can serve as an object or sample, e.g. a biological sample, semiconductor or electronic parts, rock or material samples.

Superimposed overview images

Superimposed overview images can be understood to mean the two overview images after calculation of a displacement mapping, through which their image contents are shifted over one another. This is also referred to as registration of the two overview images. The superimposed overview images can be formed by separate images or by a single image whose content represents an overlay of the corresponding overview images. If, for example, segmentation masks are calculated from the overview images, the segmentation masks can be superimposed to form a single image. Different pixel values in this single image indicate whether the corresponding pixel belongs to a segmented object only in the first overview image, only in the second overview image, in both overview images or in neither overview image. It can too only relevant image areas of the overview images are selected and superimposed as described.

goodness of fit

The quality of agreement represents a measure of how similar the at least two superimposed overview images are. The similarity is determined after the displacement mapping has been applied, so that the overview images should essentially match one another if the calibration parameters are valid. At least in the case of flat objects or even surfaces that lie in a plane that is assumed by the calibration parameters, the agreement should be particularly high.

As a simple method for calculating a goodness of fit, the image area of superimposed segmentation masks (after the shift) can be determined in which the two segmentation masks differ. This image area can be compared to a predetermined threshold value in order to distinguish between correct and incorrect calibration.

The displacement mapping and the calculation of the quality of agreement can also be carried out by a joint calculation process, e.g. by convolving the two overview images.

Alternatively, the quality of correspondence between the superimposed overview images can also be calculated via a ratio of overlapping areas to non-overlapping areas of the superimposed overview images. A respective deviation direction can also be calculated for each of the non-overlapping areas. For example, the overview images can be superimposed and superimposed as segmented overview images (segmentation masks). In an overlap area, both segmentation masks have the same pixel value. In non-overlapping areas, on the other hand, the two segmentation masks have different pixel values. If the calibration parameters are imprecise, a segmented object usually has several non-overlapping areas at its edges. The deviation or displacement direction of a non-overlapping area indicates in which image direction the next overlapping area (of the segmented object) is located. The ratio of overlapping areas to non-overlapping areas and the respective directions of deviation are characteristic of calibration parameters that deviate from correct calibration parameters. It can be used here that field-dependent shifts and scaling of the segmentation masks or the image content occur due to imprecise calibration parameters, which qualitatively differ from, for example, imprecise segmentations.

The quality of agreement can be determined using the entire superimposed overview images or only using specific image areas (evaluation image areas) of the overview images. In particular, a machine learning model can be used to select evaluation image areas on which the quality of agreement is calculated. At least one of the overview images or at least one image calculated from one or both of the overview images is input to the machine learning model. As an output, the machine learning model indicates image areas that are used as the evaluation image areas. For example, a learned detection model can be used, which has learned to mark certain image regions in input images using predefined training images with annotated image regions. For example, the detection model can be trained to locate all corners of a rack or sample carrier, or only certain corners, e.g., those rack corners that are adjacent to rack edges that hold a sample carrier or face the sample carrier.

The quality of match between the superimposed overview images can also be calculated using a (further) trained machine learning model, which receives the superimposed overview images or image areas thereof as input and calculates the quality of match as output.

Instead of or in addition to calculating the ratio between overlapping areas and non-overlapping areas, the quality of match can also be determined using image distances between corresponding structures in the overlaid overview images. Specific objects or parts of objects can be localized as structures, for example several corners of an object. The overview images can be shifted and/or rotated relative to one another in such a way that the image distances between the corners that correspond to one another in the two overview images are minimal. The image distances can be measured, for example, as pixel distances. If the calibration parameters are not ideal, there are systematic errors due to scaling effects, in particular the image distances for the corners point in different directions, it being possible to conclude from the sizes and directions of the image distances that the calibration parameters are valid or invalid.

Superimposed overview images, on the basis of which the quality of agreement is determined, can also be used be represented by two-dimensional sequences of points. In this case, only individual points of the overview image are used instead of the entire image content of an overview image. The quality of agreement is calculated as to how precisely the point sequences of the various overview images can be matched.

Goodness of fit over time

Optionally, more than two overview images are recorded one after the other and with the help of these overview images, a time profile of the quality of agreement is calculated. In particular, a respective quality of match can be calculated from a pair of overview images recorded one after the other. The time profile of the quality of agreement can be used to assess whether the calibration parameters are valid.

A uniquely poorer quality of agreement can be distinguished from a permanently poorer quality of agreement over a period of time. A one-off worse value can be caused, for example, by incorrect segmentation or shift calculation and does not necessarily have to result from incorrect calibration parameters. If, on the other hand, the quality of agreement is several times below a predetermined limit value, it can be concluded with certainty that the calibration parameters are invalid.

A repeated acquisition of overview images with the sample stage shifted relative to one another can often be provided in normal microscopy operation for estimating the height by means of triangulation methods. These recordings of overview images, which are therefore made anyway for height estimation, can also be used by the invention in the manner described. As a result, no additional overview image recordings and sample table movements, which would not be provided for height estimation anyway, have to be carried out for the verification of the calibration parameters presented.

Validity of calibration parameters, follow-up actions

The determination of whether calibration parameters are valid can be made as a categorical classification (e.g. yes/no) or by an indication of quality using a discrete or continuous number in a range of values.

The calculated quality of agreement can be used directly as an indication of the quality of the calibration parameters or as a measure for this. Alternatively, an indication of the quality of the calibration parameters can be inferred from an averaging or from the above-described time profile of the quality of agreement.

If the calibration parameters are classified as invalid, a warning or notice can be issued to a user that a calibration is required. Alternatively or additionally, a calibration process can also be triggered automatically.

General properties

A microscopy system is understood to mean a device which comprises at least one computing device and one microscope. In principle, a microscope can be understood to mean any magnifying measuring device, in particular a light microscope, X-ray microscope, electron microscope, macroscope or also a differently designed magnifying image recording device.

The computing device may be physically designed as part of the microscope, located separately in the microscope environment, or located at any location remote from the microscope. The computing device can also be decentralized and communicate with the microscope via a data connection. It can generally be formed by any combination of electronics and software and in particular can include a computer, a server, a cloud-based computing system or one or more microprocessors or graphics processors. The computing device can also be set up to control the microscope camera, the image recording, the sample stage control and/or other microscope components.

The overview camera for recording an overview image can be present in addition to a sample camera, with which more enlarged images of a sample area are recorded. Alternatively, however, this can also be the same camera, with different lenses or optical systems being used for recording an overview image and a more greatly enlarged sample image. The overview camera can be attached to a stationary device frame, eg a microscope stand, or to a movable component, eg a sample table, focusing drive or objective turret. The overview image can be a raw image as captured by a camera, or a processed image from one or more raw images. A recorded raw image / overview image can be further processed before it is evaluated in the manner described here. Method variants of the invention can be based on previously recorded overview images and obtain them from a memory, for example, or alternatively the recording of the overview images can also be part of the claimed method variants. Presently described images, e.g. About visual images may be composed of pixels, vector graphics, or mixtures thereof. In particular, segmentation masks can be vector graphics or can be converted into such. For easier understanding, various embodiments are described in relation to two overview images, which can be understood in the sense of exactly two or at least two overview images.

A computer program according to the invention comprises instructions which, when executed by a computer, cause one of the method variants described to be carried out.

The learned models or machine learning models described here each refer to a model that was learned by a learning algorithm using training data. The machine learning models can, for example, each include one or more convolutional neural networks (CNN), which receive at least one input image as input, in particular an overview image or an image calculated therewith. The training of the machine learning model can have taken place through a monitored learning process, in which training overview images with the respective annotation/identification were specified. A learning algorithm is used to set model parameters of the machine learning model based on the annotated training overview images. For this purpose, a given target function can be optimized, e.g. a loss function can be minimized. The loss function describes deviations between the given labels and current outputs of the machine learning model, which are calculated with the current model parameter values from training overview images. The model parameter values are changed to minimize the loss function, which can be calculated, for example, by (stochastic) gradient descent. In the case of a CNN, the model parameters can in particular include entries from convolution matrices of the various layers of the CNN. Other model architectures of a deep neural network are also possible instead of a CNN. Instead of a supervised learning process, unsupervised training can also take place, in which no annotations are specified for the training images. Partially supervised training or a learning method of reinforcement learning are also possible.

The properties of the invention described as additional device features also result in variants of the method according to the invention when used as intended. Conversely, a microscopy system can also be set up to carry out the method variants described. In particular, the computing device can be set up to carry out the method variants described and/or to output control commands for carrying out the method steps described. In addition, the computing device can include the computer program described. While a fully trained machine learning model is used in some variants, further variants of the invention result from the execution of the corresponding training steps.

character list

• 1 ist eine schematische Darstellung eines Ausführungsbeispiels eines Mikroskopiesystems der Erfindung;
• 2 ist eine schematische Darstellung von Prozessen eines Ausführungsbeispiels eines Verfahrens der Erfindung;
• 3 ist eine Fortführung der Darstellung aus 2;
• 4 ist eine schematische Darstellung von Prozessen zur Verarbeitung eines Übersichtsbildes gemäß Varianten der Erfindung;
• 5 ist ein Flussdiagramm von Prozessen eines Ausführungsbeispiels der Erfindung; und
• 6 ist ein Flussdiagramm von Prozessen eines Ausführungsbeispiels der Erfindung.

Further advantages and features of the invention are described below with reference to the attached schematic figures:

• 1 Fig. 12 is a schematic representation of an embodiment of a microscopy system of the invention;
• 2 Fig. 12 is a schematic representation of processes of an embodiment of a method of the invention;
• 3 is a continuation of the presentation 2 ;
• 4 12 is a schematic representation of processes for processing an overview image according to variants of the invention;
• 5 Fig. 12 is a flowchart of processes of an embodiment of the invention; and
• 6 Figure 12 is a flow chart of processes of an embodiment of the invention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Various exemplary embodiments are described below with reference to the figures. Components that are the same and have the same effect are usually identified by the same reference symbols.

FIG. 1

1 12 shows an exemplary embodiment of a microscopy system 100 according to the invention. This comprises a computing device 20 and a microscope 1, which in the example shown is a light microscope, but in principle can also be a different type of microscope. The microscope 1 includes a stand 2, over which further microscope components are held. This can include in particular: an objective changer or turret 3, on which an objective 4 is mounted in the example shown; a sample table 5 with a holding frame 6 for holding a sample carrier 7 and a micro scope camera 8. If the lens 4 is swiveled into the microscope beam path, the microscope camera 8 receives detection light from one or more samples held by the sample carrier 7 in order to record a sample image. The sample carrier 7 can be, for example, a microtiter plate, a slide made of a flat carrier with a cover glass, a chamber slide, a Petri dish, a gel or a gel holder.

The microscope 1 also includes an overview camera 9 for recording an overview image of a sample environment. As a result, the overview image can in particular show the sample carrier 7 or a part thereof. A field of view 9A of the overview camera 9 is larger than a field of view when recording a sample image. In the example shown, the overview camera 9 looks at the sample carrier 7 via a mirror 9B. The mirror 9B is arranged on the objective turret 3 and can be selected instead of the objective 4 . In modifications of this embodiment, the mirror or another deflection element can also be arranged at a different location. Alternatively, the overview camera 9 can also be arranged in such a way that it looks directly at the sample carrier 7 without a mirror 9B. While the overview camera 9 looks at an upper side of the sample carrier 7 in the example shown, the overview camera 9 can alternatively also be directed at an underside of the sample carrier 7 . In principle, the microscope camera 8 can also represent an overview camera if a different lens, in particular a macro lens, is selected via the lens turret 3 to record an overview image.

The computing device 20 uses a computer program 80 according to the invention in order to process overview images and optionally to control microscope components based on processing results. For example, the computing device 20 can evaluate an overview image to determine where the wells of a microtiter plate are located, in order to then control the sample stage 5 in such a way that a specific well is approached. The computing device 20 uses calibration parameters P for correct processing of an overview image and also for the implementation of how microscope components are to be controlled based on location information from an overview image. These enable an interpretation of the overview image, for example how directions in the overview image are related to directions on the sample table, in particular quantitatively. The calibration parameters P can also include information about a scaling, in particular the shape or size of objects in the overview image at a specific sample stage height. For an evaluation of an overview image, the calibration parameters P and optionally current settings of microscope components, for example a current height of a motorized sample table, can then be taken into account. Alternatively, without knowing the settings of microscope components, the height of a sample carrier can also be estimated from the overview image using the calibration parameters P, for example. In particular, the calibration parameters P can be used to describe a relationship between location information from an overview image and location information relating to a microscope reference location.

The calibration parameters P can become invalid if a user exchanges or repositions microscope components. Impacts, loose connections or the effects of environmental parameters such as changes in air humidity or temperature can also lead to maladjustments of microscope components, as a result of which the calibration parameters P become inaccurate or incorrect. In order to determine such changes automatically, the computing device 20 analyzes overview images that were recorded at different positions. This is described in more detail with reference to the following figures.

FIG. 2 and 3

the 2 and 3 schematically illustrate sequences of an embodiment of a method of the invention. The method can start from the computer program or the computing device 1 to be executed.

In 2 At least two overview images 11 and 12 are first obtained via the same overview camera, with the recorded structures being moved between the image recordings. In the example shown, the overview images 11 and 12 show a sample carrier 7 with a round cross section, which is held between two holding frame clamps 6A and 6B of a holding frame. A surface of the sample table 5 is also visible as a background. The sample table was moved between the two image recordings, as a result of which the holding frame clamps 6A, 6B and the sample carrier 7 are shifted to the right in the overview image 12 . In principle, displayed objects could also be shifted manually between the two image recordings.

In the example shown, the image content of recorded raw images has already been converted into top views, which can be done by homographic imaging using the calibration parameters. Top views correspond to a vertical view of a sample stage plane and thus also perpendicular to a top side of the holding frame clamps 6A, 6B and a top side of the sample rägers 7. The same homographic mapping can be done for both images. The result of this is that a displacement of the sample table between the image recordings (in a direction perpendicular to the direction of view of the top views) should only lead to a displacement in the overview images 11 and 12 present as top views. If no top views were calculated, the displacement of the sample table would result in image content not only being displaced between the overview images, but also being perspectively distorted in accordance with the displacement.

The execution variant of 2 uses the fact that with an exact calculation of top views, the displacement of the sample table should lead to an exact displacement of image contents of a plane in the overview images 11, 12 present as top views. If this is not the case, it can be concluded that the plan views were not precisely calculated. Because the top views were calculated using the calibration parameters, this means that the calibration parameters used are inaccurate or invalid.

The implementation of these steps takes place according to 2 , in that the two overview images 11 and 12 present as plan views are first entered into a segmentation model S in step S1. The segmentation model S is a learned model, which can include a deep neural network, eg a CNN. The present segmentation model S was trained using annotated training data to distinguish holding frame brackets from the rest of the image content. The segmentation model S outputs a segmentation mask for each input image, ie the inputted overview image 11 is calculated to form a segmented overview image 21 and the inputted overview image 12 is calculated to form a segmented overview image 22, step S2.

In the segmented overview images 21, 22, a specific pixel value designates image areas 26 that have been classified as holding frame brackets, while the remaining image areas are classified as background 27 by a different pixel value. As can be seen in the exemplary segmented overview images 21, 22, minor segmentation errors can occur here.

The segmented overview images 21, 22 are now entered in step S3 into a shift calculation program 30, which is set up to calculate a shift image 31 in order to superimpose the segmented overview images 21, 22 as consistently as possible. In this example, since there are plan views and the sample stage movement is within or parallel to the plane depicted in the plan views, the displacement map 31 may be set as a linear displacement. The shift calculation program 30 now iteratively calculates for which shift between the two segmented overview images 21, 22 a match of the image contents is maximum. Knowledge of the movement of the sample table is not absolutely necessary and can be used optionally, e.g. to define start values of the iteration or limit values of the displacement to be calculated. Alternatively, a shift for the segmented overview images 21, 22 can also be calculated from a known sample stage movement using the calibration parameters, without the image content of the segmented overview images 21, 22 playing a role in this calculation for the size of the shift.

A superimposed overview image 32 is calculated in step S4 with the calculated displacement image 31 . This corresponds to a pixel-by-pixel addition of the two overview images 21, 22 after they have been shifted relative to one another according to the shift image 31. In the example shown, the overview image 22 is shifted to the left and then a pixel-by-pixel addition can take place, for example by addition, subtraction, multiplication or division of the respective pixel values of the overview images 21 and 22.

The superimposed overview image 32 includes overlapping areas 38 in which the segmented overview images 21 and 22, which have been shifted relative to one another according to the displacement image 31, both show a segmented object (here one of the holding frame clamps 6A, 6B). The superimposed overview image 32 also includes a background 37 in image areas in which the segmented overview images 21 and 22 , which have been shifted relative to one another according to the displacement image 31 , both show a background 27 . In addition, the superimposed overview image 32 includes non-overlapping areas 38 and 39. In non-overlapping areas 38, an object (holding frame clamp) was only determined in the overview image 21, but not in the overview image 22 shifted according to the displacement image 31. Similarly, in the non-overlapping areas 39, only in the Overview images 22 locate an object, but not in the overview image 21, the overview images 21, 22 being compared to one another after the displacement image 31 has been applied.

The description of the further procedure is with reference to 3 continued. In step S5, the superimposed overview image 32 is supplied to a detection model 40, which is trained to detect specific image areas, subsequently evaluation image areas 41-44. That Detection model 40 is a machine learning model learned from annotated training images. In principle, instead of the superimposed overview image 32, one or more of the overview images 11, 12, 21, 22 can also serve as input. In step S6, the detection model 40 outputs delimitations of the determined evaluation image areas 41-44, which in 3 are drawn into the superimposed overview image 32 for better understanding.

In addition, the evaluation image areas 41-44 are enlarged in the left half of FIG 3 shown. The right half of 3 shows, by way of example, corresponding evaluation image areas 41-44 which were calculated for two other overview images in the manner described above.

In particular, the detection model 40 can be trained to find edges or corners of segmentation masks and to define them as evaluation image areas 41-44. Image areas around such edges or corners can be evaluated in a particularly meaningful way as to how high an image match is. In addition, it can be taken into account in which direction within the evaluation image areas 41-44 displayed image contents, which originate from the two underlying overview images 11, 12 or 21, 22, have been incorrectly shifted or deviate. For example, the non-overlapping area 38 in the evaluation image area 42 in the example on the left is off 3 relatively narrow, but relatively wide in the example on the right. A deviation direction can be defined as the direction of the area 38 to the overlapping area 36 .

The accuracy of the calibration parameters determines an area size of the non-overlapping areas 38 and 39, how the non-overlapping areas 38, 39 relate to one another and what their respective directions of deviation are. In addition, an imprecise segmentation can be distinguished from imprecise calibration parameters on the basis of the aforementioned features. In principle, these features can be evaluated by a classic algorithm without a learned model, with a trained machine learning model 50 being used for this purpose in the example shown. This can include a deep neural network, for example a CNN, of which model parameters P1-P9 are shown as examples as entries in a convolution matrix. The machine learning model 50 can be designed as a classification or regression model and receives the multiple evaluation image areas 41-44 as a common input in step S7. From this input, the machine learning model 50 calculates a quality of match Q as a measure of a match between the superimposed overview images in step S8. The quality of agreement Q is, for example, a classification into good or bad agreement, although finer classifications or continuous values are also possible as output. In this example, the quality of agreement Q is used directly as an assessment of whether the calibration parameters are valid. In the left example off 3 a validity is affirmed, but denied in the example on the right. With a finer gradation of the validity of the calibration parameters, an accuracy of the calibration parameters can also be specified in this way.

FIG. 4

4 1 illustrates how, in various exemplary embodiments of the invention, an overview image is processed into a top view image and further into a segmentation mask.

In the example shown, an overview image 11 ′ was recorded, with an overview camera viewing an upper side of a sample carrier 7 at an angle. The sample carrier 7 is a microtiter plate with several circular wells as sample vessels. Also visible is a condenser 10.

In step S0, the computer program 80 uses the calibration parameters P to convert the overview image 11' into a different perspective, namely into a top view, as illustrated in the overview image 11". For a plane of the top side of the sample carrier 7, the top view corresponds to a view perpendicular to Top of the sample carrier 7. The representations of other levels in the overview image 11" (e.g. bottom areas of the partially transparent sample carrier 7), on the other hand, do not correspond to a top view. In order to convert the plane of the upper side of the sample carrier 7 correctly into a plan view, a height plane of the upper side of the sample carrier 7 must be precisely known through the calibration parameters.

The overview image 11" present as a top view is segmented next, as is also the case with 2 described. In step S1, the overview image 11" is thus supplied to a segmentation model S', which in this case is trained to segment sample vessels from sample carriers 7. In step S2, the segmentation model S' outputs a segmentation mask, with an overlay 21' from the overview image 11" and the segmentation mask.

The further process steps can as to 2 and 3 be carried out as described.

FIG. 5

5 shows a flowchart to explain different sequences of the steps of exemplary embodiments of the method according to the invention.

At least two overview images are recorded in step S10, in particular by the same camera. A recorded object, e.g. a sample or a sample table, is shifted or otherwise moved between the recordings.

Subsequently, in step S11, the processes of segmenting, shifting and converting to a common perspective are carried out with the aid of the calibration parameters with regard to the overview images. The order of these processes can be chosen variably.

For example, as with the 2 until 4 First, a homographic conversion of one or more of the overview images to a common perspective takes place, in particular a homographic mapping of all overview images to a top view, step S12. Then, in step S13, the overview images present as plan views are segmented. Then, in step S14, a displacement map for superimposing the segmented overview images present as plan views is calculated.

The sequence can be changed such that the overview images are first segmented in step S13' and then in step S12' a homographic mapping of the segmented overview images into plan views is calculated with the aid of the calibration parameters. This may be followed by step S14.

A further possible sequence provides that the segmentation takes place first according to step S13' and then in step S14' a displacement mapping is calculated, in which at the same time a perspective adjustment takes place with the aid of the calibration parameters. If the image content of an overview image is shifted by a certain distance, a perspective adjustment takes place in the same way as would correspond to a shift of the objects shown in actual space, which can be calculated with the aid of the calibration parameters.

Step S14 or S14' is followed by step S15, in which the quality of correspondence between the superimposed (shifted one on top of the other) overview images present as segmentation masks is calculated in order to evaluate the validity of the calibration parameters.

Further variants of the illustrated exemplary embodiments are possible. For example, the segmentation described can be implemented as semantic segmentation or as instance segmentation. the to 3 The determination of evaluation image areas described above can be replaced by selecting one or more objects, which were determined by instance segmentation, or predetermined object parts thereof, as evaluation image areas. In addition, the entire superimposed overview image 32 can also be switched off 2 can be used as input to the machine learning model 50 to calculate the goodness of fit Q . In further versions, the segmentation can be omitted. For example, the detection according to step S6 can be switched off 3 can also be carried out on overview images, which have optionally been converted to top views, without a segmentation having been calculated beforehand.

In particular, the locations of a number of objects in the respective overview image can be determined by the detection, with the locations forming a point pattern for each overview image. A match between the dot patterns after displacement determines the goodness of fit.

FIG. 6

6 shows a flowchart of a method variant in which the verification of the calibration parameters is integrated into a procedure for triangulation-based height estimation. As will be explained, no additional image acquisitions and no additional devices or markings are required to supplement the verification of the calibration parameters for a triangulation-based height estimation.

In step S20, a procedure for triangulation-based height estimation is started. This can be done, for example, as part of an automatic navigation on a microscope.

In step S21, at least two overview images of the same objects are recorded at two sample stage positions that are laterally shifted relative to one another.

In step S22, the height is estimated according to the triangulation principle. For this purpose, the fact that the size of an image shift caused by the sample stage shift is dependent on a height level at which the structures shown are located is used. The structures or objects shown can be, for example, parts of a sample carrier, a holding frame or a sample table. The distance of the sample table displacement can be known here. A connection of the image displacement to a Elevation level is determined using specified calibration parameters.

In addition to step S22, the validity or accuracy of the calibration parameters is now estimated in step S23 (simultaneously, beforehand or subsequently) in the manner already described, in particular according to steps S11 and S15 5 .

Then, in step S24, the estimated value of the elevation level can be output together with an indication of the accuracy of this estimate, based on the quality of agreement determined as described.

The segmented overview images converted to plan views can be used for both calibration verification and height estimation. Likewise, the determined displacement can optionally be used for the triangulation-based height estimation.

Depending on the result from S24, for example, a calibration process can be started or a component control/movement can be adjusted. For example, a safety distance to avoid collisions can be defined depending on the accuracy or validity of the calibration parameters. In a modification of this embodiment, step S23 is carried out first to check the calibration parameters, and step S22 to estimate the height level follows only if the calibration parameters are confirmed to be valid.

The embodiments described are purely illustrative and modifications thereof are possible within the scope of the appended claims.

Reference List

1

microscope

2

tripod

3

nosepiece

4

microscope lens

5

rehearsal table

6

holding frame

6A

holding frame clamp

6B

holding frame clamp

7

sample carrier

8th

microscope camera

9

overview camera

9A

Field of view of the overview camera

9B

mirror

10

condenser

11, 12

overview screen

11'

Overview image before perspective conversion to a top view

11''

Overview image after perspective conversion to a top view

20

computing device

21

segmented overview image

21'

Overlay of the overview image 11" and an associated segmentation mask

26

Image areas segmented as holding frame brackets

27

Image areas that have been segmented as background

30

displacement calculation program

31

displacement mapping

32

superimposed overview image

36

Overlapping areas in the superimposed overview image

37

Background in the overlaid overview image

38

non-overlapping areas in the overlaid overview image

39

non-overlapping areas in the overlaid overview image

40

Detection model / machine learning model

41-44

evaluation screen areas

50

machine learning model

80

Computer program of the invention

100

Microscopy system of the invention

P

calibration parameters

P1-P9

model parameters

Q

goodness of fit

S

segmentation model

S'

segmentation model

S0-S8

Process steps of embodiments of the invention

S10-S15, S12'-S14'

Process steps of embodiments of the invention

S20-S24

Process steps of embodiments of the invention

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• US9344650B2 [0005]
• DE 102013012987 A1 [0005]
• DE 102017109698 A1 [0005]
• DE 102013222295 A1 [0005]
• DE 102013006994 A1 [0005]
• DE 102019114117 [0005]
• DE 102020118801 [0005]
• DE 102020101191 [0005]
• DE 102018133188 A1 [0006]
• DE 102018133196 A1 [0008]

Claims (16)

Microscopy system comprising an overview camera (9) for recording overview images (11, 12) of a sample location environment and a computing device (20) which is set up for evaluating the overview images (11, 12), the computing device (20) using calibration parameters (P) for interpretation of image coordinates of the overview images (11, 12); characterized in that the computing device (20) is set up to - control the overview camera (9) to record at least two overview images (11, 12) at different sample positions or sample stage positions; - to calculate a displacement image (31) for superimposing the overview images (11, 12; 21, 22), the overview images (11, 12; 21, 22) being converted to the same perspective with the aid of the calibration parameters (P); and - to assess whether the calibration parameters (P) are valid based on a quality of agreement (Q) between the superimposed overview images (32). Calibration verification method comprising Obtaining at least two overview images (11, 12) of a sample location environment at different sample positions or sample stage positions of a microscope system; calculating a displacement image (31) for superimposing the overview images (11, 12; 21, 22), the overview images (11, 12; 21, 22) being converted to the same perspective with the aid of calibration parameters (P); and Assess whether the calibration parameters (P) are valid based on a goodness of fit (Q) between the overlaid overview images (32). Method according to the preceding claim, wherein the at least two overview images (11, 12) are transformed into a plan view (S0) with the aid of the calibration parameters (P) and the displacement image (31) is then calculated as a linear displacement. procedure after claim 2 or 3 , wherein the at least two overview images (11, 12) are first segmented (S2) by a segmentation model (S) and the displacement mapping (31) is calculated based on the segmented overview images (21, 22). Procedure according to one of claims 2 until 4 , wherein the displacement map (31) is calculated as a displacement that maximizes correspondence between the overview images (11-12; 21-22). Procedure according to one of claims 2 until 5 , wherein depending on the calibration parameters (P) allowed displacement limit values for the calculation of the displacement image (31) are specified and / or wherein depending on the calibration parameters (P) starting values of an iterative calculation of the displacement image (31) are specified. Procedure according to one of claims 2 until 6 , wherein the displacement image (31) is calculated using a known displacement between the sample stage positions of the at least two overview images (11, 12). Procedure according to one of claims 2 until 7 , a perspective change corresponding to the displacement image (31) being calculated together with the calculation of the displacement image (31), also with the aid of the calibration parameters (P). Procedure according to one of claims 2 until 8th , wherein at least one relevant area is determined from the overview images (11, 12) and the displacement image (31) is calculated with respect to the at least one relevant area. Method according to the immediately preceding claim, wherein the at least one relevant area is determined via a machine learning model which is trained for segmentation or detection. Procedure according to one of claims 2 until 10 , wherein a ratio of overlapping regions (36) to non-overlapping regions (38, 39) of the superimposed overview images (32) is calculated to calculate the quality of agreement (Q) between the superimposed overview images (32), with each non-overlapping region (38, 39) a respective deviation direction is calculated. Procedure according to one of claims 2 until 11 , wherein a machine learning model (40) selects evaluation image areas (41-44) on which the quality of agreement (Q) is calculated, wherein the machine learning model (40) is assigned at least one of the overview images (11, 12) or at least one image (32 ) is input and the machine learning model (40) indicates as output image areas which are used as the evaluation image areas (41-44). Procedure according to one of claims 2 until 12 , wherein the calculation of the quality of agreement (Q) between the superimposed overview images (32) takes place via a trained machine learning model (50), which receives the superimposed overview images (32) or image areas thereof as input and calculates the quality of conformity (Q) as output. Procedure according to one of claims 2 until 13 , wherein the quality of agreement (Q) is determined on the basis of image distances between corresponding structures in the superimposed overview images (11, 12). Procedure according to one of claims 2 until 14 that more than two overview images (11, 12) are recorded one after the other and with the help of these overview images (11, 12) a chronological progression of the quality of agreement (Q) is calculated, that the chronological progression of the quality of agreement (Q) is used to assess whether the Calibration parameters (P) are valid. Computer program with instructions which, when executed by a computer, carry out the method according to one of claims 2 until 15 cause.

Images