CN113933984A

CN113933984A - Method and microscope for generating an image composed of a plurality of microscope subimages

Info

Publication number: CN113933984A
Application number: CN202110699203.4A
Authority: CN
Inventors: 格哈德·莫勒
Original assignee: Carl Zeiss Microscopy GmbH
Current assignee: Carl Zeiss Microscopy GmbH
Priority date: 2020-07-14
Filing date: 2021-06-23
Publication date: 2022-01-14
Anticipated expiration: 2041-06-23
Also published as: CN113933984B; DE102020118500A1

Abstract

The invention relates to a method and a microscope for generating an image composed of a plurality of microscopic sub-images. According to the method, an overview image of the sample is taken. Further, a first set of microscopic sub-images of the sample is taken. The first set of microscopic sub-images have the same resolution. The positioning of the first set of microscopic sub-images within the overview image is detected. The first set of microscopic sub-images is stitched into a first set of images (02). In addition, a second set of microscopic sub-images of the sample is taken. The microscopic sub-images of the second set have the same resolution. The positioning of the second set of microscopic sub-images within the overview image is detected. The second group of microscopic sub-images (06) is stitched to form a second group of images (03). At least the first set of images (02) and the second set of images (03) are stitched into a composite image (09) of the sample based on the detected locations of the first set of microscopic sub-images and the detected locations of the second set of microscopic sub-images.

Description

Method and microscope for generating an image composed of a plurality of microscope subimages

Technical Field

The invention relates firstly to a method for generating an image of a sample composed of a plurality of microscopic sub-images. Several microscopic sub-images of the sample are taken for Stitching, e.g. by Stitching, to obtain a high resolution microscopic image of the sample, which images at least a larger part of the sample. The invention also relates to a microscope suitable for this purpose.

Background

DE 10005852C 2 shows a method for the creation of height images of technical surfaces in a microscopically resolved manner. After the measurement of the sample regions, the sample is traversed in the X/Y plane perpendicular to the optical axis of the measuring microscope, so that adjacent sample regions can be measured and a composite image of the surface structure of the sample is obtained by electronic juxtaposition, that is to say by stitching the measurement results of the respective adjacent regions.

EP 3156967B 1 teaches a method for creating a microscopic observation panorama, in which a first and a second microscopic observation shot are stitched together taking into account the change information. For this purpose, motion vectors are known which describe lateral movements in the x and/or y direction.

A method for generating a merged image of a specimen on a slide is known from EP 2596472B 1. The displacement vector between pairs of mutually overlapping sub-images is known by analyzing the similarity of the overlapping sub-images. The optimization problem of finding the best set of capture map localization variables for a sub-image in order to minimize the extent of deviations between the displacement vectors of pairs of mutually overlapping sub-images on the one hand and the extent of differences in the capture map localization variables of pairs of mutually overlapping sub-images on the other hand should be solved while respecting the auxiliary conditions for the capture map localization variables.

DE 102013006994 a1 teaches a method for optimizing the workflow in a digital microscope. The digital microscope includes at least one monitoring sensor for observing a sample, a sample stage, an optical unit, or a user, and a monitoring unit. The observation data from the monitoring sensors are detected and automatically analyzed and evaluated in the monitoring unit in order to generate control data and to use this control data for controlling the workflow of the digital microscope. The overview image or the overall image can be generated by stitching together the individual images, i.e. by stitching together a plurality of photomicrographs.

DE 102018104704 a1 shows a method for changing the magnification of a digital microscope having at least two automatically switchable objectives with different imaging scales, from which one objective is selected in each case. The magnification of an image output through the microscope is continuously changed, and for this purpose, the magnification of an image photographed with a selected objective lens and converted by an image converter is continuously changed by digital image processing. The microscope stage can be moved, for example, in the y direction. In this case, an image field equivalent to the sensor is recorded in the sensor image plane and stitched together in a stitching manner over the displacement path of the microscope stage.

DE 102017101188 a1 relates to a method for microscopic observation of a specimen with a microscope. The field of view of the microscope can be changed by selecting a section of the image sensor. In one step of the method, an initial image of at least one subregion of the sample is taken with a microscope, for which purpose a first field of view is selected on the microscope. The initial image is analyzed in order to know the field of view of at least two different imaged sub-areas, wherein a sub-area of the initial image is imaged by the field of view of each of the imaged sub-areas. An image of the subregion of the sample is taken for each of the known fields of view in which the subregion is imaged.

WO 2017/196885 a1 shows a method for producing color images with high color fidelity and high resolution, according to which a plurality of holographic images taken at a single wavelength are fused with a number of color-calibrated images. The color-corrected image is photographed with a microscope having a lower magnification using a wavelet transform-based coloring method. A Discrete Wavelet Transform (DWT) is used in order to transmit a final image that stitches together the components with lower resolution from the color-aligned image and the components with high resolution from the high-resolution monochromatic holographic image.

Generating an image composed of many microscopic sub-images (e.g. an image composed of more than 1000 microscopic sub-images) is problematic, because in many cases this will either make the amount of calculations provided too large, or the software provided is not suitable for this, and has to be developed further. A further problem of the known solutions is that if an error occurs while taking an image, all previously taken images must be taken again, which is particularly troublesome when there are a large number of images. During the taking of a very large number of microscopic sub-images, the focus setting may also need to be corrected; for example in the case of a tilted sample. Taking a large number of microscopic sub-images at the same high resolution will result in a considerable amount of data, which is not always necessary when, for example, it is not necessary to microscopically observe a specific area of the sample at high resolution. The same is true in the case where only some regions of the sample are captured according to the extended capture mode, for example as a three-dimensional image or as an image with an extended depth of field.

Disclosure of Invention

Based on the prior art, the object of the invention is to be able to generate images composed of a plurality of microscopic sub-images with little effort.

The object is achieved by the method according to the invention and by the microscope according to the invention.

The method according to the invention is used to generate a microscopic image of a sample that is composed of a plurality of microscopic sub-images. The resultant microscopic image images at least a majority of the sample. The resultant microscopic image may, for example, image the entire sample. The synthesized microscopic image may also be a panoramic image of the sample.

In one step of the method, an overview image of the sample is taken with a microscope, which overview image images at least a majority of the sample. For example, the overview image may image the entire sample. The overview image has a relatively low resolution, in particular the overview image images that part of the sample that should be imaged with a higher resolution through the composite microscopic image to be generated at a relatively low resolution.

In a further step of the method, a first set of microscopic sub-images of the sample is taken. The microscopic sub-images of the first group have the same resolution, which is in particular higher than the resolution of the overview image. These microscopic sub-images each image a small portion of the content imaged by the overview image. These microscopic sub-images may also be referred to as tile tiles. The positioning of the first set of microscopic sub-images within the overview image is detected. This is therefore the relative positioning of the first set of microscopic sub-images with respect to the overview image.

In a further step of the method, the first set of microscopic sub-images is stitched into a first set of images, which may be done, for example, by stitching. The positioning of the first set of microscope sub-images relative to the overview image is also known, since the positioning of the first set of microscope sub-images within the overview image is detected.

In a further step of the method, a second set of microscopic sub-images of the sample is taken. The microscope sub-images of the second group have the same resolution, which preferably differs from the same resolution of the microscope sub-images of the first group of microscope sub-images and in particular is higher than the resolution of the overview image. These microscopic sub-images each image a small portion of the content imaged by the overview image. These microscopic sub-images may also be referred to as tile tiles. The positioning of the second set of microscopic sub-images within the overview image is detected. This is therefore the relative positioning of the second set of microscopic sub-images with respect to the overview image.

In a further step of the method, the second set of microscopic sub-images is stitched into a second set of images, which may be done, for example, by stitching. The positioning of the second set of microscope sub-images relative to the overview image is also known, since the positioning of the second set of microscope sub-images within the overview image is detected.

In a further step of the method, at least the first and second sets of images and possibly further sets of images and/or other image elements are stitched to form a composite image of the sample depending on the detected locations of the first set of microscope sub-images and the detected locations of the second set of microscope sub-images. Stitching of sets of images may also be referred to as virtual scanning, since this may lead to the same result as scanning a sample over a large area. Preferably, in a further step, the overlapping regions of the sets of images are smoothed by stitching, so that visible jumps between the sets of images are avoided.

A particular advantage of this method is that it can be carried out with little effort, since the stitching of the partial images takes place at two hierarchical levels, since the partial images are first stitched into a group image, for which purpose the hardware and software already provided can be used. By using the detected group locations, stitching of the group images can be done with little effort. Furthermore, the stitching of the sub-images at the two hierarchical levels also allows the taking of sub-images with different resolutions, so that only important parts of the sample need to be microscopically observed with a very high resolution. Refocusing can also be done for each group of sub-images, which is advantageous in case of e.g. tilted samples, and sharpness jumps are avoided.

A preferred embodiment of the method comprises the step of relating to a third set of microscopic sub-images of the sample. A third set of microscopic sub-images of the sample is taken. The microscope sub-images of the third group have the same resolution, which preferably differs from the same resolution of the microscope sub-images of the first group of microscope sub-images and from the same resolution of the microscope sub-images of the second group of microscope sub-images, and in particular is higher than the resolution of the overview image. These microscopic sub-images each image a small portion of the content imaged by the overview image. These microscopic sub-images may also be referred to as tile tiles. The positioning of the third set of microscopic images within the overview image is detected. This is therefore the relative positioning of the third set of microscopic sub-images with respect to the overview image. In a further step of the method, a third set of microscopic sub-images is stitched into a third set of images, which may be done, for example, by stitching. Since the positioning of the third set of microscope sub-images within the overview image is detected, the positioning of the third set of images relative to the overview image is also known. The first and second sets of images are jointly stitched with the third set of images into a composite image of the sample in dependence on the detected locations of the first set of microscope sub-images, the detected locations of the second set of microscope sub-images and the detected locations of the third set of microscope sub-images.

The method is not limited to three sets of microscopic sub-images of the sample. Thus, a preferred embodiment of the method comprises the step of referring to one or more further sets of microscopic sub-images of the sample, whereby one or more further sets of microscopic sub-images of the sample are taken. The microscope sub-images of the other set or sets have the same resolution, which preferably differs from the same resolution of the microscope sub-images of the remaining sets of microscope sub-images, and in particular is higher than the resolution of the overview image. These microscopic sub-images may also be referred to as tile tiles. The positioning of one or more further sets of microscope sub-images within the overview image is detected separately. This is therefore the relative positioning of the respective further set of microscopic sub-images with respect to the overview image. In a further step of the method, one or more further sets of microscopic sub-images are respectively stitched into a further set of images, which may be done, for example, by stitching. Since the positioning of the respective further set of microscope sub-images within the overview image is detected, the positioning of the respective further set of images relative to the overview image is also known. The first, second and third sets of images are jointly stitched with one or more further sets of images into a composite image of the sample in dependence on the detected locations of the first set of microscope sub-images, the detected locations of the second set of microscope sub-images, the detected locations of the third set of microscope sub-images and the detected locations of the further set of microscope sub-images or the detected locations of the further sets of microscope sub-images.

The number of groups of microscopic sub-images is the same as the number of groups of images, preferably at least three, and more preferably at least ten.

The method is not limited to those sets of microscopic sub-images of the sample having different resolutions of the respective sub-images. Thus, for example, in addition to the first or second set of sub-images, two sets of sub-images may be taken and respectively stitched into a set of images, wherein the resolution of the sub-images of one set is not different from the resolution of the sub-images of the other set.

In a preferred embodiment of the method, the entire set of sub-images has the same field of view as the overview image as a whole. Thus, the full set of sub-images as a whole images the same section of the sample as the overview image. This also applies to the resulting composite image of the sample. The composite image of the sample preferably has the same field of view as the overview image. Thus, the composite image images the same section of the sample as the overview image. In this respect, the composite image represents an improvement of the overview image, wherein the improvement can be formed by a higher resolution, but also by an increased depth of field or three-dimensional information. The overview image thus represents a preview of the composite image that is already available before many sub-images with higher resolution are captured and processed.

In an alternative preferred embodiment of the method, the entire set of sub-images has a smaller field of view than the overview image as a whole. Thus, the full set of sub-images as a whole images a smaller section of the sample than the overview image. This also applies to the resulting composite image of the sample. The composite image of the sample preferably has a smaller field of view than the overview image. Thus, the composite image images a smaller section of the sample than the overview image. In this respect, the composite image represents an improvement of the overview image, wherein the improvement can be formed by a higher resolution, but also by an increased depth of field or three-dimensional information. The overview image thus represents a preview of the composite image that is already available before many sub-images with higher resolution are captured and processed.

The positioning of the respective group of microscope sub-images within the overview image is preferably detected in that the positioning of the microscope sub-images in the respective group within the overview image is first detected, which is preferably carried out during the acquisition of the respective microscope sub-images. The locations of the respective groups are then determined from the locations of the microscopic sub-images in the respective groups.

Stitching of the sets of microscopic sub-images into respective sets of images may be performed according to methods known in the art, such that existing hardware or software may be used. For this purpose, an overlap region between every two partial images is preferably determined. The sub-images are pieced together according to the overlapping area determined in this way. This stitching is also referred to as texture stitching. Alternatively, the position of the respective partial image relative to the sample is preferably detected in each case, in particular during the acquisition of the partial image. The sub-images are stitched according to the detected positioning. These positions are preferably measured in the form of stage coordinates associated with the sample stage.

In a preferred embodiment of the method, the microscopic sub-images of the first set have the same image size. The microscope sub-images of the second set also have the same image size, but are different from the same image size of the microscope sub-images of the first set. The advantage of this method is therefore that different sets of sub-images can be captured with different image resolutions and with different sizes. The size of the partial images, that is to say the image size of the partial images, is preferably defined by the number of pixels in each case, in particular by the image height and the image width in pixels. The image height and image width of the microscopic sub-images, respectively, are preferably at least 1000 pixels. In the case of taking a third set of microscopic sub-images and possibly further sets of microscopic sub-images, the microscopic sub-images in the respective sets preferably have the same size, wherein the sizes differ between the sets.

The sets of images preferably have different sizes. The size of the group of images, that is to say the image size of the group of images, is preferably defined by the number of pixels in each case, in particular by the image height and the image width in pixels. The image height and image width of the sets of images are preferably at least 10000 pixels each.

The respective sets of microscopic sub-images are preferably associated separately so that they image a coherent region of the sample. The coherent region is preferably rectangular. The respective groups of partial images are preferably arranged in a matrix. The microscopic sub-images of the individual groups arranged in a matrix form thus each form a matrix. These matrices have columns and rows, wherein the columns and rows each have a length preferably between one and ten subimages. Wherein each group preferably comprises between 3 and 100 microscopic sub-images.

In a preferred embodiment of the method, the groups of microscope sub-images, the group images, the overview image and the composite microscope image are each rectangular.

The composite image has an image size preferably defined by the number of pixels, preferably defined by the image height and image width in pixels. The image height and the image width of the composite image are preferably at least 20000 pixels, and more preferably at least 30000 pixels, respectively.

The sum of all groups of sub-images is preferably at least 100 sub-images; more preferably at least 300 sub-images; more preferably at least 500 sub-images, even more preferably at least 1000 sub-images.

It is possible that the sub-image is photographed not only as a general two-dimensional image but also as an image with an extended depth of field or as a three-dimensional image. The Extended Depth of Field is also known as Extended Depth of Field (EFoF or EDF). In order to capture microscopic sub-images with an extended depth of field or three dimensions, it may be necessary to capture a plurality of image planes with different focal positions or viewed from different angles and process them into respective microscopic sub-images. Preferably, the sets of microscope sub-images are respectively captured according to a capture mode selected from the group consisting of: two-dimensional image capture, three-dimensional image capture, and image capture with extended depth of field. It is preferable to capture these sub-image groups according to different capture modes because this method has an advantage that not all groups have to be captured according to the same capture mode. For example, a first set of microscopic sub-images may be taken as two-dimensional images, while a second set of microscopic sub-images may be taken as three-dimensional images.

The microscope according to the invention is preferably digital and comprises firstly an objective for magnifying optical imaging of the sample. The objective lens comprises optical components for magnifying optical imaging of the sample. These optical components are preferably formed by optical lenses and possibly also by one or more mirrors, diaphragms and filters. The microscope preferably further includes a zoom lens and a tube lens.

The microscope further comprises an image sensor for converting an image directly or indirectly imaged by the objective lens onto the image sensor into an electrical signal.

The microscope further comprises an electronic image processing unit for processing the electrical signal generated by the image sensor. The image processing unit is configured to perform the above-described method. The image processing unit is preferably configured to implement the preferred embodiments of the method described above. The microscope preferably has the features described in connection with the above method and preferred embodiments thereof.

Drawings

Further details and refinements of the invention emerge from the following description of a preferred embodiment of the invention with reference to the figures. Wherein:

FIG. 1: showing one overview image and three grouped images generated according to a preferred embodiment of the method of the present invention; and is

FIG. 2: an image synthesized from the set of images shown in fig. 1 according to a preferred embodiment of the method of the present invention is shown.

Detailed Description

Fig. 1 shows an overview image 01, a first group of images 02, a second group of images 03 and a third group of images 04, which are produced by means of a microscope as an example according to a preferred embodiment of the method according to the invention. The overview image 01 is a microscopic image with a relatively low resolution, but is taken with a large field of view, and thus fully or largely fully images a sample (not shown) to be microscopically observed. After the overview image 01 has been recorded, a plurality of microscope subimages 06, each having a higher resolution than the overview image 01, are recorded in groups. The microscope subimages 06 as a whole image the same field of view as the overview image 01. The first set of microscopic sub-images 06 illustratively comprises 3 by 3 microscopic sub-images 06, which are captured at a first resolution and stitched into a first set of images 02. For this purpose, the overlap region 07 between the microscope image 06 is known, and therefore the position of the microscope images 06 relative to one another. The first group of microscopic sub-images 06 is exemplarily captured as a normal two-dimensional image. The second set of microscopic sub-images 06 exemplarily comprises 4 by 4 microscopic sub-images 06, which are captured at a second resolution different from the first resolution and stitched into a second imaging set 03. For this purpose, the overlap region 07 between these microscope images 06 is determined again. The second group of microscopic sub-images 06 is for example recorded as a normal two-dimensional image. The third set of microscopic sub-images 06 exemplarily comprises 1 by 3 microscopic sub-images 06, which are taken at a third resolution different from the first and second resolution and stitched into a third set of images 04. The overlap region 07 between these sub-images 06 is determined. For example, the third group of microscopic sub-images 06 is captured as a three-dimensional image. The first, second and

third group images

02, 03, 04 are therefore stitched according to known methods and with conventional image quantities, so that the available hardware and software can be used. The hardware and software need not be further developed or extended. Stitching the microscopic sub-images 06 into sets of

images

02, 03, 04 is also referred to as stitching.

When three groups of microscopic sub-images 06 are acquired, their position relative to the overview image 01 is recorded in the form of coordinates. Thus, for each of the three

group images

02, 03, 04, its position relative to the overview image 01 can also be present in the form of coordinates.

Fig. 2 shows a composite image 09, which is synthesized from the

triplet images

02, 03, 04 shown in fig. 1 according to a preferred embodiment of the method according to the invention. For this composition, the coordinates that exist for each of the three sets of

images

02, 03, 04 are used, which describe their positioning relative to the overview image 01 (shown in fig. 1). Since it is not necessary to search for an overlapping area or the like to stitch the three sets of

images

02, 03, 04 into the composite image 09, the stitching can be achieved with little effort. Since the

triplet images

02, 03, 04 differ in their resolution and in their two-dimensional, three-dimensional recording, the composite image 09 has a different resolution and, in addition to the two-dimensional content, also a three-dimensional content.

List of reference numerals

01 overview image

02 first group image

03 second component image

04 third group image

06 microscopic image

07 overlapping area

09 composite image

Claims

1. A method for generating an image (09) of a sample synthesized from a plurality of microscopic sub-images (06); the method comprises the following steps:

-taking an overview image (01) of the sample;

-taking a first set of microscopic sub-images (06) of the sample, wherein the microscopic sub-images (06) of the first set have the same resolution, and wherein the positioning of the first set of microscopic sub-images (06) within the overview image (01) is detected;

-stitching the first set of microscopic sub-images (06) into a first set of images (02);

-taking a second set of microscopic sub-images (06) of the sample, wherein the microscopic sub-images (06) of the second set have the same resolution, and wherein the positioning of the second set of microscopic sub-images (06) within the overview image (01) is detected;

-stitching the second set of microscopic sub-images (06) into a second set of images (03); and is

-stitching at least the first set of images (02) and the second set of images (03) into a composite image (09) of the sample based on the detected locations of the first set of microscopic sub-images (06) and the detected locations of the second set of microscopic sub-images (06).

2. Method according to claim 1, characterized in that it comprises the further step of:

-taking a third set of microscopic sub-images (06) of the sample, wherein the microscopic sub-images (06) of the third set have the same resolution, and wherein the positioning of the third set of microscopic sub-images (06) within the overview image (01) is detected;

-stitching the third set of microscopic sub-images (06) into a third set of images (04);

-stitching the first (02), second (03) and third (04) sets of images into a composite image (09) of the sample based on the detected locations of the first (06), second (06) and third (06) sets of microscope sub-images.

3. The method according to claim 1, characterized in that the same resolution of the microscopic sub-images (06) of the second set differs from the same resolution of the microscopic sub-images (06) of the first set.

4. A method according to any one of claims 1 to 3, wherein the step of stitching the sets of microscopic sub-images (06) into respective sets of images (02; 03; 04) comprises the sub-steps of:

-determining an overlap region (07) between each two of the microscopic sub-images (06); and is

-stitching the microscope sub-images (06) according to the determined overlap region (07).

5. A method according to any one of claims 1 to 3, wherein the step of stitching the sets of microscopic sub-images (06) into respective sets of images (02; 03; 04) comprises the sub-steps of:

-detecting the positioning of the respective microscopic sub-images (06) with respect to the sample, respectively; and is

-stitching the microscopic sub-images (06) according to the detected locations.

6. The method according to any one of claims 1 to 3, characterized in that the microscopic sub-images (06) of the first set have the same image size and the microscopic sub-images (06) of the second set have the same image size, the same image size of the microscopic sub-images of the second set being different from the same image size of the microscopic sub-images (06) of the first set.

7. Method according to one of claims 1 to 3, characterized in that the individual groups of microscopic sub-images (06) are arranged in a matrix, wherein the sub-images (06) arranged in a matrix form a matrix with columns and rows, wherein the columns and rows each have a length between one and ten sub-images (06).

8. The method according to any one of claims 1 to 3, characterized in that the sum of all sets of microscopic sub-images (06) is at least 100.

9. Method according to any one of claims 1 to 3, characterized in that the groups of microscopic sub-images (06) are respectively captured according to a capture mode selected from the group comprising: two-dimensional image capture, three-dimensional image capture, and image capture with extended depth of field; wherein the groups of microscope partial images (06) are recorded according to different recording modes.

10. A microscope for microscopic observation of a sample; the microscope has:

-an objective for magnifying optical imaging of the sample;

-an image sensor for converting an imaged image into an electrical signal; and

-an electronic image processing unit configured for implementing the method according to any one of claims 1 to 9.