WO2019077157A1

WO2019077157A1 - Method for evaluating microscopic samples and device for carrying out said method

Info

Publication number: WO2019077157A1
Application number: PCT/EP2018/078834
Authority: WO
Inventors: Jiri Snaidr; Claudia Beimfohr; Axel Bonsen; Peter MÜHLHAHN
Original assignee: Vermicon Ag
Priority date: 2017-10-20
Filing date: 2018-10-22
Publication date: 2019-04-25
Also published as: DE102017009804A1; EP3698271A1

Abstract

The invention relates to a method for evaluating microscopic samples and to a device for carrying out said method. The method according to the invention comprises the following steps: providing a slide (1) having microscopic samples (7); arranging the slide (1) on a microscope stage (9); imaging, with magnification, a portion (3) of the slide (1) by means of a digital camera (5) and a magnifying optical unit (11), the same portion (3) being imaged multiple times and, during the imaging of successive images, the focal plane of the images being shifted; creating a combined image from the plurality of successive images; and evaluating the combined image in order to evaluate the microscopic samples.

Description

description

The application relates to a method for evaluating microscopic samples and an apparatus for carrying out this method.

To quantify bacteria in a larger volume of liquid, for example 100 ml or more, the liquid to be examined is filtered in microbiological tests. Usually filters with a side length or diameter of about 2.5cm are used. Subsequently, the microorganisms in a known manner by culturing the filter on agar plates and

subsequent optical determination of microcolonies

analyzed.

As a modern method for the determination of microorganisms, preferably bacteria, on filters has now established a specific staining method, the so-called fluorescence in situ hybridization (Fish). Here, the cells are visualized on a carrier / filter by fluorescence-labeled nucleic acid probe molecules under a microscope. This method is in: Amman et. all Microbial. Rev. 59 (1995), pages 143-169.

In other embodiments, specific

Staining methods can be applied to the filter itself, and the fluorescence-labeled microorganisms can be counted by fluorescence microscope.

Since the filter in the surface is much larger than the field of view of the microscope, a complete quantification of the microorganisms on the filter

Variety of fields of view or sections are counted. Depending on the magnification, size of the clipping and size There may be some 10,000 cutouts in the filter that need to be evaluated individually.

With a high concentration of microorganisms, it has therefore been natural to count only a statistically representative number of 10 - 100 segments and to extrapolate the result accordingly to the entire area. However, this approach is inaccurate and fraught with large errors. Also is the manual counting of the

Microorganisms in a section often difficult.

To solve this problem, it is known to scan the filter using automatic microscopes that have a motorized XYZ table and to take a picture of each cutout via a digital camera, ie to take a digital image. This image can then be analyzed with a pattern recognition or object recognition in a known manner. Current embodiments of these automatic microscopes still require more than twelve hours for the evaluation of a carrier / filter of the type mentioned above.

A problem with this method is that the filter is not completely flat. That is, for each

First, the focus plane must be adjusted so that a sharp image is obtained.

This focusing process to a target level in which the microorganisms to be recognized are sharply imaged is achieved in the prior art by using an auxiliary system, for example a separately fed-in

Laster beam, the surface of the filter is detected, and then the Z-motor of the microscope stage is moved so that the filter surface is in focus. Due to the lack of planarity of the filter that preceded has been addressed, this focusing process is repeated for each section. This focusing process is compared to the actual recording of the digital

Figure as well as the method of the table in the XY direction the most time-consuming step of the whole process.

The object of the present invention is to provide a novel method for the evaluation of microscopic samples and a corresponding device, which can reliably and quickly determine the number of microscopic samples to be examined on a support.

This object is achieved by a method according to claim 1 and an apparatus according to claim 10. The dependent claims relate to further advantageous embodiments of the invention.

In the method according to the invention for the evaluation of microscopic samples, a carrier with microscopic samples formed thereon is first provided. This carrier is placed on a microscope stage. With the help of a digital camera and a magnifying optics a section of the carrier is shown.

The method according to the invention is characterized

characterized in that the same cutout multiple times

is photographed, during the imaging of

successive pictures a focal plane of the

Pictures is moved.

From the thus obtained several successive images a combined image is created. This combined image is evaluated to evaluate the microscopic samples. Since according to the invention no focusing step for the

individual images is necessary, the method is determined only by the exposure time of each shot, and thus much faster than the known method.

In a preferred embodiment of the method, the combined image is obtained by, for each pixel of the combined image, determining the maximum value of the positionally corresponding pixels of the plurality of pixels

successive pictures is determined.

It is assumed that a pixel is one

Figure takes the maximum value then, if this

Illustration the focal plane in the plane of the image to be imaged

Object, so the microscopic sample has. In the

Areas of the section where no microscopic sample is present, so where no fluorescence

Although the maximum value of the pixels is also used, this is only determined by the noise, and will clearly differ from a positive result.

In a further embodiment of the invention, the combined imaging by means of automatic pattern recognition, in particular based on a histogram of Oriented

Gradients (HOG) in combination with a learning algorithm evaluated by Support Vector Machine (SVM).

HOG in conjunction with SVM is a well known pattern recognition technique used, for example, for face recognition or handwriting recognition.

Examples of such pattern recognition are in Hamayun A. Khan, "MCS HOG Features and SVM Based Handwritten Digit

Recognition Systems ", Journal of Intelligent Learning Systems and Applications, 2017, 9, pages 21-33, or in Chandrasheka, TR et al.". Face Recognition Based on Histogram of Oriented Gradients, Local Binary Pattern and SVM / HMM

Classifiers ", International Journal of Engineering Sciences 145 and Research Technology, August 2014, pages 344-352,

described.

The combined image obtained according to the invention is not clear to the human eye. However, the pattern recognition described above allows, if appropriate

Parameter sets in a specially prepared

Database, a reliable and rapid detection of microorganisms on the carrier / filter.

155 According to another aspect of the invention

different sections of the carrier are sequentially brought into the field of view of the digital camera and the optics, where for each section several images are created with different focal planes, for each section one

160 combined image is created and evaluated, and based on the evaluation of all sections the

Total microscopic samples on the carrier can be evaluated.

With the method according to the invention, the XYZ table of the microscope can be moved automatically from one cutout to the next until the entire filter / carrier has been scanned. In this way is a fast and

complete evaluation of the filter / carrier possible.

170

According to the invention, the carrier is a filter for

Filtration of a contaminated with microorganisms

Liquid was used and colored. Instead of a filter, it is also possible to scan the surface of a dish filled with agar or other culture medium.

Furthermore, it is preferred that, on average, between each two consecutive images, the distance between the Support and the digital camera by 1 - ΙΟμτη, particularly preferably changed by 2μπι, thereby to the focal plane

change.

Depending on the size of the investigated microorganisms or microcolonies, the choice of the distance should be selected so that there is a sufficiently high probability that each microorganism will actually be present in one of the microorganisms

Illustrations of a stack of consecutive

Focusing on pictures at least once. In addition, the distance should also be sufficiently small, even to a plurality of shots, no significant change in the section due to the magnification or

Reduction of the distance between carrier and digital camera. That is, in the conventional digital mappings where the pixels are arranged in rows and columns, a pixel of a first image located on a particular line and column also corresponds to approximately the same location on the filter as any other image of the same section at changed distance

between carrier and camera.

In this way, it is easily possible to change the focal plane in small steps, with only the microscope stage in the Z direction must be moved.

It is not necessary for the distance to remain constant during the exposure time of an image.

Alternatively, it is of course also possible to adjust the optics so that the focal plane changes, or to move the digital camera, and to hold the carrier stationary.

According to a further advantageous method of

Invention, it is provided initially a Ausgangsfokusebene 215, where distributed over the filter several

Focus values for sharp images of individual sections are determined, and wherein a Ausgangsfokusbene for the successive images of the remaining sections based on the average of the multiple focus values, the

220 Extent of change of focal plane between

successive pictures and the number of

successive recordings for a combined recording is calculated.

225 For example, if 16 illustrations of a section with

2μιτι be made a distance of the focal plane, so it makes sense to set the first focal plane offset by 8μη behind the determined by the averaging of the focus values level, and then in 2pm steps

230 the distance between the digital camera and the carrier

enlarge. In this case, pictures are included

Focus planes in a range of plus minus 8μπι to obtain the above-determined average.

Next, it's beneficial for every clipping

between 5 and 50, preferably between 10 and 30

more preferably, 16 consecutive images are generated, and a combined image

are processed.

240

It has been shown that a range of 32μπι

sufficient to obtain a useful combined image in conventional filters of the type described above.

Furthermore, it is preferred that the focal plane

continuously or gradually change. Whether a continuous or stepwise change of the focus plane is selected depends on the exposure time on the one hand and above all on the adjustment speed of the focal plane. As 250 it has proven advantageous to the focal plane in stages too to wait, and then to wait

Focus plane to move to another level. This can be done by synchronizing the exposure time of the camera and the means for adjusting the focal plane. Alternatively, it is also possible to have only one

Time control of the camera and the microscope stage

Make sure to get a majority of images of the same section with different focal planes. This is particularly advantageous when a complicated structure and a complicated control

should be avoided.

The inventive device for carrying out the above-mentioned method comprises a microscope stage for

Arranging a vehicle, a digital camera and a

magnifying optics for imaging an enlargement of a section of the carrier, means for varying a

Focus plane of the digital camera and the magnifying optics with respect to the carrier, a controller for controlling the

Picture by the digital camera and the change of the

Focus plane, so as to produce a plurality of images of a section of the carrier with different focal planes, and an evaluation unit which is designed to produce a combined image of the plurality of successive images and to evaluate the combined image for the determination of the microscopic samples.

In the following, the invention is based on preferred embodiments and with reference to the accompanying figures

described.

Fig. 1: is a schematic view of the invention

Device;

Fig. 2 is a schematic view of the carrier with the

Section and the samples to be evaluated according to the invention; 3 is a flowchart for explaining the

Process of the invention.

The invention is essentially based on that instead

290 for each field of view / excerpt of the wearer to try to take a sharp focused image, take a fast sequence of images while simultaneously moving the microscope stage 9 in Figure 1 in the Z direction. Within this recorded picture sequence

295 (for example, 16 images with a distance in the Z direction of

2μιη) there are several illustrations in which the

Microorganisms / cells are in focus. Through a clearing process, an overall image is calculated from this image stack, which contains all the information of the

Contains 300 consecutively recorded images, and thus

of course, the information of fluorescently labeled cells.

The appearance of the cells is different

By nature, of course, as you focussed on one

Figure of the cells would be found. Since a learning algorithm is used for object recognition, the cell can still be recognized as a cell, since the recognition algorithm is trained on the representation in the overall picture, 310 and does not focus on a focused image of the cell.

By this method it is possible to one

to refrain from automatic focusing, even simple microscopes without mechanical and electronic focusing

315 in a shortest possible time of less than two hours to measure a carrier of the type mentioned and evaluate. By dispensing with a technically complex and expensive autofocusing cost-effective fluoroscopic microscopes for an automatic

320 evaluation to be upgraded. As stated above, an image stack becomes off

Frames focused in different planes, at any position on a filter / carrier

325 recorded. The pictures of a picture taken in this way

Image stacks differ only from the focal plane in the Z direction, with the X or Y direction not being changed, i. the images of a stack belong to a fixed XY coordinate on the filter / carrier. The size of the stack

330 is chosen so that the plane, the sharp

Figure shows the cells over the entire filter / support always located inside the pictures of the stack. The average difference in Z direction over one

typical filter / carrier moves in the range of 20μιτι, 335 wherein preferably a cover of 32μπι is set.

According to the invention, the movement of the

Microscopic 9 in Z-direction of the pure

Exposure time of the picture decoupled. That is, the

340 movement of the microscope stage 9 not with the recording of individual images by the digital camera. 5

is synchronized. For this, the digital camera 5

be parameterized and provide continuous images, and on the other hand at the same time the microscope stage 9

345 be moved accordingly.

Due to the additional decoupling of the image and the subsequent processing of the images of the stack to a combined image results in the

350 scanning speed for the filter / carrier essentially from the values of the exposure time, the number of

images to be taken and the distance between the images in the Z direction and the time required for moving the microscope stage 9.

The time required to take an image batch then becomes: T-stack = T-direction * Π, where n is the number of images of the stack

In a preferred embodiment of the method, 16 images are taken, the exposure time being determined as a function of the signal intensity, the sensitivity of the camera and the aperture setting.

The distance relative to the Z-direction of the images within the stack and thus the total travel during the Z-movement is calculated from:

Stack = formation

Where Sstapei is the total travel in the Z direction and the formation is the travel path of a single image.

The resulting speed is in

known manner: V = stack / stack

In practice, the resulting velocity V for commercial Z motors of microscope stage 9 is too low, i. the traversing speed must be through

additional waiting times are artificially throttled to meet the requirements of the exposure time of the camera. The method of the microscope stage 9 in the Z direction can be replaced by a clocked operation for this purpose. For this purpose, the microscope stage during the creation of the

Stacks of images do not proceed continuously at constant speed in the Z direction, but the

Microscope table 9 is moved in stages with intermediate waiting times. In this case, the microscope stage 9 is first only by the desired distance of the focal planes of two 395 consecutive images. Subsequently, a defined time is waited and then renewed the engine to process this pause, while the camera 5

continuously created a picture after another. The waiting time between two traversing commands is calculated from:

400

Twarten = TBelight * nNumber of images / HlNumber of steps ·

With mAnza i of the steps is the number of desired ones

Driving interruptions of the engine meant. This may differ from

405 the number of figures differ. In particular, the number of steps may be greater than the number of mappings of a stack. For example, it is possible first to start the exposure of an image while the microscope stage 9 is stationary, then to the microscope stage 9

410 proceed while the exposure of the same figure

is continued through the Kamrea 5. Then, pause the procedure again while still exposing the same image through the camera. After the end of the imaging time, the motor can be moved further. In that

In a case, two or more focal planes are in the image

Superimposed on a "multiple exposure".

The step size which the microscope stage 9 travels in the Z direction during the exposure of an image may thereby

420 not larger than the focal range of a cell level or

not larger than the cell size itself. Otherwise, there is a possibility that a cell may not be "focused" in any of the images, and it may be lost to detection.

425 distance between two images is according to the invention 1 to

ΙΟμη particularly preferably 2μπι. The entire travel time can be calculated by including the manufacturer-specific engine speed, for example, then calculated from:

TFtime per step = formation / VMotor ·

It is assumed that no significant time is needed to accelerate and decelerate the engine.

Since the creation of images by the camera 5 is not synchronized with the movement of the microscope stage 9 in the Z direction, additional deviations occur in practice, which can be corrected. The

Correction factor is:

K = total number of images / number of images already taken

To compensate for the additional deviations, the waiting time is dynamically adjusted using the correction factor during the current sampling:

Twarten corrected ⁼ Twarten * K

According to the invention, it is also useful a starting position of the focal plane for creating the

To determine pictures.

At the beginning of automated capture of a

Filter / carrier 3 is therefore according to the invention in a first step, the start position of the focal plane of the stack set so that the cells are at least in a range that is covered by the process of the microscope stage 9 in the Z direction. For this purpose, in the simplest embodiment, three optional points on the edge of the filter are manually optimally focused and thus yielded Mean value for the focal plane (Z value) as starting point

465 used.

In another embodiment, multiple points on the filter are manually optimally focused and then used to calculate a topology level of the filter. Based on this topology level, the start point in the Z direction for the image stack is dynamically corrected based on the XY coordinates of the selected section on the filter / carrier.

475 The position for starting the image stack then results, for example, from:

P image stack = average value ^~ ((images * nnumber of steps) / 2)

480 In one example of the invention became a Leica

Fluorescence microscope DMRB with a 40x objective and a 14 megapixel color camera (Basler acA4600-10uc USB3 camera) used. The microscope was extended by an automatic XYZ table OptiScanll ^® from Prior Scientific, Inc..

485

With an average exposure time of the camera 5 of 100 milliseconds and a distance of the selected 20 images with each other of 2μιη at 16 steps, the following values result:

490

Total for a batch: 20 * 100ms = 2000ms

Total travel for the stack: 2μπι * 16 = 32pm

Waiting time: 100ms * 20/16 = 125ms

495 In this example, the number of images is

greater than the number of traversing steps. Even if two images should have the same focus plane, this does not matter in the proposed evaluation, as only the maximum value of one pixel under the images of the Batch is considered when creating the combined image.

So it is the motor drive command for 2i to the

Motor control sent and at the same time the timer for the waiting time started. The engine stops automatically after 2 μιη and gets a drive command again when the timer for the 125ms has expired. It is important that the engine takes no longer than 125ms for the track, but in the

Reality is certainly the case.

In the following case the motor speed is 200ym per second

Then:

Traditional time per step = 2 μΐΏ / 2 0 0 μΐΐΙ / S = 0.01s = 10ms

In this example, recording takes one

Picture stack 2 seconds. Within this time, the camera takes 20 pictures at an exposure time of 100ms. At the magnification used of 40, there is a sampling time of 1 hour 54 minutes for the entire filter / carrier 1. The total sampling time is additionally extended by the much lower proportion for the process of

Microscope 9 in XY direction to a maximum of 2 hours.

As shown at the beginning, a combined image containing the information of all the individual images of the stack is calculated on the individual images of an image stack. For this purpose, the maximum pixel value (PW) over all images of the stack is determined at each position and entered at the same pixel position in the combined image. This is done for each pixel of the images of the stack. All pictures of the stack have the same pixel number given by the camera in the XY direction.

PW (X, Y) Combined mapping = PW (X, Y) Max of all frames. In this way, there is one for the human

Eye blurred image or image, since the pixel values, preferably brightness values, out of focus

Images / images - including motion blur - lead to corresponding pixels in the combined image. An evaluation of these combined mappings can, however, take place in that an adaptive

For this purpose, in the illustrated method, a combination of the

Object recognition based on HOG (Histogram of Oriented

Gradients) along with learning algorithm SVM (Support Vector Machine) trained on these imaging objects, so that these representatives are recognized for a fluorescently labeled cell. Details of the pattern recognition are not further elaborated here, but they are those skilled in the art from the above-mentioned documents for other purposes,

For example, handwriting recognition or face recognition, known. In Fig. 1 it can be seen how the device is constructed in principle from the microscope stage 9, the attached filter or carrier 1, the optics 11 and the digital camera 5. The microscope optics 11 and the camera 5 are here only

shown schematically as a single lens and simple box. However, those skilled in the art are well-versed in the design of these components.

FIG. 2 shows a plan view of the carrier 1 with a cutout 3 marked therein, on which the microorganisms 7 are arranged. As described at the beginning, the method according to the invention requires that first, after the

Filter / carrier 1 arranged on the microscope stage 9 and a section has been selected, a Ausgangsfokusebene

is set.

Subsequently, a section 7 of the filter / carrier 1 is imaged with the digital camera 5 and this image is stored. In a next step - but preferably simultaneously with the exposure of the image - the focal plane is changed, for example, in the

Microscope table 9 is moved in the Z direction by 2μπι. The procedure then checked whether the maximum change in the focus plane has already been reached or whether the desired number of images of the stack has been created. If this is not the case, another image of the same section is produced and stored. The formation of an image and the method of the focal plane can

done concurrently or concerted, so the procedure in the breaks between two figures

he follows. However, this is not preferred because it can extend the time for creating the image stack and since a synchronization between microscope stage 9 and camera 5 must be established.

Once the maximum change is the

Focus plane / maximum number of images of the stack reached becomes one of the stored maps

Combined image created by, as described, for the value of a pixel of the combined image of the

Maximum value of the corresponding pixels of the individual

Pictures is chosen. On the combined image thus obtained, a pattern recognition method is applied and the number of

Microorganisms / cells determined. In a next step, the microscope stage 9 in

XY direction for setting a new section

method. It is conceivable, even during this movement in the XY direction, to move the microscope stage 9 also in the Z direction to the new output focal plane. Alternatively, the Z-direction of motion can be reversed when creating successive image stacks, so that the

Focus plane of the last image of a first stack is the output focus plane of the next stack. It is checked whether all sections of the filter have already been scanned. If this is the case, it ends

Device the procedure. If this is not the case, the output focus level is set again and a new one

Pile of illustrations created.

Although the invention has been described above with reference to a preferred embodiment, is the

Invention not limited thereto. Various modifications and modifications can be made which are familiar to those skilled in the art.

The pattern recognition described here is only one

Example. Other pattern recognition techniques will be apparent to those skilled in the art and may be used.

With the invention it is possible to use the usual

To reduce processing times to automatically scan a filter from 12 hours to approximately 2 hours. This considerably increases the throughput and the productivity in the evaluation of such filters and lowers the costs. The invention has been described with reference to the scanning of a filter. It is not limited to this. It is also possible to see the surface of a shell,

for example, a petri dish with agar medium,

examine microscopically.

Fluorescence was used in the invention. Again, this is not mandatory if other optical possibilities for detecting microorganisms or microcolonies are given, for example in transmitted light.

In the example described, the brightness value was selected as the pixel value. However, it is also possible the

Brightness value of a particular color to use to achieve better selectivity.

Claims

claims

1. Method for evaluating microscopic samples, comprising the steps of:

Providing a support (1) with microscopic samples (7);

Arranging the carrier (1) on a microscope stage (9);

Imaging enlarging a section (3) of the carrier (1) by means of a digital camera (5) and a magnifying optical system (11);

characterized in that

the same detail (3) is mapped multiple times, during the mapping of successive ones

Pictures a focal plane of the pictures is moved;

Creating a combined image from the plurality of consecutive mappings; and

Evaluate the combined image for evaluation of the microscopic samples.

2. Method according to claim 1, characterized in that the combined image is obtained by determining, for each pixel of the combined image, the maximum value of the positionally corresponding pixels of the plurality of successive images.

3. The method according to claim 1 or 2, characterized in that

the combined picture by means of automatic

Pattern recognition, in particular based on a histogram of orientated gradient (HOG) in combination with a

Learning algorithm using Support Vector Machine (SVM)

is evaluated.

4. The method according to any one of claims 1 to 3, characterized in that

sequentially different sections of the carrier (1) in a field of view of the digital camera and the optics (11) are brought, wherein for each section a plurality

Illustrations with different focal planes are created, for each section a combined image is created and evaluated, and based on the evaluation of all sections, the microscopic samples are evaluated.

5. The method according to any one of claims 1 to 3, characterized in that

the carrier (1) is a filter which has been used and filtered to filter a microorganism-contaminated liquid.

6. The method according to claim 5, characterized in that between two consecutive images of the distance between the carrier (1) and the digital camera by 1 - 10 μτη, preferably by 2μπι is changed so as to change the focal plane.

7. The method according to claim 5 or 6, characterized in that

for determining an output focal plane distributed over the filter (1) a plurality of focus values for the sharp imaging of individual sections are determined, and wherein the

Initial focus plane for the successive mappings of the remaining sections based on the mean of the multiple focus values, the extent of the change in the

Focus plane between successive pictures and the number of consecutive pictures for one

combined intake is calculated.

8. The method according to any one of claims 1 to 7, characterized in that for each section 5 to 50, preferably 10 to 30, particularly preferably 16 consecutive pictures are processed into a combined image.

9. The method according to any one of claims 1 to 8, characterized in that

the focal plane is changed step by step.

10. A device for carrying out a method according to one of claims 1 to 9, characterized in that the

Device comprises:

a microscope stage (9) for arranging a carrier (1); a digital camera (5) and a magnifying optical system for imaging enlarging a section (3) of the carrier (1),

Means for changing a focal plane of the digital camera (5) and the magnifying optic with respect to the carrier (1); a controller (11) for controlling the image by the digital camera (5) and changing the focal planes so as to produce a plurality of images of a section of the carrier (1) having different focal planes; and

an evaluation unit configured to generate a combined image from the plurality of successive images and to evaluate the combined image to determine the microscopic samples.