KR20230103394A - Slide imaging device including focus restoration function and method therefor - Google Patents

Abstract

The present invention provides a slide imaging device including a focus restoration function and a method therefor. The slide imaging method comprises the steps of: moving an XY stage to an initial position; photographing a slide or sample; restoring an out-of-focus area of the photographed image using a focus restoration network; moving the XY stage to a next position and determining whether slide photographing ends; returning to the step of photographing the slide or sample if the slide photographing has not ended; and matching the images to generate an entire slide image when the slide photographing is completed. Thus, the present invention can resolve problems related to out-of-focus images without Z-axis scanning to reduce scan time and storage space.

Description

Slide imaging device including focus restoration function and method therefor

The present invention relates to a method for resolving an out-of-focus image without Z-axis scanning in order to reduce scan time and storage space.

Pathology is a field that identifies the cause of a disease by examining cells or tissues in a specific area. In general, the collected cells or tissues are applied to a pathology slide and the pathology slide is closely examined under an optical microscope to determine the cause of the disease. .

Since an optical microscope uses a 10x or higher magnification objective lens, cells or tissues are enlarged and displayed, and users (doctors, pathologists, etc.) observe the enlarged cells or tissues through the eyepiece and diagnose the presence or absence of disease by inspecting them closely. can do. Optical microscopy enables precise examination of cells or tissues, but there is a limitation in that only a part of cells or tissues can be observed due to a field of view (FOV) problem of a high-magnification objective lens.

Engineering microscopes used to examine cells or tissues provide an XY stage and a stage knob that can move the pathology slide in four directions (east, west, north, south) to solve the problem of the viewing angle, but pathology is manually performed through the knob. Since the entire area of the slide must be observed, lesions may be missed. In order to reduce misdiagnosis caused by manually observing the entire area, various products for a whole slide imaging (WSI) device that automatically scans the entire area of a slide have recently been introduced.

The entire slide imaging device automatically scans the entire slide area with an XY stage, stitches the scanned images, converts them into a super high resolution/high resolution digital image, and then provides the image to the user. Whole-slide imaging devices provide super-resolution digital images of the entire area of a pathology slide, making them one of the key technologies for converting the field of pathology from analog to digital. However, samples with thick cells or tissues have an out-of-focus area that is blurry, and since cells or tissues in the area cannot be intuitively identified, a problem in that the lesion cannot be properly screened occurs. This is a problem that occurs because the depth of field (DoF) of high-magnification objective lenses is very narrow on the order of micrometers (μm), and if the thickness of a cell or tissue exceeds the depth of focus, out-of-focus (OOF) : An out-of-focus area is captured.

General optical microscopes provide a focusing knob to manually adjust the focus in preparation for cases where the thickness of a cell or tissue is outside the depth of focus (out of focus). Since the entire slide imaging device automatically scans the entire area, a function capable of automatically controlling the focus is required.

Some commercially available whole-slide imaging devices provide a “Z-Stack” function to solve the depth-of-focus problem. “Z-Stack” is a function that provides the user with N total slide images taken while moving a specific focus range at equal intervals using Z-axis scanning, and provides a 3-dimensional image by stacking N images in the Z-axis. Save the entire slide image in the form of a volume image. The entire slide image in the form of a 3D volume can be checked through the image viewer software provided by the product. In particular, since the 3D volume image includes images taken at different focal positions, it is similar to an optical microscope. can be changed.

The Z-Stack function is effective in resolving the focus problem (out-of-focus area), but compared to products that do not provide the Z-Stack function, the scan time and file capacity ( storage space) is increasing. An increase in scan time results in a decrease in productivity because the number of slides that can be scanned per hour decreases, and an increase in file size results in an increase in data maintenance costs because the data space occupied per slide increases.

In conclusion, we need a way to solve the out-focus problem, scan time and storage space problems at the same time.

The present invention relates to a method for solving out-of-focus images without Z-axis scanning in order to reduce scanning time and storage space, and constructs training data consisting of pairs of in-focus images and out-of-focus images for various slides. After learning a focus restoration network that restores an out-focus image to an in-focus image based on the learned training data, the out-focus area of the image captured in the process of scanning the entire slide area is converted into an in-focus area through the learned network. Provides a way to restore it.

In order to solve the problems of the prior art described above, the slide imaging method according to an embodiment of the present invention includes moving the XY stage to an initial position; photographing slides or samples; Restoring an out-of-focus area of the photographed image using a focus restoration network; moving the XY stage to the next position and determining whether slide shooting is to end; returning to the step of photographing the slide or sample if the photographing of the slide has not ended; and generating an entire slide image by matching the images when the slide shooting is finished.

The focus restoration network includes an input layer inputting an out-focus image, an output layer outputting an in-focus image, a restoration block calculating restoration information, and a summing unit summing the output value of the restoration block and the input value of the input layer. can include

The reconstruction block is composed of a plurality of sub-blocks, each of which includes a first convolution operation unit, a first batch normalization operation unit, a Rectified Linear Unit (ReLU) operation unit, a second convolution operation unit, It may include a second batch normalization calculator and a summing unit that sums the resultant value of the second batch normalization calculator and the input value of the corresponding sub-block in units of pixels.

The focus restoration network may undergo a network training process of adjusting weights of the first and second convolution operators through training data.

The training data consists of a pair of an out-focus image and an in-focus image taken at the same location as the out-focus image, and the input value or input image of the training data is the out-focus image, The output value or output image of the training data may be the in-focus image.

The in-focus image may be obtained by generating an all-in-focus image including all focal areas from N focal plane images captured by sequentially moving a focal position through a Z-axis scanner.

The slide imaging method distinguishes whether an output image of the focus restoration network is an output image of the training data or an output image generated by the focus restoration network through a discrimination network after restoring an out-of-focus region of the captured image. It may further include the step of outputting a value that is.

A slide imaging device according to an embodiment of the present invention includes an objective lens; an XY stage driven to linearly move the slide in the X and Y axes; a Z-axis scanner driven to linearly move the objective lens in the Z-axis; a memory storing one or more routines for performing a slide imaging method; and a processing component configured to execute one or more routines stored in the memory, wherein the one or more routines, when executed by the processing component, include moving the XY stage to an initial position, photographing a slide or sample. An operation of restoring an out-of-focus area of the photographed image using a focus restoration network, an operation of moving the XY stage to the next position and determining whether slide shooting is to end, if the slide shooting has not ended, the slide Alternatively, an operation of returning to the step of photographing the sample and, when the photographing of the slide is finished, an operation of generating an entire slide image by matching images may be performed.

According to the present invention, it is possible to provide a slide imaging device and method capable of taking a high-resolution image without out-of-focus using only a single scan without using a Z-axis scanner. Since the slide imaging device and method according to the present invention do not perform Z-axis scanning as well as a focus problem, there is an effect of reducing scan time and storage space. That is, it is possible to solve the out-of-focus problem, which is an existing problem, while reducing scan time (productivity improvement) and storage space (data maintenance cost reduction).

1 is a diagram showing the basic configuration of an entire slide imaging device including selective Z-axis scanning according to the present invention.
2 is a diagram illustrating an image conversion network according to an embodiment of the present invention.
3 is a diagram showing an example of the configuration of learning data according to the present invention.
4 shows a schematic drawing of a discriminating network according to one embodiment of the present invention.
Fig. 5 shows the overall learning flow chart in which a discriminant network is added to a focus recovery network.
6 shows a flowchart of a slide image generating method according to an embodiment of the present invention.
7 shows an image resulting from restoring an out-focus image using the slide image generating method of the present invention.

The following merely illustrates the principles of the present invention. Therefore, those skilled in the art will be able to invent various devices that embody the principles of the present invention and fall within the spirit and scope of the present invention, even if not explicitly described or shown herein.

In addition, all conditional terms and embodiments listed in this specification are, in principle, expressly intended only for the purpose of understanding the concept of the present invention, and should be understood not to be limited to such specifically listed embodiments and conditions. do.

In describing the present invention, terms such as first and second may be used to describe various components, but the components may not be limited by the terms.

For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

When a component is referred to as being connected or connected to another component, it may be directly connected or connected to the other component, but it may be understood that another component may exist in the middle. .

Terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention, and singular expressions may include plural expressions unless the context clearly dictates otherwise.

In this specification, the terms include or include are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features or numbers, It can be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Hereinafter, a preferred embodiment of the coin earning system according to the present invention will be described based on the accompanying drawings.

1 is a diagram showing the basic configuration of an entire slide imaging device including selective Z-axis scanning according to the present invention.

The slide imaging device 100 may generate one or more images of a sample (cell or tissue) disposed on a slide.

Referring to FIG. 1 , a slide imaging device 100 includes an objective lens 110 , an XY stage 120 , and a Z-axis scanner 130 . In addition, the slide imaging apparatus 100 may include a memory for storing one or more routines for performing the slide imaging method of the present invention and a processing component configured to execute the one or more routines stored in the memory.

An objective lens (110) is used to photograph samples such as tissues or cells with high resolution in micrometer (μm) units. Objective lens 110 may vary in magnification based on considerations such as application and size of imaged sample features. In one embodiment, objective lens 110 may be a high magnification objective lens that provides a magnification of 20x or greater and has a numerical aperture of 0.5 or greater than 0.5 (small focal depth). As can be appreciated, in other embodiments, objective lens 110 may provide different degrees of magnification and/or may have a larger or smaller numerical aperture. By way of example, in one embodiment, objective lens 110 may be spaced from slide 140 in the Z-direction by a distance ranging from about 200 microns to about a few millimeters and has a field of view of 750 μ×750 μ at the focal plane. Light can be condensed from

The XY stage 120 can be driven to linearly move the slide in the X and Y axes. The Z-axis scanner 130 may be driven to linearly move the objective lens 110 in the Z-axis.

In certain embodiments, the sample may be a biological sample, such as a tissue sample for analysis using pathology or histology techniques. In other cases, the sample may be an industrial item such as an integrated circuit chip or microelectromechanical system (MEMS). As an example, such samples may have a thickness averaging from about 5 microns to about 7 microns, and may vary by several microns. An example of such a sample may also have a lateral surface area of approximately 15 mm by 15 mm.

In addition, although not shown in the slide imaging device 100, an image sensor or camera for generating an image of a sample placed on a slide and a position controller or driver for adjusting the positions of the Z-axis scanner 130 and the XY stage 140 can include

An image sensor or camera creates one or more images of the sample corresponding to each field of view as images are acquired. In certain embodiments, the image sensor may be any suitable digital imaging device, such as a commercially available charge coupled device (CCD).

The position controller may be implemented with a piezoelectric actuator, to allow sequential acquisition of multiple images with respect to the sample. The position controller has fine motor control and can provide quick small field of view adjustments to the objective 110 and/or adjust the position of the slide or the XY stage on which the slide is positioned. In addition, the position controller may adjust the position of the objective lens 110 in the Z axis.

In one embodiment, objective lens 110 may be spaced from the sample in the Z-direction by a distance ranging from about 200 microns to about a few millimeters. As can be appreciated, depending on the application, the working distance, field of view, and focal plane may vary depending on the configuration of the slide imaging device 100 and/or characteristics of the imaged sample.

A sample may be placed between a cover slip and a slide. Samples, cover slips, and slides are placed on the XY stage 120. Cover slips and slides may be made of a transparent material such as glass. In certain embodiments, slide imaging device 100 may be part of an automated slide scanning system or may include an automated slide transporter capable of transporting and loading slides for imaging.

Meanwhile, the slide imaging device 100 may be configured as a high-speed imaging device. Such high-speed imaging devices may be configured to rapidly capture a larger number of digital images of the sample, each image corresponding to a specific field of view of the sample. In certain applications, a particular field of view associated with an image may represent only a limited fraction of the total sample. Also, each field of view associated with a sequence of images may be adjacent to each other or may overlap each other. In an example of this embodiment, slide 28 is repeatedly imaged in adjacent or overlapping regions or passed in a scanning sweep through the image acquisition region, i.e., the field of view. In this embodiment, an image is acquired, the XY stage 120 is advanced in the X and Y directions to a position where adjacent or overlapping regions are moved into the field of view, and another image can be acquired.

Specifically, the XY stage 120 is used to extend the viewing angle, which is one of the limitations of the objective lens, and moves the pathology slide coated with cells or tissues in the X or Y axis to automatically scan the entire area of the slide. play a role The Z-axis scanner 130 is used to expand the focal range, which is the limit of the high-magnification objective lens, and serves to move the focal position of the objective lens.

The slide imaging device 100 can automatically take a high-resolution digital image of the entire slide area 140, but since the depth of field (DoF) of the high magnification objective lens is very narrow, the thickness of the sample is thicker than the depth of focus. An out-of-focus image is taken in the area.

The present invention proposes a method that does not use a Z-axis scanner, which causes scan time and storage space problems of the existing Z-Stack technique, and more specifically, an image conversion network that restores an out-of-focus area to an in-focus area. (network) based focus restoration method is provided.

2 is a diagram illustrating an image conversion network according to an embodiment of the present invention.

According to the present invention, an image conversion network used to restore an out-focus area to an in-focus area is shown in FIG. 2 .

The relationship between the out-focus image and the in-focus image can be expressed as Equation 1. That is, Equation 1 shows the relationship between the out-focus image X and the in-focus image Y.

Here, F(X) represents the difference between the out-focus image X and the in-focus image Y, that is, a value (reconstruction information) required for restoring from out-focus to in-focus. Referring to FIG. 2, the focus restoration network 200 proposed in the present invention constructed based on Equation 1 includes an input layer 210 where an out-focus image X is input and an output layer where an in-focus image Y is output ( 220), a restoration block 230 that calculates restoration information, and a summing unit 260 that adds the output value of the restoration block 230 and the input value of the input layer 210.

The restoration block 230 is composed of a plurality of sub-blocks 240, and each sub-block 240 has five operation units and a summing unit for summing the input value of the sub-block 240 and the output value of the last operation unit among the five operation units. include The five operation units consist of a first convolution (Conv) operation unit, a first batch normalization (BN) operation unit, a Rectified Linear Unit (ReLU) operation unit, a second convolution operation unit, and a second batch normalization operation unit. do. The summing unit transmits a result value obtained by summing the result value of the second batch normalization operation unit and the input value of the corresponding sub block in units of pixels to the next block.

The output result after sequentially calculating N blocks performs pixel summing with the image input through the input layer, which represents X+F(X) in Equation 1.

When an image including the out-focus areas 210 and 270 is passed through the focus restoration network input layer, the result image restored to the in-focus areas 220 and 280 to the output layer after sequentially calculating the operation of each block It is the purpose of the focus recovery network proposed in the present invention to output . In order to output the in-focus images 220 and 280 for the out-focus 210 and 270 through the output layer, network training is performed to adjust the weights of the first and second convolution operations through training data. ) process should be performed.

In order to proceed with network learning, first, training data must be collected, and training data is composed of an in-focus image pair taken at the same location as an out-of-focus image. The Z-axis scanner 130 is used to acquire an in-focus image, which is an output value of learning data, and the present invention uses the Z-axis scanner 130 only for the purpose of collecting learning data.

First, N focal plane images taken by sequentially moving the focal position with a Z-axis scanner are taken, and then an all-in-focus image including all focal areas is created from the N focal images. . An image fusion method is used as an embodiment for multifocal generation, and various image fusion methods have recently been introduced. For example, [Y. Liu et al., “Multi-focus Image Fusion: A Survey of the State of the Art,” Information Fusion, 64, 71-91, 2020]. The present invention may utilize one of the conventional fusion methods.

Since the generated multifocal image includes all focal regions, it is used as an in-focus image, which is an output value of training data. In addition, N focal plane images taken to create one multi-focal image are not all images but include an out-focus area, so the input value of training data is used as an out-focus image.

3 is a diagram showing an example of the configuration of learning data according to the present invention.

Referring to FIG. 3 , an example of a pair of training data input 310 and output 320 generated from three focal plane images is shown. As described above, one multifocal image is generated from a plurality of focal plane images. A multifocal image becomes one in-focus image, and a plurality of focal plane images become out-of-focus images. Three pairs of training data can be constructed by matching one in-focus image and three out-of-focus images.

After collecting sufficient training data for various slide samples, focus restoration network training is performed. Learning is a process of fine-tuning network weights so that an output layer can output an in-focus image for an input image when an input image (out-focus image) of training data is passed to the input layer. In general, training calculates the difference between the output image of the network and the output image of the training data (e.g. pixel difference average), and then fine-tunes the weight of each block in reverse order based on the calculated difference.

When learning is performed with only the difference between the two images (the output image of the network and the output image of the training data), an out-focus image tends to be output as a whole when the out-focus image is restored through the focus restoration network that has been trained.

Since the object of the present invention is to restore a blurry out-focus area to a sharp in-focus area, it is necessary to solve the problem that the network output image is blurred as a whole, and for this purpose, an additional network is introduced in the learning process.

The additionally introduced network may serve to determine whether the image is an output image of the focus restoration network or an output image of training data. The purpose of the additionally introduced network is to improve the performance of the focus recovery network so that it can output an image that is indistinguishable from the training data output image. Since the role of the additionally introduced network is discrimination, it is called a discrimination network in the present invention.

4 shows a schematic drawing of a discriminating network according to an embodiment of the present invention.

Referring to FIG. 4 , the discrimination network 400 has a structure similar to a convolutional neural network (CNN) used for image classification. The discrimination network 400 largely consists of an input layer 410, an output layer 420, and N convolution blocks 430.

Each convolution block 440 is composed of a convolution operation (Conv), a batch normalization (BN) operation, and a Rectified Linear Unit (ReLU) operation. The discrimination network 400 receives the output image 411 of the restoration network or the output image 412 of the training data through the input layer, and determines whether the input image is the output image 411 of the training data or the output generated by the restoration network. A value 421 for distinguishing whether it is an image is output through the output layer 420 .

A value output through the output layer 420 is an NxM matrix, and each element of the matrix outputs a value indicating whether a partial region of the input image is an output image of training data. Referring to FIG. 4 as an example, the output layer 420 outputs a 2x2 size matrix, and when the image input to the input layer 410 is divided into 4 parts, each of the upper left, upper right, lower left, and lower right regions is a real image. Outputs the result of determining whether the image is output through the cognitive network.

Fig. 5 shows the overall learning flow chart in which a discriminant network is added to a focus restoration network.

Referring to FIG. 5 , the focus recovery network 510 proceeds with learning to output an image that the discrimination network 520 cannot distinguish, and the discrimination network 520 outputs an image of the focus recovery network 610. Proceed with learning so that it can be determined that is fake. Through this competitive learning, the focus restoration network outputs a non-blurred image similar to the output image of the training data.

The discrimination network is used only in the learning process, and in the scanning process, the input image is restored to an in-focus image using only the focus recovery network 650.

6 shows a flowchart of a slide image generating method according to an embodiment of the present invention.

In order to photograph the sample placed on the slide or the entire area of the slide, in step 610, the XY stage 140 is moved so that the initial position of the sample or slide can be photographed through the objective lens 110. An example of an initial location could be the top left or top right of the slide.

Next, in step 620, the sample or slide is photographed at the initial location. Then, in step 630, focus restoration is performed on the corresponding image. That is, the out-focus area of the input image, that is, the captured image, is restored using the focus restoration network of the present invention. Focus restoration of an image may be performed through a focus restoration network 200 as shown in FIG. 2 . That is, an out-focus image may be restored to an in-focus image through the focus restoration network 200 .

Subsequently, in step 640, the XY stage 140 is moved so that the next position of the sample or slide can be photographed through the objective lens 110. Accordingly, the slide on the XY stage 140 moves, so that the next position of the sample can be photographed.

In step 645, it is determined whether the photographing of the sample is finished, and if the photographing is not finished, the process returns to step 620 to capture an image of the sample.

In this way, the movement of the XY stage 140 (step 640) and the photographing of the slide (step 620) are repeatedly performed until the photographing ends so that the entire slide area can be photographed. If the photographing of the entire area of the slide is completed, step 220 is performed.

When the photographing of the entire area of the slide is completed, the images are matched in step 650, and accordingly, the entire slide image can be generated in step 660.

7 shows an image 720 resulting from restoring an out-focus image 710 using the slide image generating method of the present invention.

Referring to FIG. 7 , an out-of-focus image 710 may be restored to an in-focus image 720 through a focus restoration network according to the present invention.

As such, the present invention discloses a slide imaging device capable of taking out-of-focus high-resolution images with only a single scan without using a Z-axis scanner. Since the slide imaging device according to the present invention does not perform Z-axis scanning as well as a focus problem, it has an effect of reducing scan time and storage space. That is, it is possible to solve the out-of-focus problem, which is an existing problem, while reducing scan time (productivity improvement) and storage space (data maintenance cost reduction).

Those skilled in the art to which the present invention pertains will be able to understand that the present invention may be embodied in other specific forms without changing its technical spirit or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: slide imaging device
110: objective lens
120: XY stage
130: Z-axis scanner

Claims (8)

Steps to move the XY stage to the initial position
photographing slides or samples;
Restoring an out-of-focus area of the photographed image using a focus restoration network;
moving the XY stage to the next position and determining whether slide shooting is to end;
returning to the step of photographing the slide or sample if the photographing of the slide has not ended;
And if the photographing of the slide is finished, generating an entire slide image by matching the images. According to claim 1,
The focus restoration network is
An input layer into which an out-of-focus image is input,
An output layer where an in-focus image is output,
A restoration block for calculating restoration information; and
and a summing unit summing the output value of the restoration block and the input value of the input layer. According to claim 2,
The reconstruction block is composed of a plurality of sub-blocks,
Each of the sub-blocks includes a first convolution operation unit, a first batch normalization operation unit, a Rectified Linear Unit (ReLU) operation unit, a second convolution operation unit, a second batch normalization operation unit, and a second batch normalization operation unit. A slide imaging method comprising a summing unit for summing the result value and the input value of the corresponding sub-block in units of pixels. According to claim 2,
Slide imaging method, characterized in that the focus restoration network undergoes a network training process of adjusting weights of the first and second convolution operators through learning data. According to claim 4,
The learning data consists of a pair of an out-focus image and an in-focus image taken at the same location as the out-focus image,
The slide imaging method, characterized in that the input value or input image of the training data is the out-focus image, and the output value or output image of the training data is the in-focus image. The method of claim 5, wherein the in-focus image generates an all-in-focus image including all focal areas from N focal plane images taken by sequentially moving a focal position through a Z-axis scanner. Slide imaging method, characterized in that obtained by doing. According to claim 5,
After restoring the out-focus area of the captured image, a value for distinguishing whether an output image of the focus restoration network is an output image of the training data or an output image generated by the focus restoration network is output through a discrimination network Slide imaging method further comprising the step. In the slide imaging device,
objective lens;
an XY stage driven to linearly move the slide in the X and Y axes;
a Z-axis scanner driven to linearly move the objective lens in the Z-axis;
a memory storing one or more routines for performing a slide imaging method; and
a processing component configured to execute one or more routines stored within the memory;
When the one or more routines are executed by the processing component,
moving the XY stage to an initial position;
photographing a slide or sample;
restoring an out-of-focus region of the photographed image using a focus restoration network;
an operation of moving the XY stage to the next position and determining whether slide shooting is to end;
returning to the step of photographing the slide or sample if the photographing of the slide has not ended; and
A slide imaging device for performing an operation of generating an entire slide image by performing matching of images when the photographing of the slide is finished.

Images