KR20230073005A - Device and method for generating whole slide image having z axis scanning - Google Patents

Abstract

The present invention relates to an apparatus and method for generating an image of an entire slide including Z-axis scanning. The method for generating an image of an entire slide including Z-axis scanning according to the present invention includes an XY-axis stage on which a slide is arranged as X-axis and Y-axis. A first step of moving; A second step of moving a Z-axis stage on which an objective lens is mounted in the Z-axis to take a plurality of focal plane images having different focal points for each area of the slide by the objective lens; a third step of generating a single multi-focal image by an image compression unit by compressing the plurality of focal plane images for each region; and a fourth step of generating an entire image of the slide by stitching the plurality of multi-focus images by an image matching unit.

Description

Whole slide image generation device and method including Z-axis scanning {DEVICE AND METHOD FOR GENERATING WHOLE SLIDE IMAGE HAVING Z AXIS SCANNING}

The present invention relates to an apparatus and method for generating an image of an entire slide including Z-axis scanning, and an apparatus for generating an image of an entire slide including Z-axis scanning that solves a problem of deterioration in image quality and enables checking of all cell nuclei in a single image. and methods.

Pathology is a field that identifies the cause of a disease by examining cells or tissues in a specific area.

In general, the cause of the disease is identified by closely examining the cells or tissues applied to the pathology slide using an optical microscope.

An optical microscope magnifies cells or tissues through an objective lens with a magnification of 10 times or more, and a user precisely inspects the enlarged cells or tissues through an eyepiece.

However, as the magnification of the objective lens increases, the field of view (FOV) narrows, and for this reason, only a part of cells or tissues can be observed when a high magnification objective lens is used.

Optical microscopes used in pathology provide an XY stage and stage adjustment knobs that can move the slide in four directions (east, west, north, south) to solve the field of view problem, but it is difficult to manually observe the entire area of the pathology slide, so it is difficult to observe the lesion. Missing may happen.

In order to reduce misdiagnosis caused by manually observing the entire area, a whole slide imaging (WSI) device that automatically scans the entire area of a slide has recently been introduced.

In the whole slide imaging device, an XY stage that automatically scans the entire slide area is added to the optical microscope on the hardware side, and a function that automatically stitches the scanned images to create one super-resolution digital slide image on the software side. provides

Since whole-slide imaging devices provide high-resolution digital images of the entire slide area, they are regarded as one of the key technologies for converting the field of pathology from analog to digital, and various products have recently been introduced.

However, when a cell or tissue to be inspected is thick, the corresponding region may be blurred.

This is a problem that occurs because the depth of field (DoF) of the objective lens used for high-magnification imaging is very narrow in the micrometer (um) unit, and if the thickness of the cell or tissue is out of the depth of focus, it is out of focus. An out of focus (OOF) area is captured.

An out-of-focus image not only poses a video quality problem of the final entire slide image, but also poses a problem of not properly performing follow-up tasks such as image analysis and quantitative analysis.

In the case of an optical microscope, a focusing knob is provided to adjust the focus in case cells or tissues are out of focus (out of focus depth).

Some commercially available whole-slide imaging devices provide a "Z-stack" function to solve the focus problem.

"Z-stack" creates N focal plane images by stacking several images taken while moving the focal position in the Z axis, and stores the image in the form of a 3D volume image.

The stored N focal plane images (3D volume images) can be checked through the image viewer software provided by the product. In particular, since the 3D volume images include images taken at different focal positions, optical microscopes and Similarly, it provides the ability to change focus.

Here, changing the focus means selecting one focal image from among several focal images, and the user selects (controls the focus) the image by scrolling a mouse or the like on image viewer software.

The "Z-stack" function is provided as an option to solve the focus problem, but compared to conventional equipment that shoots only a single focus, the scan time and file capacity (storage space) increase the number of focal plane images constituting a 3D volume image. It has a problem that increases in proportion to .

In particular, since file size is directly related to the cost of maintaining data, there is a tendency not to use the "Z-stack" function despite the advantage of solving the focus problem.

Digital slide images are usually managed in a cloud service and are paid in proportion to the size of the file.

Therefore, there is a need for a method capable of simultaneously solving the focus problem and the file capacity problem of the prior art.

Patent Document 1: Registered Patent Publication No. 10-1423897 (2014.07.28)

The present invention has been made to solve the above problems, the present invention is to reduce the file capacity of a 3D volume image generated by Z-stack, according to the present invention, several focal points included in a 3D volume image It is intended to reduce the size of an image file that is saved by creating an all-in-focus image in which only the focal region is projected from a plane image.

In addition, the present invention is intended to provide improved results in terms of image quality because there is no out-of-focus blurry area, and to perform detailed inspection more quickly by enabling all cell nuclei to be identified in a single image.

A method for generating an entire slide image including Z-axis scanning according to an embodiment of the present invention for solving the above-described problem includes a first step of moving an XY-axis stage on which a slide is disposed in an X-axis and a Y-axis; A second step of moving a Z-axis stage on which an objective lens is mounted in the Z-axis to take a plurality of focal plane images having different focal points for each area of the slide by the objective lens; a third step of generating a single multi-focal image by an image compression unit by compressing the plurality of focal plane images for each region; and a fourth step of generating an entire image of the slide by stitching the plurality of multi-focus images by an image matching unit.

According to another embodiment of the present invention, the third step may include calculating, by the image compression unit, a degree of focus for all pixels of the plurality of focal plane images; calculating, by the image compression unit, focus weights for all the pixels based on the calculated degree of focus; and generating a single multi-focal image by compressing the plurality of focal plane images based on the calculated focus weights by the image compression unit.

According to another embodiment of the present invention, in the step of calculating the degree of focus, the image compression unit may calculate the degree of focus for all pixels of the focal plane image through Equation 1 below.

[Equation 1]

는 i 번째 초점면 이미지의

좌표 픽셀값, k는 윈도우 크기를 나타내며,

는 i 번째 초점면 이미지

좌표의 초점정도를 나타낸다.here,

is the ith focal plane image of

The coordinate pixel value, k represents the window size,

is the ith focal plane image

Indicates the degree of focus of the coordinates.

In addition, according to another embodiment of the present invention, in the step of calculating the focus weight, the image compression unit may calculate the focus weight for all the pixels based on the calculated degree of focus through Equation 2 below. there is.

[Equation 2]

는 초점 가중치,

는 초점정도,

로서 모든 초점면 이미지에서

좌표의 초점정도 합을 나타낸다.here,

is the focus weight,

is the degree of focus,

in all focal plane images as

It represents the sum of the degree of focus of the coordinates.

In addition, according to another embodiment of the present invention, in the generating of the multi-focal image, the image compression unit compresses the plurality of focal plane images based on the calculated focus weight through Equation 3 below. One multifocal image can be created.

[Equation 3]

는 출력 이미지의

좌표 픽셀값,

는 초점 가중치,

는 i 번째 초점면 이미지의

좌표 픽셀값을 나타낸다.here,

of the output image

coordinate pixel value,

is the focus weight,

is the ith focal plane image of

Indicates the coordinate pixel value.

In addition, an entire slide image generating device including Z-axis scanning according to an embodiment of the present invention includes an XY-axis stage on which a slide is disposed and moved in the X-axis and Y-axis; A Z-axis stage on which an objective lens is mounted and moved in the Z-axis; an objective lens for capturing a plurality of focal plane images having different focal points for each area of the slide; an image compression unit generating one multi-focal image by compressing the plurality of focal plane images for each region; and an image stitching unit configured to create an entire image of the slide by stitching the plurality of multi-focus images.

In addition, according to another embodiment of the present invention, the image compression unit calculates the degree of focus for all pixels of the plurality of focal plane images, and the degree of focus for all the pixels based on the calculated degree of focus. A focus weight may be calculated for each focal plane, and a plurality of focal plane images may be compressed based on the calculated focus weight to generate a single multi-focal image.

The present invention is intended to reduce the file size of a 3D volume image generated by Z-stack, and according to the present invention, a single focal plane image in which only a focal region is projected from several focal plane images included in a 3D volume image is created. By creating an all-in-focus image, the size of an image file can be reduced.

In addition, according to the present invention, since there is no out-of-focus blurry area, the result is improved in terms of image quality, and since all cell nuclei can be identified in a single image, there is an advantage in that detailed inspection can be performed quickly.

1 is a diagram showing an entire slide image generating device including Z-axis scanning according to an embodiment of the present invention.
2 is a flowchart illustrating a method of generating an entire slide image including Z-axis scanning according to an embodiment of the present invention.
3 is a diagram for explaining a method of generating a multi-focal image by compressing a plurality of focal plane images for each region according to an embodiment of the present invention.
4 is a flowchart illustrating a method of generating a multi-focal image by compressing a plurality of focal plane images for each region according to an embodiment of the present invention.
FIG. 5 is a diagram showing a plurality (N) focal images that are inputs of a focal region projection method according to an embodiment of the present invention and a multi-focus image that is an output image onto which all focal regions are projected.
6 is a diagram for explaining an entire image of a slide created according to an embodiment of the present invention.
7 is a block diagram of an entire slide image generating apparatus including Z-axis scanning according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, a preferred embodiment of the present invention will be described in detail. However, in describing the embodiments, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present invention, a detailed description thereof will be omitted. In addition, the size of each component in the drawings may be exaggerated for description, and does not mean a size that is actually applied.

1 is a diagram showing an entire slide image generating device including Z-axis scanning according to an embodiment of the present invention, and FIG. 2 is a diagram showing a whole slide image generating method including Z-axis scanning according to an embodiment of the present invention. It is a flow chart to explain.

3 is a diagram for explaining a method of generating a multi-focal image by compressing a plurality of focal plane images for each region according to an embodiment of the present invention, and FIG. This is a flowchart for explaining a method of generating a multifocal image by compressing a plurality of focal plane images for each region.

Hereinafter, an apparatus and method for generating an entire slide image including Z-axis scanning according to an embodiment of the present invention will be described with reference to FIGS. 1 to 4 .

Referring to FIG. 1, an apparatus for generating an entire slide image including Z-axis scanning according to an embodiment of the present invention includes an objective lens 110, an XY-axis stage 120, and a Z-axis stage 130. can be configured.

The objective lens 110 magnifies a sample to be observed, and generally consists of an objective lens providing magnification of 20 times, 40 times, or more, so that the slide 101 can be photographed.

Magnified images taken through a high-magnification objective lens are helpful for detailed examination of cells or tissues, but as the magnification increases, the field of view (FOV) and depth of field (DoF) decrease, so slide 101 ), only a small part of the entire area, or a sample located within a specific focal range, can be precisely observed.

Therefore, the XY-axis stage 120 is used to expand the viewing angle, which is one of the limitations of the objective lens 110, and the entire area of the slide 101 or the area where cells or tissues are applied can be photographed. 101) to the X-axis and Y-axis.

In addition, the Z-axis stage 130 is used to photograph the slide 101 in a wider focal range, and serves to move the objective lens 110 mounted on the Z-axis stage 130 in the Z-axis. carry out

Referring to FIG. 2 , describing the method of generating an entire slide image including Z-axis scanning according to an embodiment of the present invention, the XY-axis stage 120 on which the slide is placed moves in the X-axis and Y-axis (S210 ). At this time, the XY-axis stage 120 may move to the upper left or upper right of the slide 101 to capture the initial position of the slide 101 .

Thereafter, the Z-axis stage 130 equipped with an objective lens is moved along the Z-axis, and the objective lens photographs the slide 101 to generate a plurality of focal plane images 320 having different focal points (S220). That is, the Z-axis stage 130 sequentially captures several images while moving a predefined focus range for each area of the slide 101, and generates several images sequentially taken while changing the focus, As described above, in the present invention, several images sequentially photographed while changing the focus for each region (predefined focal range) are defined as the focal plane image 320 .

At this time, the image compression unit may compress the plurality of focal plane images 320 generated in this way for each region (S230), and then the objective lens 110 slides using an XY stage so that the next position can be photographed. Move (S240).

Accordingly, as shown in FIG. 3 , one multi-focal image 340 may be generated by compressing the focal plane image 320 for each focus.

3 is a diagram for explaining a focal plane image 320, a multi-focal image 340, and an overall image 350 of a slide according to an embodiment of the present invention.

At this time, as shown in FIG. 3, in order to solve the focal range problem, the Z-stack technique is used to match the images 320 on a focal plane basis to generate an image of the entire slide 350 (320). According to the prior art, this Stacked in the Z axis, a 3D volume image 330 was output. As a result, in the prior art, since high resolution whole slide images are generated as many as the number of focal planes, file capacity increases in proportion to the number of focal planes compared to devices that generate entire slides with only a single focal plane.

Therefore, according to one embodiment of the present invention, the focal plane image 320 is compressed for each region (predefined focal range) of the slide 101 to generate one multi-focal image 340, and the slide is used. This problem was solved by constructing the entire image 350 of (101).

That is, since the purpose of the slide image generating device 100 according to an embodiment of the present invention is to convert the entire area of the slide 101 into an image, the XY-axis stage 120 and the Z-axis stage 130 The process of capturing the focal plane image 320 by moving is repeated, and when the image capturing of the entire area of the slide 101 is completed, the next step proceeds.

Accordingly, in the next step, a plurality of high-magnification multi-focus images 340 generated while moving the XY stage 120 are stitched (S250), and finally, the whole of the high-resolution slide 101 is stitched. An image 350 is created (S260).

Hereinafter, referring to FIG. 4 , a method of generating one multi-focal image 340 by compressing a plurality of focal plane images 320 for each region according to an embodiment of the present invention will be described in detail.

According to an embodiment of the present invention, after generating one multi-focal image 340 by projecting a focal region from all focal plane images 320, a plurality of multi-focal images 340 are matched to form one image ( 350), and compressing multiple (N) focal plane images 320 into one multi-focal image 340, it is called image compression 225. That is, by compressing a plurality of focal plane images 320 generated by photographing the slide 101 by the objective lens 110, the focal region is projected onto a single image, and a multi-focal (all-focus region) including all focal regions is projected. An in-focus image 340 may be output.

First, the image compression unit calculates a degree of focus for all pixels of the plurality of focal plane images (S231). In this case, the higher the degree of focus, the higher the probability of being projected onto the final image.

Equation 1 below shows a method of calculating the degree of focus for all pixels based on the Brenner algorithm.

[Equation 1]

는 i 번째 초점면 이미지의

좌표 픽셀값을, k는 윈도우 크기를 나타내며,

는 i 번째 초점면 이미지

좌표의 초점정도를 나타낸다.here,

is the ith focal plane image of

The coordinate pixel value, k represents the window size,

is the ith focal plane image

Indicates the degree of focus of the coordinates.

)를 기반으로 상기 모든 픽셀에 대해 초점 가중치를 계산한다(S232).Thereafter, the image compression unit calculates the degree of focus (

), focus weights are calculated for all the pixels (S232).

좌표에서

번째 픽셀의 초점정도

가 상대적인 얼마나 높은 값을 가지는지를 비율로 표현한 값으로서, 하기의 수학식 2와 같이 표현될 수 있다.the focus weight is

at the coordinates

degree of focus of the second pixel

As a value expressed as a ratio of how high a relative value has, it can be expressed as in Equation 2 below.

[Equation 2]

는 초점 가중치,

는 초점정도,

로서 모든 초점면 이미지에서

좌표의 초점정도 합을 나타낸다. 따라서, 초점가중치

는

좌표의 모든 초점정도 합에 대한

번째 초점면 이미지의 초점정도 비율을 의미한다.here,

is the focus weight,

is the degree of focus,

in all focal plane images as

It represents the sum of the degree of focus of the coordinates. Therefore, the focal weight

Is

For the sum of all focal degrees of coordinates

It means the ratio of the degree of focus of the second focal plane image.

In the process of applying cells to a slide, cells may overlap on top of cells, and the two cells tend to be clearly visible in different focal planes.

Therefore, in one embodiment of the present invention, focus weights are used to project both cells or more cell information photographed from different focal planes into one image, and the image is projected with the average of the focus weights of each cell area. A final multifocal image can be created.

Thereafter, the image compression unit compresses the plurality (N) of the focal plane images based on the calculated focus weights to generate one multi-focal image (S233), which can be expressed as Equation 3 below. .

[Equation 3]

는 출력 이미지의

좌표 픽셀값,

는 초점 가중치,

는 i 번째 초점면 이미지의

좌표 픽셀값을 나타내며, 픽셀값에 초점가중치를 곱한 합으로 N개의 초점 이미지로부터 한 장의 출력 이미지(다초점 이미지)를 생성한다.here,

of the output image

coordinate pixel value,

is the focus weight,

is the ith focal plane image of

It represents coordinate pixel values, and one output image (multifocal image) is created from N focus images as the sum of pixel values multiplied by focus weight.

Since a value with a relatively high degree of focus has a high focus weight, there is a high probability that a pixel value at a corresponding position is projected onto a final focused image.

The finally generated output image is the multi-focus image 350, and since only the focal area among multiple (N) focal plane images is compressed into a single image, it can be expressed as an all-in-focus image.

FIG. 5 is a diagram showing a plurality (N) focal images that are inputs of a focal region projection method according to an embodiment of the present invention and a multi-focus image that is an output image onto which all focal regions are projected.

5C shows an example of overlapping cells, and the two cells tend to be clearly visible in different focal planes 560 and 570, respectively.

According to the present invention, a single multifocal image is created by averaging two or more cell information taken at different focal planes with a focus weight, so that all focal regions can be projected into a final image even when cells overlap.

Figure 6 is a view for explaining the entire image of a slide created according to an embodiment of the present invention, Figure 6 (a) is a view showing an image according to the prior art, Figure 6 (b) is this It is a diagram showing the entire image of a slide created according to one embodiment of the present invention.

As shown in (b) of FIG. 6, the multifocal image according to an embodiment of the present invention showed improved results in terms of image quality because there is no out of focus (OOF) area (blurred area out of focus). In particular, since all cell nuclei can be identified in a single image, it has the advantage of being able to perform precise inspection quickly.

In this way, according to the present invention, compared to the method of generating an entire slide image with only a single focal plane in the prior art, the output image includes all focus information and can reduce the OOF area (out of focus blurry area), resulting in improved image quality can be provided.

In addition, according to the present invention, compared to the method using the Z-Stack technique proposed to reduce the OOF area of the prior art, since only one entire image is stored, the size of the stored file can also be reduced.

7 is a block diagram of an entire slide image generating apparatus including Z-axis scanning according to an embodiment of the present invention.

Hereinafter, the configuration of an entire slide image generating apparatus including Z-axis scanning according to an embodiment of the present invention will be described with reference to FIG. 7 .

Referring to FIG. 7 , an entire slide image generating apparatus 100 including Z-axis scanning according to an embodiment of the present invention includes an objective lens 110, an XY-axis stage 120, a Z-axis stage 130, an image It is configured to include a processing unit 140 and an image matching unit 160 .

The XY-axis stage 120 has a slide and moves in the X-axis and Y-axis, and the Z-axis stage 130 has the objective lens 110 mounted thereon and moves in the Z-axis.

The objective lens 110 mounted on the Z-axis stage 130 captures the slide and generates multiple focal plane images with different focal points.

Accordingly, the image compression unit 140 may generate one multi-focal image by compressing the plurality of focal plane images for each region.

More specifically, the image compression unit 140 calculates the degree of focus for all pixels of the plurality of focal plane images, and the degree of focus for all the pixels based on the calculated degree of focus. Weights may be calculated, and a plurality of focal plane images may be compressed based on the calculated focus weights to generate one multi-focal image.

Accordingly, the image matching unit 160 may generate an entire image of the slide by stitching the plurality of multi-focus images.

In the detailed description of the present invention as described above, specific embodiments have been described. However, various modifications are possible without departing from the scope of the present invention. The technical spirit of the present invention should not be limited to the above-described embodiments of the present invention and should not be defined, and should be defined by not only the claims but also those equivalent to these claims.

100: Full slide image generating device including Z-axis scanning
110: objective lens
120: XY axis stage
130: Z-axis stage
140: image compression unit
150: image matching unit

Claims (7)

A first step of moving the XY-axis stage on which the slide is arranged in the X-axis and Y-axis;
A second step of moving a Z-axis stage on which an objective lens is mounted in the Z-axis to take a plurality of focal plane images having different focal points for each area of the slide by the objective lens;
a third step of generating a single multi-focal image by an image compression unit by compressing the plurality of focal plane images for each region; and
a fourth step of generating an entire image of the slide by an image matching unit stitching a plurality of the multi-focus images;
A method for generating an entire slide image including Z-axis scanning.
The method of claim 1,
The third step,
calculating, by the image compression unit, a degree of focus for all pixels of the plurality of focal plane images;
calculating, by the image compression unit, focus weights for all the pixels based on the calculated degree of focus; and
generating a single multi-focal image by compressing the plurality of focal plane images based on the focal weight calculated by the image compression unit;
A method for generating an entire slide image including Z-axis scanning.
The method of claim 2,
In the step of calculating the degree of focus,
The entire slide image generation method including Z-axis scanning in which the image compression unit calculates a degree of focus for all pixels of the focal plane image through Equation 1 below.
[Equation 1]

here,

is the ith focal plane image of

Coordinate pixel values, k represents the window size,

is the ith focal plane image

Indicates the degree of focus of the coordinates.
The method of claim 3,
The step of calculating the focus weight,
The entire slide image generation method including Z-axis scanning in which the image compression unit calculates focus weights for all the pixels based on the calculated degree of focus through Equation 2 below.
[Equation 2]

here,

is the focus weight,

is the degree of focus,

in all focal plane images as

Indicates the sum of the degree of focus of the coordinates.
The method of claim 4,
The step of generating the multifocal image,
The entire slide image generation method including Z-axis scanning in which the image compression unit generates one multi-focal image by compressing a plurality of the focal plane images based on the calculated focus weight through Equation 3 below.
[Equation 3]

here,

of the output image

coordinate pixel value,

is the focus weight,

is the ith focal plane image of

Indicates coordinate pixel values.
An XY-axis stage on which slides are arranged and moved in the X-axis and Y-axis;
A Z-axis stage on which an objective lens is mounted and moved in the Z-axis;
an objective lens for capturing a plurality of focal plane images having different focal points for each area of the slide;
an image compression unit generating one multi-focal image by compressing the plurality of focal plane images for each region; and
an image stitching unit generating an entire image of the slide by stitching the plurality of multi-focus images;
Entire slide image generating device including Z-axis scanning including a.
The method of claim 6,
The image compression unit,
Calculate a degree of focus for every pixel of a plurality of the focal plane images;
Calculate focus weights for all the pixels based on the calculated degree of focus;
An entire slide image generating device including Z-axis scanning for generating one multi-focal image by compressing a plurality of focal plane images based on the calculated focal weight.

Images