JP2016051168A - Image acquisition device and control method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image acquisition device capable of rapidly and accurately determining an image capturing range using a simple configuration.SOLUTION: An image acquisition device divides a sample into multiple areas to capture an image comprising multiple layers with different focal positions in an optical axis direction or a single layer in each area and is configured as follows. The image acquisition device has image capturing means. The image acquisition device also includes specimen information acquisition means that uses a single- or multi-layered image of a first area of a sample captured by the image capturing means to estimate a presence range or the most in-focus position, in the optical axis direction, of a specimen included in a second area of the sample other than the first area, and sets an image capturing range in the optical axis direction for the second area based on a result of the estimation.SELECTED DRAWING: Figure 1A

Description

The present invention relates to an image acquisition device and a control method thereof.

In recent years, in the field of pathology, an image acquisition device such as a virtual slide system that acquires a microscopic image of a pathological specimen such as a tissue piece as a digital image has attracted attention. By digitizing the pathological diagnosis image, it becomes possible to improve the efficiency of data management, remote diagnosis, and the like.

A sample to be imaged by the apparatus is a slide (also called a preparation), and a tissue piece sliced thinly to several to several tens [um] is fixed between a slide glass and a cover glass via an encapsulant. Yes. In general, the thickness of a tissue piece is not always constant, and in addition to the surface having irregularities, the tissue piece itself does not necessarily have a substantially planar shape but has undulations. Therefore, it is necessary to appropriately set the existence range in the thickness direction of the tissue piece, that is, the range of the imaging layer in the optical axis direction for each imaging range that is an imaging field of the apparatus. This is because a focused image of the entire tissue piece in the thickness direction is acquired through the optical system of the pathological observation microscope, which has a shallow depth of field of about 0.5 to 1 [um] because of high resolution.

For such a problem, the in-focus position calculated based on the pixel data of the overlapping portion with the adjacent imaging range is set as the imaging start position of the adjacent imaging range, and then scanned up and down along the optical axis direction from there. . A method for determining a necessary imaging range by doing so has been proposed (Patent Document 1).

In addition, by providing a system for imaging and imaging a phase difference image separately, the unevenness of the subject is detected, and the search process in the optical axis direction is not performed based on the relationship between the detected unevenness range and the depth of field. A method for determining the range of the imaging layer has been proposed (Patent Document 2).

JP2013-029551A JP 2011-090222 A

However, the conventional image acquisition apparatus as described above has the following problems. That is, in the method of performing scanning processing in the optical axis direction for each imaging range as in Patent Document 1, it takes time to determine the imaging layer range in the optical axis direction, and it is difficult to improve throughput. It was.

Further, in the method of providing an optical system dedicated to focusing and an imaging element as in Patent Document 2, high-speed processing can be expected, but it is difficult to avoid the increase in size and cost of the apparatus.

The invention according to the present application has been made in view of such circumstances, and an object thereof is to provide an image acquisition apparatus capable of determining an imaging range with high speed and high accuracy with a simple configuration.

In order to achieve the above object, the present invention adopts the following configuration. That is, an image acquisition device that divides a sample into a plurality of regions and captures images of a plurality of layers or single layers having different focal positions in the optical axis direction for each region, the imaging unit that images the sample, and the imaging The presence range or the most in-focus in the optical axis direction of the specimen contained in the second region of the sample different from the first region based on the single-layer or multi-layer image of the first region of the sample imaged by the means An image acquisition apparatus comprising: a sample information acquisition unit configured to estimate a position and set an imaging range of the second region in the optical axis direction based on the estimation result.
The present invention adopts the following configuration. That is, a method of controlling an image acquisition apparatus that divides a sample into a plurality of regions and captures images of a plurality of layers or single layers having different focal positions in the optical axis direction for each region, Imaging a layer or a plurality of layers, and a sample contained in a second region of the sample different from the first region based on the imaged single layer or a plurality of layers of the first region of the sample A method for controlling an image acquisition apparatus, comprising: estimating an existence range or a most in-focus position in the optical axis direction; and setting an imaging range in the optical axis direction of the second region based on the estimation result. It is.

As described above, according to the present invention, it is possible to provide an image acquisition device capable of determining an imaging range with high speed and high accuracy with a simple configuration.

Block diagram showing a first embodiment of the image acquisition apparatus of the present invention (first embodiment) Sectional drawing which shows the slide of the image acquisition apparatus in Example 1 6 is a flowchart illustrating an imaging process of the image acquisition apparatus according to the first embodiment. Schematic diagram illustrating a method of calculating the XY imaging range in the first embodiment Schematic diagram showing multilayer imaging in the Z direction in the first embodiment. Flow chart of Z stack imaging in the first embodiment A flowchart showing Z search imaging in the first embodiment. The figure which shows the setting method of the Z stack range in Example 1 Flowchart showing a second embodiment of the image processing apparatus of the present invention (second embodiment) The figure which shows the additional imaging of Z stack imaging in Example 2 3 is a perspective view showing a third embodiment of the image acquisition device of the present invention (third embodiment). FIG. The perspective view which shows Example 4 of the image acquisition apparatus of this invention (Example 4).

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the invention according to the claims, and all combinations of features described in the embodiments are not necessarily essential to the solution means of the invention.

<Example 1>
(Device configuration)
FIG. 1A is a block diagram illustrating a first embodiment of an image acquisition device according to the present invention. The image acquisition device 1 (hereinafter abbreviated as “device 1”) includes a main imaging device 200 (corresponding to the imaging unit) that performs the main imaging, and a wide-area imaging device 300 (for the wide-area imaging unit) that performs preliminary imaging prior to the main imaging. Corresponding), and a main body control unit 100 that performs operation control of the apparatus, image processing, and the like. In the figure, broken line arrows indicate data signals relating to image information, and solid line arrows indicate control command signals, status signals, and the like. First, an outline of these will be described. There are other things that are not shown, but these will be described later as appropriate.

The imaging apparatus 200 captures a microscopic image of a slide 10 that is a sample in which a specimen such as a tissue piece is enclosed. The imaging apparatus 200 includes an illumination unit 210 that irradiates light to the slide 10 and a stage 220 that positions the slide 10 and supports the slide 10, that is, the sample. Furthermore, it has a lens unit 230 that is an imaging optical system for condensing and imaging light from the slide 10, and an imaging device 240 for converting the imaged light into an electrical signal. In the present embodiment, as shown in FIG. 1A, the optical axis direction of the lens unit 230 is defined as the Z direction, and the horizontal plane direction orthogonal to the optical axis direction is defined as the XY direction. As an imaging method, a multilayer image (Z stack image) of the specimen 14 described later is acquired for each small section (each area) described later. Hereinafter, this multilayer image is referred to as a Z stack image. The Z stack image refers to a plurality of two-dimensional images obtained as a result of imaging a subject while changing the focal position little by little in the optical axis direction. That is, it is an image obtained as a result of imaging for each focal position. Z stack imaging refers to processing for obtaining a plurality of two-dimensional images by imaging a subject while gradually changing the focal position in the optical axis direction. In addition, a two-dimensional image at each focal position constituting the Z stack image is referred to as a layer image.

The wide-area imaging device 300 captures an entire image of the slide 10 as viewed from above, and includes a sample placement unit 310 that installs the slide 10 and a wide-area imaging unit 320 that images the slide 10. The acquired image is used to create a thumbnail image of the slide 10, to divide and generate a small section 801, which will be described later, and to acquire sample identification information such as a barcode or a two-dimensional code on the slide 10. Used.

The main unit 100 controls the operation of the apparatus 1 and communicates with an external device (not shown), image processing on image data of the wide-area imaging unit 320 and the image sensor 240, and output of image data to an external device (not shown). An image processing unit 120 is provided. Furthermore, it has the calculating part 130 (it respond | corresponds to a sample information acquisition means) which performs the calculation regarding a focus. In the figure, the blocks are divided for each function for convenience. However, the implementation means may be implemented as software operating on a CPU or DSP, or hardware such as an ASIC or FPGA, and the separation may be designed appropriately. good. Examples of the external device (not shown) include a user interface between the device 1 and the user, a PC workstation that functions as an image viewer, an external storage device that performs image data storage management, an image management system, and the like. In addition, examples of the apparatus 1 that are not shown include a slide stocker that sets a large number of slides 10, a sample transport unit that transports the slides 10 to a mounting table, that is, a sample mounting unit 310 and a stage 220. Detailed descriptions of those not shown are omitted.

Each of the above components will be further described. The illumination unit 210 includes a light source that emits light and an optical system that collects the light with respect to the slide 10. As the light source, a halogen lamp, an LED, or the like is used. The stage 220 has a position control mechanism that holds the slide 10 and moves it precisely in each of the XYZ directions. This is realized by a driving mechanism such as a combination of a motor and a ball screw or a piezo element. In addition, a slide holding and fixing mechanism such as a vacuum is provided to prevent displacement of the slide 10 due to acceleration during stage movement. The lens unit 230 includes an objective lens and an imaging lens, and forms an image of the transmitted light of the slide 10 irradiated from the illumination unit 210 on the light receiving surface of the image sensor 240. As a lens, the object side FOV (Field
of view: field of view, imaging range) of approximately 1 [mm] square and a depth of field of about 0.5 [um] is desirable. The image sensor 240 is an image sensor such as a CCD (charge coupled device) or a CMOS (complementary metal oxide semiconductor). The light received in accordance with the exposure time, sensor gain, exposure start timing, etc. set based on the control signal from the control unit 110 is photoelectrically converted into an electrical signal, which is output to the image processing unit 120 and the calculation unit 130. The sample mounting unit 310 is a table for mounting the slide 10. The table is provided with an abutment mechanism so that the XY position of the slide 10 can be positioned with respect to the sample placement unit 310. The stage 220 may also serve as the sample placement unit 310 without being limited to the configuration of FIG. 1A, and in that case, the stage 220 can be configured by widening the XY movable range.

The wide area imaging unit 320 includes an illumination unit (not shown) that irradiates the slide 10 placed on the sample placement unit 310 with illumination light, and a camera unit (not shown) that includes a lens, an imaging element, and the like. Based on the control signal from the control unit 110, the exposure time, sensor gain, exposure start timing, illumination amount, and the like are set, and the captured data is output to the image processing unit 120. Note that magnification, position, and the like are configured so that dark field illumination using ring illumination provided around the lens is performed, and the entire image of the slide 10 can be captured by a single imaging. The resolution of the camera unit can be configured at low cost because it may be about the imaging range in the imaging apparatus 200 or low resolution or low resolution that can identify the two-dimensional code so that rough detection of the existence range of the specimen 14 can be performed.

The control unit 110 controls the operation of each component of the device 1 based on an operation process described later. Specifically, setting of operation conditions, instruction of operation timing, and the like are performed. The wide area imaging unit 320 is set and controlled for exposure time, sensor gain, exposure start timing, illumination light quantity, and the like. The illumination unit 210 is instructed to switch the light amount, the aperture, and the color filter. For the stage 220, the stage 10 is moved in the XYZ directions based on the output result of the arithmetic unit 130, information on the small section 801 described later, the current position information of the stage by an encoder (not shown), and the like. It controls so that a site | part can be imaged. The image sensor 240 is set and controlled for exposure time, sensor gain, exposure start timing, and the like. Between the image processing unit 120, the setting and control of the operation mode and timing, and the processing result of the wide area imaging data such as information on the small section and the barcode are received. Further, the control unit 110 communicates with an external device (not shown). Specifically, acquisition of operation conditions set by the user via an external device, control of operation start / stop of the device, instruction to output image data to the image processing unit 120, and the like are performed.

The image processing unit 120 mainly has two functions. The first is processing of wide area imaging data of the slide 10 received from the wide area imaging unit 320. Wide-area imaging data is analyzed, barcode information is read, rough (rough) detection of the existence range of the specimen 14 in the XY directions, division of the small sections 801, and generation of thumbnail images are performed. “Rough” as used herein means that, for example, the resolution or resolution of the wide-area imaging unit 320 is lower than the resolution or resolution of the imaging apparatus 200 as described above. The imaging optical system of the imaging apparatus 200 is an enlargement optical system, whereas the imaging optical system of the wide area imaging unit 320 is a reduction optical system. With this configuration, the wide-area imaging unit 320 can be configured at a low cost, and the amount of calculation is reduced during image processing, so that the speed of image processing increases. Here, the control unit 110 controls the main imaging process using the imaging apparatus 200 based on the information (coordinates, number of sheets, etc.) of the generated small section 801 group. The division generation of the small section 801 group will be described in detail in the section “Calculation of XY direction imaging range in the first embodiment”. The second is processing of the main image data of the slide 10 received from the image sensor 240. The main image data is subjected to various correction processes such as sensitivity differences between RGB and γ curves, data compression processing as necessary, protocol conversion processing, etc., and a viewer and image storage based on instructions from the control unit 110 Data is transmitted to an external device such as a device.

The calculation unit 130 includes a distribution calculation unit 131, a sample estimation unit 132, and a setting unit 133. The calculation unit 130 performs a calculation related to focusing based on the main imaging data received from the imaging element 240 and determines the Z stack range based on the calculation result. The Z stack range is a range from the focal position at the first imaging (position in the Z direction) to the focal position at the last imaging (position in the Z direction) in Z stack imaging. Z stack imaging is a process of obtaining a plurality of two-dimensional images by imaging a subject within the Z stack range (within the imaging range) while gradually changing the focal position in the optical axis direction. is there. Then, the determined Z stack range is output to the control unit 110. The distribution calculating unit 131 calculates a two-dimensional distribution of the focus evaluation index (for example, contrast value) of each pixel of the main imaging data acquired by the main imaging for each layer by the imaging apparatus 200, and 2 of each layer. A three-dimensional distribution of the focus evaluation index of the specimen 14 is calculated by combining the dimensional distributions. The sample estimation unit 132 inputs the three-dimensional distribution from the distribution calculation unit 131, and based on the input three-dimensional distribution, the sample estimation unit 132 is adjacent to the small section 801 and focused on the sample 14 in the small section 801 to be imaged next. Estimate the three-dimensional distribution of the evaluation index. The setting unit 133 inputs the three-dimensional distribution estimated from the sample estimation unit 132. Based on the inputted three-dimensional distribution, the Z stack range of the small section 801 to be imaged next is set so that the range in which the specimen 14 is estimated to be included is included without excess or deficiency, and the setting result is sent to the control unit 110. Output. The operation of the calculation unit will be described in detail in the section “Calculation of Z-direction imaging range”.

In addition, implementation of this invention is not restricted to a present Example. For example, it is good also as a structure which has a some image pick-up element and acquires the several image focused on different Z positions simultaneously by each moving to an optical axis direction. In this case, since the number of imaging required to obtain a multilayer image is reduced, an improvement in the throughput of the apparatus can be expected. Further, if the optical conjugate relationship is the same as the above-described configuration, for example, the configuration may be such that the sample is fixed to the mounting table and the position of the imaging element and the lens unit is controlled by a stage or the like.

FIG. 1B is a cross-sectional view illustrating the slide of the image acquisition device according to the first embodiment. In the slide 10, a specimen 14 such as a tissue piece to be imaged is fixed between a slide glass 12 serving as a base of the slide and a cover glass 11 serving as a protective film via an encapsulant 13.

(Imaging process)
FIG. 2 is a flowchart illustrating an imaging process of the image acquisition apparatus according to the first embodiment. The imaging process is roughly divided into three steps: preliminary imaging (wide area imaging) in steps S101 to S103, initial Z search in steps S104 to S108, and main imaging in steps S104, S105, and S109 to S113.

The flow starts when the slide 10 is placed on the sample placement unit 310. This installation is a preparation stage of image acquisition, and may be automatically processed from the slide stocker by the sample transport means or may be performed manually. In step S <b> 101, the wide area imaging device 300 captures the entire area of the slide 10. In step S102, the image processing unit 120 roughly detects the existence range of the specimen 14 on the XY plane described later based on the imaging data. The accuracy of this detection may be approximately the same as the FOV of the imaging apparatus 200, that is, the imaging range. That is, it is sufficient if the size of one pixel of the whole image captured by the wide-area imaging device 300 is equal to or smaller than the imaging field (imaging range) of the imaging device 200. The image processing unit 120 detects the existence range of the specimen 14 on the XY plane based on the whole image. In step S103, the calculation unit 130 performs a process of dividing into a plurality of small sections 801, which will be described later, so as to cover the entire existence range of the specimen 14. A specific processing method in steps S102 and S103 will be described in detail in “Calculation of XY-direction imaging range”. In parallel with steps S102 and S103, the slide 10 that has undergone wide-area imaging is placed on the stage 220 and fixed. As described above, this slide movement process may be performed manually or automatically by a transport mechanism. Alternatively, the stage 220 may serve as the sample placement unit 310 so that the movement process can be omitted. When steps S101 to S103, which are preliminary imaging, are completed, the process proceeds to step S104.

In step S <b> 104, the stage 220 on which the slide 10 is placed is moved by the control unit 110 so that the small section 801 that performs the first imaging by the imaging apparatus 200 is positioned directly below the lens included in the lens unit 230. The small section 801 to be imaged first can be arbitrarily determined, but it is desirable to select one that is in a position where it can be imaged while moving all the small sections like a single stroke. In step S105, since the initial search process is not performed at this time, the process proceeds to NO, and the process proceeds to step S106. That is, step S105 proceeds to NO only once at the beginning, and from the second time, it proceeds only to YES until all imaging processes for the slide 10 are completed. In step S <b> 106, the imaging apparatus 200 performs imaging processing for a Z search, which will be described later, performed only in the first imaging section. In step S107, the distribution calculation unit 131 calculates a focus evaluation index based on the multilayer imaging data (Z stack image data) in the Z direction acquired in step S106, and the existence range of the specimen 14 in the Z direction to be described later. Is calculated. In step S108, the calculation unit 130 sets a Z stack range in order to perform main imaging so as to cover the calculated existence range. The Z search in steps S106 to S108 is an imaging process for detecting the specimen existing range in the optical axis direction, and will be described in detail in “Search for Z direction imaging range”.

When the Z search performed only for the small section 801 to be imaged first is completed, the imaging apparatus 200 performs continuous multilayer imaging in the Z direction (Z stack imaging) for the small section 801 in step S109. Step S109 will be described in detail in “Continuous multilayer imaging in the Z direction”. In step S110, the distribution calculation unit 131 calculates a three-dimensional distribution of the focus evaluation index based on the continuous multilayer imaging data acquired in step S109. In step S111, the arithmetic unit 130 determines whether or not the last small section 801 that has been finally imaged is the last small section. Here, since it is not the last small section, it progresses to NO and transfers to step S112. In step S 112, the three-dimensional distribution of the focus evaluation index calculated by the distribution calculation unit 131 is input to the specimen estimation unit 132. The specimen estimation unit 132 then estimates the three-dimensional distribution (existing range of the specimen 14) of the focus evaluation index of the adjacent small section 801 to be imaged next based on the three-dimensional distribution. Then, in step S113, the setting unit 133 inputs the estimated three-dimensional distribution from the sample estimation unit 132 and includes all of the estimated three-dimensional distributions of the focus evaluation indexes of the sample 14. The Z stack range in the small section 801 to be imaged is set. And it transfers to step S104 again. Steps S110 and S112 to S113 will be described in detail in “Setting of Z-direction imaging range”. In step S104, the stage moves XY to the adjacent small section 801 to be imaged next. After that, the main imaging process represented by steps S104, S105, and S109 to S113 is repeated until all the small sections have been imaged, and the process proceeds to YES in step S111 at the time of the last small section imaging, and the flow ends.

(Calculation of XY direction imaging range)
FIG. 3 is a schematic diagram illustrating a method for calculating the XY imaging range in the first embodiment. This shows a state where the calculation unit 130 is divided into a plurality of small sections 801 so as to include all the specimens 14 in the slide 10. As an image processing method for detecting the existence range of the specimen 14 from the wide-area imaging data obtained by the wide-area imaging device 300 imaging the entire area of the slide 10, a method of discriminating a high-contrast area by binarizing the contrast of the image. Representative. This utilizes the fact that cells, tissue fragments, etc. in the slide 10 are stained to distinguish them from the surrounding encapsulant 13, and as a result, the contrast is rendered high. In addition, there is a method in which the edge of the specimen 14 is detected by paying attention to the change in contrast and the inside of the specimen 14 is set as the existence range of the specimen 14. That is, it is desirable to prevent or remove dust, dust, or the like that rarely adheres to the slide from being erroneously detected as the specimen 14 and to make an algorithm that targets only the specimen 14 as an imaging target. The small section 801 is substantially equal to the size of one pixel of the wide-area imaging device 300 or a plurality of pixel data of the wide-area imaging device 300 is averaged, and the size is set to the FOV, that is, the imaging range of the imaging device 200. It is a thing that is almost matched. Note that the actual imaging range (small section) is slightly larger than that shown in FIG. 3 in order to connect the images of the adjacent sections without displacement or distortion in the subsequent image processing. Therefore, in actuality, adjacent small sections are arranged so that their four sides slightly overlap.

(Search for Z direction imaging range)
FIG. 4 is a schematic diagram illustrating multilayer imaging in the Z direction according to the first embodiment. FIG. 4A is a cross-sectional view of the slide 10, and FIG. 4B is an enlargement of the alternate long and short dash line region 901 in FIG. The method of step S106 which is the Z search imaging process is also shown. The imaging range 802 is determined by the imaging range (small section) in the XY direction and the depth of field in the Z direction, and is a three-dimensional region that can be imaged by one exposure. In FIG. 4B, a plurality of imaging ranges 802 are arranged at a constant interval d in the Z direction. An imaging range 802 is arranged from the upper end of the area 901, that is, near the lower end of the cover glass 11, to the lower end of the area 901, that is, the upper end of the slide glass 12. Then, by setting the interval d between the imaging ranges 802 to be about a thin specimen thickness (about several um), it is possible to include all the ranges where the specimen 14 may exist. With this arrangement, an area where the specimen 14 and any imaging range 802 overlap due to distortion of the specimen 14 or the like is formed. Therefore, it is possible to include all ranges in which the specimen 14 may exist.

FIG. 4C is a diagram schematically illustrating the distribution of the focus evaluation index on the line of the a-a ′ cross section (the right end of the imaging range) in FIG. In FIG. 4B, the distribution calculation unit 131 inputs imaging data obtained by the main imaging apparatus 200 in the imaging range 802 having eight images, and the distribution calculation unit 131 performs the Z direction with respect to the imaging data. Then, the interpolation evaluation index distribution is calculated by interpolation (S107). As the focus evaluation index, the contrast or brightness of the image can be used. Then, the calculation unit 130 sets the Z stack range so as to include the entire specimen presence range R (S108). The specimen presence range R is a width R in the Z direction of a focus evaluation index that is equal to or greater than a predetermined threshold Th set in advance. Further, since the existence range R of the specimen 14 in the Z direction can be regarded as the thickness of the specimen 14, the range R can be determined as the specimen thickness. By this imaging process, a multilayer image of the specimen 14 can be acquired without excess or deficiency.

FIG. 6 is a flowchart illustrating Z search imaging in the first embodiment. That is, it is a flow showing a subroutine of step S106. Hereinafter, the Z search imaging will be described with reference to FIG. As described above, NO is selected in step S105 of FIG. 2, and the flow is started by moving to step S106. In step S301, first, the interval d is set to about the thickness of the specimen 14. In step S302, the stage 220 is moved in the Z direction so that the vicinity of the lower end of the cover glass, which is the first imaging layer (the layer including the imaging range 802 closest to the lower end of the cover glass in FIG. 4B), can be imaged. In step S303, an image is captured. In step S304, it is determined whether the imaging layer (the layer including the imaging range 802 farthest from the lower end of the cover glass in FIG. 4B) has reached the upper end of the last slide glass. In step S305, the stage is moved stepwise in the Z direction by an interval d so that the next imaging layer can be imaged. Thereafter, steps S303 to S305 are repeated, and the process proceeds to YES in step S304 when the imaging layer reaches the upper end of the last slide glass, and the flow, that is, the Z search imaging process ends. Note that the Z step moving direction, that is, the imaging start Z position in step S302 and the imaging end Z position in step S304 need not be in this order.

(Continuous multilayer imaging in the Z direction)
FIG. 4D is an enlarged view of the alternate long and short dash line region 902 in the cross-sectional image of the slide 10 shown in FIG. 4A, and also shows the method of Z stack imaging (S109) in the main imaging process. The difference from the Z search imaging (S106, FIG. 4B) is that the imaging range 802 is arranged without a gap in the Z stack range set by the calculation unit 130 in step S108. The interval between the imaging ranges 802 at this time, that is, the interval of step movement in the Z direction of the imaging system is equal to or smaller than the depth of field.
FIG. 5 is a flowchart of Z stack imaging in the first embodiment, and specifically shows the contents of the subroutine of step S109. First, in step S201, the imaging interval in the Z direction, that is, the interval between the imaging ranges 802 is set equal to the depth of field of the imaging system. In step S202, the stage 220 is moved in the Z direction so that the uppermost layer of the Z stack imaging that is the first imaging layer can be imaged. In step S203, the current imaging layer is imaged. In step S204, it is determined whether or not the lowest layer of the Z stack range that is the last imaging layer has been reached. When it is not the last imaging layer, it progresses to NO and transfers to step S205. In step S205, the stage is moved stepwise in the Z direction by the interval determined in step S201 so that the next imaging layer can be imaged. Thereafter, S203 to S205 are repeated, and the process proceeds to YES in step S204 when the imaging layer reaches the bottom layer of the last Z stack range, and the Z stack imaging flow ends. The direction of Z step movement, that is, step S202.
The imaging start Z position in step S204 and the imaging end Z position in step S204 need not be in this order.

(Setting of Z direction imaging range)
FIG. 7 is a diagram illustrating a method of setting the Z stack range in the first embodiment. FIG. 7A shows a state in which Z stack imaging (S109) and calculation of a focus evaluation index (S110) are completed in a certain small section 801a. That is, a plurality of layers (eight layers in FIG. 7) of image data including the specimen 14 is acquired by the imaging apparatus 200 by a plurality of imaging ranges 802 being continuous in the Z direction without gaps. Thereafter, based on this, the distribution calculation unit 131 calculates a three-dimensional distribution of the focus evaluation index, and determines an area indicating the value of the focus evaluation index equal to or greater than a predetermined threshold as a specimen presence range. In FIG. 7A, thick solid line portions 701 and 702 respectively indicate the upper end surface 701 of the sample presence range and the lower end surface 702 of the sample presence range determined as described above. Although both the surfaces 701 and 702 are curved surfaces in the three-dimensional space as described above, the surfaces 701 and 702 are depicted as lines in the drawing because FIG. 7 is a cross-sectional view in the XZ plane perpendicular to the Y axis. . In step S112, the specimen estimation unit 132 performs an extrapolation operation on the three-dimensional distribution of the focus evaluation index in the small section 801 that has already been captured, and the focus evaluation in the small section 801b that is adjacent to the already captured small section 801a. Estimate the three-dimensional distribution of indices. Then, the setting unit 133 sets the Z stack range of the small section 801b to be imaged next based on the estimated three-dimensional distribution.

FIG. 7B shows a state in which the Z stack range of the small section 801b to be imaged next is set. Thick dotted line portions 751 and 752 in FIG. 7B indicate the upper end surface 751 and the lower end surface 752 of the sample presence range estimated as described above, respectively. Although the surfaces 751 and 752 are actually curved surfaces in the three-dimensional space as described above, FIG. 7 shows a cross-sectional view obtained by virtually cutting the specimen existing range and image data on the XZ plane. Then, the surfaces 751 and 752 are drawn as lines. A region 851 indicated by a thin dotted line in the drawing indicates a Z stack range to be imaged next, and includes all the ranges in which the specimen sandwiched between the surface 751 and the surface 752 is estimated to exist. In this embodiment, as a method for estimating the existence range of the specimen in the eight small sections 801b adjacent to the periphery of the small section 801a from the three-dimensional distribution of the focus evaluation index such as the contrast value in the small section 801a already captured. Use extrapolation.

Specific examples of the extrapolation calculation include the following three. The first method is a method of extrapolating the three-dimensional distribution itself of the focus evaluation index of the small section 801a. This is a method of performing extrapolation calculation for all values of the focus evaluation index forming the three-dimensional distribution. In the second method, the upper end surface 701 of the small section 801a is detected, and the upper end surface 701 of the small section 801b is estimated by extrapolating the upper end surface 701. Similarly, the lower end surface 702 of the small section 801a is detected, and the lower end surface 702 of the small section 801b is estimated by extrapolating the lower end surface 702. In the small section 801b, it is considered that the specimen exists between the estimated upper end surface 751 and lower end surface 752. In the third method, first, a focus evaluation index exceeding a predetermined value representing the presence of the specimen is detected from the three-dimensional distribution of the focus evaluation index in the small section 801a. Then, the shape of the distribution of the focus evaluation index exceeding a predetermined value is detected. And it is a method of estimating the three-dimensional distribution of the focus evaluation index of the small section 801b by extrapolating the shape. The extrapolation calculation is a known technique, and various methods are known. Since the shape of the specimen 14 is not limited to a simple plate shape and may have a complicated shape, an estimation error may increase in linear extrapolation. Therefore, it is desirable to perform extrapolation using a higher-order spline function or the like as much as possible.

In this embodiment, an extrapolation operation is performed on the three-dimensional distribution of the focus evaluation index in one small section 801a that has already been captured, so that the small section 801b adjacent to the periphery of the small section 801a that has been captured immediately before is captured. A method for estimating the three-dimensional distribution of the in-focus evaluation index is shown. However, in order to improve accuracy, it is also desirable to perform extrapolation on the data of the three-dimensional distribution of the focus evaluation index in two or more small sections 801 that have already been imaged. The greater the amount of three-dimensional distribution data used for extrapolation calculation, the better the extrapolation accuracy can be expected. In addition, among the small sections 801 adjacent to one small section 801 that has already been imaged, it is not necessary to perform the calculation again for the small section 801 that has already been captured and the three-dimensional distribution of the focus evaluation index is calculated. . Thereby, calculation time can be shortened.

As described above, the three-dimensional distribution of the focus evaluation index in one or more small sections 801 that have already been imaged is acquired. Extrapolation is performed on the acquired three-dimensional distribution to estimate the three-dimensional distribution of the focus evaluation index of the small section 801 adjacent to the small section 801. Then, based on the estimated three-dimensional distribution, a small section 801 and a Z stack range to be imaged next are set. By doing so, a multilayer image of the specimen 14 can be acquired without adding a special focusing device mechanism such as a phase difference AF device. Furthermore, the Z search imaging process (steps S106 to S108 in FIG. 2) can be omitted except for the small section 801 that performs imaging for the first time, and the throughput of the apparatus can be improved.

<Example 2>
FIG. 8 is a flowchart showing the second embodiment of the image processing apparatus of the present invention. The second embodiment of the image processing apparatus of the present invention is the same as the first embodiment except that a process for verifying and adding a layer to be imaged is added to a part of the main imaging process. Detailed description of the portion is omitted. FIG. 8 is an excerpt of the sections of steps S109 to S111 in FIG. 2, and a portion specific to this embodiment is added with a thick line and a thick frame. Similar to the first embodiment, in step S109, the imaging apparatus 200 performs Z stack imaging of the small section 801, and acquires a plurality of layers of image data (Z stack image data). In step S110, the distribution calculation unit 131 calculates a three-dimensional distribution of the focus evaluation index from the Z stack image data. In step S401, the distribution calculation unit 131 detects the actual existence range of the specimen 14 in the small section 801 based on the three-dimensional distribution calculated in step S110. Then, it is checked whether or not the Z stack range imaged in step S109 includes the actual existence range of the specimen 14. If the Z stack range includes the actual existence range of the specimen 14 (S401; Yes), since the image of the specimen 14 existing in the small section 801 can be acquired without omission, the final small section determination (S111). Move on to processing. On the other hand, if the actual existence range of the specimen 14 is out of the Z stack range, that is, if the Z stack range is narrower than the actual existence range of the specimen 14 or deviates, the Z stack image data captured in step S109. (Step S401; No). In that case, in step S402, the setting unit 133 sets the focal position of a layer (additional imaging target layer) to be additionally imaged to compensate for the shortage, and the process proceeds to step S109 again. In step S109, the additional imaging target layer is imaged by the imaging apparatus 200, and the additional imaging data is added to the original Z stack image data. If there is still a shortage in the added Z stack image data, the processes of steps S401, S402, and S109 are repeated.

With reference to Fig.9 (a), the example which requires the above additional imaging is demonstrated. Based on the existence range of the specimen 14 detected from the Z stack image data of the small section 801a, when the existence range in the Z direction of the specimen 14 in the adjacent small section 801b is estimated, the estimation result and the existence of the actual specimen 14 Deviations (errors) may occur between the range. For example, in FIG. 9A, the specimen existence range (upper end face 761, lower end face 752) in the small section 801b is estimated based on the specimen existence range (upper end face 701, lower end face 702) detected in the small section 801a. ing. Here, the estimated upper end surface 761 is slightly lower than the actual upper end surface (thin solid line) of the specimen 14. Therefore, when the Z stack range 852a of the small section 801b is set based on the estimated existence range (range from the upper end surface 761 to the lower end surface 752), the upper right end portion of the specimen 14 is out of the Z stack range 852a.

FIG. 9B schematically shows the result of Z stack imaging for the small section 801b and the state of additional imaging. First, in the imaging apparatus 200, the Z stack range 852b is set based on the estimated specimen presence range, and Z stack imaging of the small section 801b is performed (S109). Thereafter, the distribution calculation unit 131 detects the actual existence range of the specimen 14 in the small section 801b based on the imaging data (S110, S401). Reference numerals 711 and 712 respectively indicate the upper end surface and the lower end surface of the detected specimen presence range. The distribution calculation unit 131 compares the detected specimen presence range with the Z stack range 852b, and checks whether the Z stack range 852b includes the specimen presence range based on the comparison result (S401). In this example, since the upper end surface 711 of the specimen presence range exceeds the upper end of the Z stack range 852b, it is determined that the imaging data is insufficient (S401; No), and the additional imaging target layer 853 is added. Thereafter, main imaging of the additional imaging target layer 853 is performed and added to the Z stack image data, whereby Z stack image data including the specimen 14 in the small section 801b can be acquired.

Note that any method may be used for determining whether or not a layer to be additionally imaged is necessary (S401). For example, there are the following two. The first method is a method of comparing the area of the specimen region in the uppermost layer (or lowermost layer) of the Z stack range. Based on the three-dimensional distribution of the focus evaluation index actually obtained by actual imaging in the small section 801b, the number N1 of pixels in which the focus evaluation index exceeds a predetermined value in the uppermost layer image is acquired. The number N1 corresponds to the area of the specimen region in the uppermost layer. Similarly, from the three-dimensional distribution of the focus evaluation index of the small section 801b obtained by estimation, the number M1 of pixels in which the focus evaluation index exceeds a predetermined value in the uppermost image is acquired. Then, the number N1 and the number M2 are compared. As a result of the comparison, when it is determined that the number N1 is larger than the number M2 (that is, the area of the actual specimen region is larger), an additional imaging target layer 853 is further formed on the uppermost layer that is actually imaged. Is a method of setting. As a result, the specimen 14 on the side close to the imaging optical system in the Z stack range can be imaged without omission. The same determination can be made on the side farthest from the imaging optical system in the Z stack range (lowermost layer side). The second method is a method of comparing the heights of the upper end surface (or lower end surface) of the specimen presence range. First, the height of the upper end surface 711 actually obtained by actual imaging in the small section 801b is compared with the height of the upper end surface 761 obtained by estimation. As a result of the comparison, when it is determined that the upper end surface 711 is higher, the additional imaging target layer 853 is further set on the uppermost layer that has been actually imaged. As a result, the specimen 14 on the side close to the imaging optical system in the Z stack range can be imaged without omission. In addition, about the 2nd method, you may further employ | adopt the following structures. That is, the lower end surface 752 and the lower end surface 712 are compared. Then, when it is determined that the height of the lower end surface 752 is higher than the height of the lower end surface 712, an additional imaging target layer is further set below the lowest imaged image. By doing so, the additional imaging target layer can be provided on the side far from the imaging optical system. As a result, the specimen 14 on the side close to and far from the imaging optical system in the Z stack range can be imaged without omission.

Note that the user can set an operation mode for determining whether to enable the processing related to the additional imaging to the apparatus 1 via a PC workstation or the like, which is an external apparatus (not shown). When enabled, priority is given to the accuracy with respect to the acquisition range of imaging data, and when disabled, throughput is given priority.

As described above, it has the process of determining the lack of Z stack imaging data and performing additional imaging. By doing so, even when a significant error occurs in the estimation of the three-dimensional distribution of the focus evaluation index by extrapolation, a multilayer image (Z-stack image) of the specimen 14 is acquired without a shortage with a minimum downtime. be able to.

<Example 3>
FIG. 10 is a perspective view showing Embodiment 3 of the image acquisition apparatus of the present invention. Constituent elements common to those in the first embodiment are denoted by the same reference numerals and description thereof is omitted. The calculation amount of the calculation process of the three-dimensional distribution of the focus evaluation index based on the image data acquired by the Z stack imaging in the distribution calculation unit 131 and the extrapolation calculation process of the three-dimensional distribution in the specimen estimation unit 132 are as follows: It also depends on the number of pixels of the element 240. Therefore, the amount of calculation increases when data for all pixels of image data acquired by Z stack imaging is used. In the present embodiment, data for all pixels is not used in each of the above processes, but only data of a plurality of points or regions extracted at a predetermined interval is used. That is, in the first to third embodiments, the contrast value or the luminance value is calculated for all the pixels, but in the third embodiment, the above calculation is performed for some of the pixels, not for all the pixels.

FIG. 10A shows the relationship between the small section 801 and the sample 14. By dividing a small section 801 having a thin solid line frame into six small areas by thin dotted lines, there are 12 grid points including the boundary line with the adjacent small section 801. FIG. 10B is a diagram showing a three-dimensional plot of the specimen presence range. That is, the thick solid line group is a three-dimensional plot of the specimen presence range determined based on a plurality of one-dimensional distributions described later. That is, the one-dimensional distribution exists on a straight line that passes through the lattice points and is parallel to the Z axis, among the image data acquired by Z stack imaging in the lower left small section 801 that has already been acquired (corresponding to the image). It is a plurality of one-dimensional distribution of the focus evaluation index calculated using the data to be. The thick solid line portion 701 is the upper end surface 701 of the specimen presence range, and the thick solid line portion 702 is the lower end surface 702. The thick dotted line group in FIG. 10B is obtained by extrapolating a plurality of one-dimensional distributions of the above-described focus evaluation indexes and estimating the same distribution on the lattice points of the surrounding small sections 801. It shows the specimen presence range. For simplification, the range shown in the figure is limited, and two sub-sections 801 that have already been imaged and a sub-section 801 that is set as a region to be imaged next among the eight sub-sections adjacent to the surrounding area. Only small parcels. The specimen existing range upper end surface 751 in which the thick dotted line portion 751 is estimated and the thick dotted line portion 752 are the lower end surface 752. Here, the data actually exists only on a straight line including the black dots in the figure and passing through the lattice points and parallel to the Z axis, and the thick line group is linear between the black dots to represent the surface for convenience of drawing. Interpolated. The Z stack range is set so that the entire sample existence range in the right small section 801 estimated in this way is included in the imaging range. In the present embodiment, the small section 801 is divided into six for the sake of simplicity. However, the present invention is not limited to this, and the calculation accuracy increases as the number of grid points increases.

As described above, the amount of calculation is reduced by using only the data of the points or regions extracted at predetermined intervals from the image data acquired by the Z stack imaging used for the calculation processing of the three-dimensional distribution of the focus evaluation index. can do.

<Example 4>
FIG. 11 is a perspective view showing Embodiment 4 of the image acquisition apparatus of the present invention. Constituent elements common to those in the first embodiment are denoted by the same reference numerals and description thereof is omitted. The present embodiment relates to an imaging method for efficiently obtaining a single-layer image with the best focus (at the most focused position) within the specimen presence range. Note that this imaging method does not perform Z stack imaging in all of the small sections 801. FIG. 11A shows the Z-stack imaging described so far. Z stack imaging is performed in two vertical and horizontal small sections 801, and a focus evaluation index is calculated. As a result of the calculation, the imaged range 802 group to which the mesh is applied is the most in-focus position. Based on these, extrapolation is performed, and a Z stack range 851 to be imaged next is set. Also in the present embodiment, the same imaging as before is performed on several tiles after the start of imaging as shown in FIG. This is because the first tile requires Z search imaging. In addition, when only single-layer imaging is used immediately after the start of imaging, the estimation accuracy by extrapolation calculation of the most in-focus position is reduced in principle.

In FIG. 11B, the most in-focus position of the next sub-section 801 to be imaged is estimated by extrapolation from the distribution of the most in-focus position in the plurality of sub-sections 801 that have already been captured in a single layer, and the next imaging Range 87
The state set as 1 is schematically shown. In FIG. 11B, the arrangement of the small sections 801, the most in-focus positions, and the like are aligned with those in FIG. FIG. 11B shows a case where the most in-focus position in the small section 801 to be imaged next is estimated from the most in-focus position of the four small sections 801. The estimation accuracy of the most in-focus position to be imaged next increases in principle as the number of small sections 801 as an estimation source increases. For this reason, the estimation accuracy of imaging becomes higher from the initial one to the later one. Therefore, in the initial imaging with a small number of small sections 801 serving as the estimation source, imaging of a plurality of layers is performed as shown in FIG. 11A in order to ensure estimation accuracy, and the three-dimensional distribution of the focus evaluation index is obtained. It is desirable to be calculated. Thereby, it is possible to sufficiently ensure the estimation accuracy in the initial imaging. In addition, the system configuration of the device, such as when to switch to single-layer imaging at the point in the imaging process, whether to switch at once, or gradually, depends on the throughput and accuracy required for the system, and it is an experimentally optimal design value You may comprise by calculating | requiring.

As described above, the most in-focus position in the area to be imaged next is obtained by extrapolation from the distribution of the most in-focus position in the already imaged area, and is set as the imaging position. Thereby, a single-layer image of the specimen is efficiently acquired. In addition, the imaging method of a present Example may be combined with the various imaging methods in other this invention, and is not limited at all. For example, the focus evaluation index is calculated from the imaging data of the next imaging range 871. Then, it is determined whether or not the calculated focus evaluation index is out of the most focused position. If it is determined that the image is out of position, the adjacent layer is additionally imaged, and the imaging data at the most in-focus position is updated. In this case, the plurality of layers of focus evaluation index distributions that have been additionally imaged are used to estimate the most in-focus position of the area that is subsequently imaged. Thereby, further improvement in imaging accuracy can be realized. Further, the XY coordinates of the imaging region 802 corresponding to the most in-focus position may be the center of the small section 801, or may be the coordinates of the point with the highest focus evaluation index in the small section 801. The latter improves the estimation accuracy of the next imaging range 871.

<Example 5>
The object of the present invention is achieved by the following. That is, a storage medium (or recording medium) storing software program codes for realizing the functions of the above-described embodiments is supplied to a system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, an operating system (OS) or the like running on the computer performs part or all of the actual processing based on the instruction of the program code. The case where the functions of the above-described embodiments are realized by the processing is also included in the scope of the present invention. Furthermore, it is assumed that the program code read from the storage medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU of the function expansion card or function expansion unit performs part or all of the actual processing, and the function of the above-described embodiment is realized by the processing. It is included in the scope of the present invention. When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the flowcharts described above.

<Other examples>
A person skilled in the art can easily correspond to constructing a new system by appropriately combining various technologies in the above-described embodiments. Therefore, a system based on such various combinations is also within the scope of the present invention. Belongs to. Further, various implementations of the present invention are not limited to the embodiments described above.

10 slides, 120 image processing unit, 130 calculation unit, 200 imaging device, 220 stage

Claims (13)

An image acquisition device that divides a sample into a plurality of regions and captures images of a plurality of layers or single layers having different focal positions in the optical axis direction for each region,
Imaging means for imaging the sample;
Based on the single-layer or multi-layer image of the first region of the sample imaged by the imaging means, the existence range or the maximum in the optical axis direction of the specimen included in the second region of the sample different from the first region An image acquisition apparatus comprising: a specimen information acquisition unit that estimates a focus position and sets an imaging range of the second region in the optical axis direction based on the estimation result. The specimen information acquisition means obtains the specimen existing range, the most in-focus position, or the distribution of the most in-focus position in the first area from the single-layer or multiple-layer images of the first area, and the first area The existence range or the most in-focus position of the specimen included in the second region is estimated based on the existence range, the most in-focus position, or the distribution of the most in-focus position. 2. The image acquisition device according to 1. The image acquisition apparatus according to claim 2, wherein the existence range of the specimen in the first region is a three-dimensional distribution of a focus evaluation index obtained from images of a plurality of layers in the first region. The specimen information acquisition unit is configured to detect the presence of an actual specimen in the second region from a single-layer or multi-layer image of the second region captured by the imaging unit in accordance with the set imaging range of the second region. A range or a focus evaluation index is obtained, and it is determined whether or not the set imaging range of the second region includes the actual presence range or the most in-focus position of the sample in the second region. 4. The image acquisition device according to 3. When the specimen information acquisition means determines that the set imaging range of the second area does not include the actual specimen existing range or the most in-focus position of the second area, The image acquisition apparatus according to claim 4, wherein an additional layer to be imaged is set. The image acquisition apparatus according to claim 3, wherein the focus evaluation index is a contrast value or a luminance value. The image acquisition apparatus according to claim 3, wherein the first area and the second area are adjacent to each other. 8. The specimen information acquisition unit acquires a three-dimensional distribution of the focus evaluation index or a distribution of the most in-focus positions from a part of pixels in a single layer image or a plurality of layers of the first region. The image acquisition device according to any one of the above. 9. The specimen information acquisition unit calculates a presence range in the optical axis direction of the specimen included in the second area by estimating a three-dimensional distribution of a focus evaluation index in the second area. The image acquisition device according to any one of the above. The image according to claim 9, wherein the specimen information acquisition unit estimates the three-dimensional distribution of the focus evaluation index in the second region by extrapolating the three-dimensional distribution of the focus evaluation index in the first region. Acquisition device. The said sample information acquisition means estimates the most in-focus position in the said 2nd area | region by extrapolating the distribution of the most in-focus position in the said 1st area | region. Image acquisition device. A method for controlling an image acquisition device that divides a sample into a plurality of regions and captures images of a plurality of layers or single layers having different focal positions in the optical axis direction for each region,
Capturing images of multiple layers of the first region of the sample;
The existence range or the most in-focus position of the specimen included in the second region of the sample different from the first region based on the single-layer or multiple-layer image of the first region of the sample that has been imaged Estimating
And a step of setting an imaging range in the optical axis direction of the second region based on the estimation result. A program for causing a computer to execute each step of the control method of the image acquisition device according to claim 12.

Images