JP2016125913A - Image acquisition device and control method of image acquisition device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology that efficiently implements a recovery when a failure occurs in a part of tile images in an image acquisition device dividing a subject into a plurality of areas to photograph the subject, and allows a subject image of a wide field of view and a high image quality to be acquired.SOLUTION: An image acquisition device is configured to execute: a first action in which a picture is taken by imaging means a plurality of times as changing a position of a field of view with respect to a subject, and thereby acquires a plurality of tile images corresponding to a plurality of areas of the subject; a second action in which the plurality of tile images acquired by the first action is inspected, and a determination is made whether a necessary re-shooting image necessary for re-shooting is present; and a third action in which, when it is determined in the second action that the necessary re-shooting image is present, a field of view is aligned with the area corresponding to the necessary re-shooting image, and re-shooting is implemented by the imaging means, and thereby the tile image instead of the necessary re-shooting image is acquired.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a technique for imaging a subject by dividing it into a plurality of regions.

  In recent years, a pathological diagnosis support system called a virtual slide system has attracted attention in the medical field. A virtual slide system is a system that digitizes a subject image by imaging a subject (such as a slide with a pathological specimen fixed) with a high-magnification imaging device called a virtual slide scanner, and allows the image to be observed on a display. is there. By using this system, it is possible to provide a new diagnostic method such as remote diagnosis and automatic diagnosis that could not be realized with a conventional optical microscope. For pathological diagnosis, a high-definition image of the entire observation region of a specimen (stained biological tissue or the like) is required. Therefore, the virtual slide system is required to acquire a high-quality image with a wide field of view.

  A typical slide (also called a preparation) has a maximum size of 26 mm × 76 mm. The range in which the sample exists is a part of the range, and an identification label (for example, a barcode) is pasted on the slide with a length of about 15 mm to 20 mm. Is considered to be about 26 mm × 60 mm at the maximum. On the other hand, the visual field size of the objective lens of the virtual slide scanner is approximately 1 mm × 1 mm, which is extremely small compared to the slide. Therefore, in order to obtain an image of the entire specimen, it is necessary to perform imaging while moving either the stage holding the slide or the lens barrel optical system (or both) and changing the position (imaging position) of the visual field relative to the specimen. is there. As a specific method, a method in which an imaging range is continuously scanned by a line sensor, a method in which the imaging range is divided into a plurality of rectangular unit areas (called tiles), and each tile is sequentially imaged by an area sensor, (See Patent Document 1). The former is called a scanning method, and the latter is called a tiling method or a step & repeat method. The tiling method has an advantage that a high-quality and high-definition image can be obtained as compared with the scan method.

JP2013-83925A JP 2010-147691 JP 2013-050594 A

  In the tiling method, if it is possible to capture a 1 mm × 1 mm tile with a single image capture, it is necessary to divide the image into at least 26 × 60 tiles in order to obtain an image in the range of 26 mm × 60 mm. is there. The total number of tile images is 1560. At this time, it is only necessary to obtain good images for all the tiles, but one or a plurality of defective images may be generated for some reason. A defective image refers to an image in which at least one of the focus position, exposure, and magnification is not appropriate.

Several factors can be considered for the occurrence of such a defective image. Factors in the device environment (heat, vibration, electrical noise, etc.), program defects (focal point and exposure judgment errors, human setting errors, etc.), and device abnormalities (position control errors, mechanical errors, etc.) It is. Alternatively, a defect may occur due to the subject side, such as unevenness of the specimen, variation in thickness, and undulation of the cover glass covering the specimen. Image defects due to these factors are mostly concentrated in loss of focus, exposure failure, and change in magnification.

  When even one defective image is included in the tile image group acquired from a certain slide, it is necessary to re-image the slide. Since slides used for pathological examination (pathological diagnosis) do not know where important parts such as lesions exist in the specimen, even if some tiles have poor image quality or defectiveness, accurate examination and diagnosis are hindered. Because there is.

  For this reason, conventionally, after acquiring all the tile images, an image obtained by connecting the tile images is displayed, and an operator determines whether there is a defect and whether re-imaging is necessary (Patent Document 2). However, since this conventional method relies on the visual inspection and judgment of the operator, the work load is large and a long time is required for the inspection. In addition, there is a possibility that variations in judgments by operators and human errors (missing defects) may occur.

  The virtual slide scanner is expected to perform an imaging process in a walk-away style (a style in which an operator leaves the apparatus for a while and returns after the completion of all processes when a slide is set on the apparatus). In the pathological examination site, it is also assumed that many slides are batch processed continuously. From this point of view, it is desirable to eliminate operator intervention as much as possible.

  Patent Document 3 discloses an invention in which, when an unintended object appears in a panoramic image, the part is re-photographed after confirming that the object has been removed. The present invention may have a similar problem to the present invention in that it is a recovery method when an unintended defect is included when generating a large image from a plurality of images. However, the invention of Patent Document 3 is different from the present invention in the technical field, and the solution adopted in Patent Document 3 cannot be diverted to a virtual slide scanner.

  The present invention has been made in view of the above circumstances, and in an image acquisition device that captures an image of a subject divided into a plurality of regions, recovery is efficiently performed when a defect occurs in some tile images, and a wide field of view is achieved. An object of the present invention is to provide a technique capable of acquiring a high-quality subject image.

  According to a first aspect of the present invention, a stage for holding a subject, an imaging unit having a field of view smaller than the subject, a control unit for controlling the stage and the imaging unit, and an image obtained by the imaging unit are input. And the control means performs imaging by the imaging means a plurality of times while changing the position of the visual field relative to the subject by moving at least one of the stage and the imaging means. A first operation for acquiring a plurality of tile images corresponding to a plurality of regions of the subject, and the image processing means inspects the plurality of tile images acquired in the first operation; In the second operation for determining whether or not there is a re-captured image that needs to be retaken, and when it is determined that there is a re-captured image in the second operation, the control means The tile image instead of the re-captured image is obtained by moving at least one of the image capturing unit and the image capturing unit to align the field of view with the area corresponding to the re-captured image and performing re-imaging by the imaging unit. An image acquisition apparatus characterized by executing a third operation to acquire is provided.

According to a second aspect of the present invention, there is provided a control method for an image acquisition apparatus having a stage for holding a subject and an imaging unit having a field of view smaller than the subject, wherein at least one of the stage and the imaging unit is moved. A plurality of tile images corresponding to a plurality of regions of the subject are obtained by performing imaging by the imaging means a plurality of times while changing the position of the field of view with respect to the subject, and The plurality of tile images acquired in the first step are inspected, and it is determined whether or not there is a re-captured image that needs to be retaken. A second step and a re-capture required in the second step If it is determined that there is an image, move at least one of the stage and the imaging unit to align the field of view with an area corresponding to the re-captured image, and That by performing re-imaging, to obtain a replacement tile images of the main re-captured image, a control method of the image acquisition apparatus characterized by comprising a third step, the.

  According to a third aspect of the present invention, there is provided a program for causing an image acquisition device to execute each step of the control method for the image acquisition device according to the second aspect.

  According to the present invention, in an image acquisition device that captures an image of a subject divided into a plurality of regions, recovery is efficiently performed when a defect occurs in some tile images, and a wide-field and high-quality subject image is acquired. can do.

The figure which shows the structure of a virtual slide scanner typically. The flowchart which shows operation | movement of a virtual slide scanner. The figure explaining the tiling photography of a slide. The flowchart which shows the process of tiling photography. An example of the graph which plotted the evaluation value c by Tenengrad function. The flowchart which shows the process of an image determination part. 10 is a flowchart illustrating re-imaging processing according to the second and third embodiments. The flowchart which shows the process of the re-imaging of 4th Embodiment.

  The present invention relates to an image acquisition device having a function of capturing an image of a subject divided into a plurality of unit areas (tiles) and generating an entire image of the subject by connecting images corresponding to the tiles (tile images). . In the following embodiments, an example in which the present invention is applied to a virtual slide scanner (digital microscope) that acquires a digital image of a slide (also referred to as a preparation) of a pathological specimen will be described.

  As described above, when capturing a plurality of tiles, an unintended defective image may be acquired for some tiles due to an accidental reason. Therefore, in the image acquisition apparatus of the present embodiment, tile image acquisition (first operation), determination of a defective image (re-captured image required) (second operation), and re-imaging (third operation) are as follows. Follow the steps in In the first operation, by moving the stage and / or the imaging unit that holds the subject, the imaging unit performs imaging a plurality of times while changing the position of the field of view of the imaging unit with respect to the subject, and acquires a plurality of tile images. In the second operation, a plurality of tile images acquired in the first operation are inspected by image processing, and it is automatically determined whether or not there is a defective image (re-captured image that needs to be retaken). . When it is determined that there is a re-captured image, as a third operation, re-imaging is performed by matching the field of view with the position of the re-captured image, and a tile image instead of the re-captured image is acquired. The image acquisition device automatically executes the first to third series of operations, thereby efficiently recovering when a defect occurs in a part of the tile image, and has a wide-field and high-quality subject image. Can be acquired at high speed.

Here, in the first operation, the second operation, and the third operation, the slide that is the subject is held on the stage (that is, the slide on the stage is replaced after the first operation). Not) In other words, when continuously processing a plurality of slides, after the first to third operations for a certain slide are completed, the slide on the stage is replaced, and the first to third operations for the next slide are executed. It is good to take the sequence. The slide is usually fixed to the stage by abutting the corner of the slide on the positioning portion on the stage, but the positioning accuracy in that case is not so high as 10 to 50 microns. Therefore, if there is a slide change between the first operation and the third operation, the displacement between the tile image obtained by the first operation and the tile image obtained by the third operation is large, and the tile This is because there is a possibility that the images cannot be joined together.

  When re-imaging is performed in the third operation, re-shooting may be performed under the same imaging conditions as in the first operation, or re-shooting may be performed under imaging conditions different from those in the first operation. The first embodiment described below is an example in which re-imaging is performed under the same imaging conditions, the second embodiment is an example in which re-imaging is performed by changing the exposure, and the third embodiment is a method in which the focal position in the optical axis direction is changed. This is an example of performing re-imaging.

<First Embodiment>
(Configuration of virtual slide scanner)
FIG. 1 schematically shows a configuration of a virtual slide scanner according to an embodiment of an image acquisition apparatus of the present invention.

  A virtual slide scanner is an apparatus that magnifies an image of a slide placed on a stage by an optical system, converts the image into a digital image by an imaging device, performs image processing by a computer, and stores the obtained image data. The virtual slide scanner of this embodiment includes a stage 1, an optical system 3, an image sensor 4, a workstation (computer) 5, an image server (storage device) 6, a control unit 7, and an illumination unit 8. The virtual slide scanner may include a slide storage unit that stores a plurality of slides 2 and a loading device that transfers the slide 2 from the slide storage unit to the stage 1.

  The stage 1 is a member that holds a slide 2 that is a subject. In the present embodiment, the slide 2 is arranged and fixed on the upper surface of the stage 1. The stage 1 can move in three axes of xyz (x and y are directions parallel to the stage surface, and z is a direction perpendicular to the stage surface and parallel to the optical axis).

  The optical system 3 is an imaging optical system in which an objective lens, an imaging lens, a diaphragm, and the like are arranged in a lens barrel. The field of view of the optical system 3 is about 1 mm × 1 mm, and the depth of field is about 0.5 microns. When an RGB three-plate image sensor 4 is used, a beam splitter such as a dichroic prism is provided in the optical system 3.

  The imaging sensor 4 is an area sensor (two-dimensional image sensor) and is configured by a CCD or CMOS sensor. When the slide 2 is irradiated by the illumination unit 8, the transmitted light enters the optical system 3. The light passing through the optical system 3 is converted into a digital image by the imaging sensor 4, and the data is input to the workstation 5.

  The workstation 5 is an image processing unit having various functions for handling the obtained image data. For example, the workstation 5 has a function of giving color information to images, a function of stitching images, a function of determining whether a defective image needs re-imaging, a function of compressing an image, and the like. These functions are realized by the CPU reading and executing a program stored in the storage device of the workstation 5. Note that some of the functions of the workstation 5 may be configured by a circuit such as an ASIC or FPGA or an image processing processor. The image data processed by the workstation 5 is stored in the image server 6 or the like.

The control unit 7 is a controller that controls the stage 1, the illumination unit, the optical system 3, and the imaging sensor 4. For example, the control unit 7 controls the xy position of the stage 1 (moving the visual field), controls the z position of the stage 1 (moving the focal position), turns on the illumination light, adjusts the light amount, and exposes the optical system 3 (aperture). And the function of controlling the magnification and the image capture of the image sensor 4. In the present embodiment, the workstation 5 and the control unit 7 are configured as separate devices, but the function of the control unit 7 may be realized by a program of the workstation 5.

(Virtual slide scanner operation)
The operation of the virtual slide scanner of this embodiment will be described along the flowchart of FIG.

  In step S20, the slide 2 to be imaged is placed on the stage 1. An operator may place the slide 2 on the stage 1, or may automatically carry it in from the slide housing portion by a carry-in device.

  In step S <b> 21, the control unit 7 controls the stage 1, the optical system 3, and the imaging sensor 4 to perform tiling photography of the slide 2. Tiling photography is a technique for photographing a subject by dividing it into a plurality of unit areas (tiles). FIG. 3A is a schematic diagram showing the specimen 30 on the slide 2 and tiling photography. As shown in FIG. 3A, the visual field 31 of the optical system 3 is very small compared to the specimen 30, and the entire specimen 30 cannot be imaged at once. Therefore, the range including the specimen 30 (imaging range 32) is divided into a plurality of tiles 33 smaller than the field of view 31 of the optical system 3, and each tile 33 is imaged in order. In FIG. 3A, only the tile 33 in the field of view 31 (the tile 33 at the imaging position) is indicated by a thick line. For convenience of illustration, FIG. 3A shows an example in which the tiles are divided into 32 tiles 33. However, in an actual apparatus, imaging is performed by dividing into several hundred or more tiles.

  FIG. 4 is a detailed flow of tiling photography (step S21) of the first embodiment. First, in step S40, the control unit 7 reads information on the imaging range 21 of the slide 2 to be imaged. The information of the imaging range 21 is given, for example, in the form of a list of xy coordinate values of 32 tiles 33 shown in FIG. Next, in step S41, the control unit 7 reads the imaging conditions (focal position, exposure, magnification, light amount, etc.), the z position of the stage 1, the magnification of the aperture or objective lens of the optical system 3, the light amount of the illumination unit, etc. Set. As the setting values such as the focal position (z position), exposure (aperture), magnification, and light amount, values set in advance in the control unit 7 may be used, or values specified by the operator may be used. In the present embodiment, it is assumed that all tiles 33 are imaged under the same conditions.

  In step S <b> 42, the control unit 7 controls the xy position of the stage 1 and matches the first tile 33 with the field of view 31 (imaging position) of the optical system 3. In the example of FIG. 3A, the upper left tile 33 is selected as the first tile 33. The field of view 31 (imaging position) may be moved not on the stage 1 side but on the imaging system (optical system 3 and imaging sensor 4) side, or both the stage 1 and imaging system. In step S <b> 43, the control unit 7 captures a digital image by the imaging sensor 4 and acquires image data corresponding to the tile 33 in the field of view 31. In order to facilitate alignment of tile images during a joining process described later, an image of an area slightly larger than the tile 33 may be acquired as a tile image. The tile image data is transmitted to the workstation 5 and stored in the storage device together with the attribute information (for example, the xy coordinates of the tile 33 and the imaging condition information) (step S44). When an unimaged tile 33 remains (step S45; NO), the control unit 7 moves the field of view 31 to the position of the next tile 33 (step S42) and performs imaging (step S43). When the imaging of the 32 tiles 33 is completed according to the order of the arrows in FIG. 3B (step S45; YES), the tiling photographing process is terminated and the process proceeds to step S22 in FIG.

In step S22, the workstation 5 inspects each of the 32 tile images and determines the presence or absence of a defective image. For example, when a disturbance (vibration or impact) is applied to the stage 1 or the lens barrel while imaging the same depth (z position) in the specimen 30 at a fixed focal position, the focus is applied to the tile image captured at that time. There may be blurring due to displacement or vibration. Such tile images are not suitable for stitching and should be determined as defective images. In addition, during imaging of the specimen 30 with a constant exposure, there is a possibility that imaging is performed with an exposure different from the set exposure due to stray light entering the lens barrel. If the exposure is different from the surrounding tile image, uneven brightness may appear when the tile images are joined together, or a pseudo contour (artifact) may appear at the boundary between tiles. In addition, when overexposure or underexposure occurs, tone information may be lost in a high luminance region or a low luminance region. In the present embodiment, the tile image captured under the imaging conditions different from those of the other tiles 33 is determined as a defective image. Hereinafter, this function of the workstation 5 is referred to as an “image determination unit”. Details of the processing of the image determination unit will be described later.

  If any tile image is determined to be defective (step S23; YES), the image determination unit sends a re-imaging request for the tile determined to be defective to the control unit 7 (step S24). At this time, the image determination unit refers to the attribute information of the tile image determined to be defective, and passes the xy coordinates of the tile to be re-imaged and the information on the imaging condition causing the failure to the control unit 7. However, when re-imaging is performed under the same imaging condition as in the present embodiment, information on the imaging condition may be omitted. In the example of FIG. 3C, it is assumed that the image of the two tiles 33A indicated by hatching is determined to be defective.

  In step S25, the control unit 7 performs an imaging process on each of the one or more defective tiles 33A that has received the re-imaging request in the same manner as in steps S42 to S43, and acquires a new tile image. In the example of FIG. 3C, re-imaging of the two defective tiles 33A is performed. The re-acquired tile image is sent to the workstation 5 and stored in the storage device instead of the defective image (step S26). Thereafter, the process returns to step S22, and the tile image is inspected again by the image determination unit. By repeating the processes in steps S22 to S26 until there is no defective image, it is possible to acquire good images for all the tiles on the slide 2.

  Thereafter, the workstation 5 joins all the acquired tile images (referred to as stitching processing), generates an entire image of the specimen 23, and stores it in the storage device (step S27). Thus, the image acquisition process for one slide 2 is completed. If another slide 2 is to be processed, the process returns to step S20, the slide 2 on the stage 1 is replaced, and the processes after step S21 are repeated.

  When tiling imaging is performed as in the present embodiment, imaging may fail on some tiles due to accidental reasons, and an unintended defective image may be acquired. If a lesioned part is included in a tile of a defective image, the acquired image is not usable and an appropriate diagnosis cannot be performed. Therefore, in this embodiment, the tile image is inspected by the image determination unit of the workstation 5 to determine the presence or absence of a defective image. Then, the control unit 7 retakes the tile 33 determined to be a defective image, and re-acquires an appropriate image for stitching. As shown in FIG. 2, by repeating this loop until there is no defective image, good tile images can be obtained for all the tiles 33 in the imaging range 32 of the specimen 30.

In the present embodiment, the process of step S21 corresponds to “first operation” and “first step”, and the process of step S22 corresponds to “second operation” and “second step”. The process of S26 corresponds to “third operation” and “third step”. The process of Step S21 (first operation; first step) and the process of Step S22 (second operation; second step) may be performed sequentially or in parallel. For example, step S
Even during the process 21, the process of step S <b> 22 is started when the data necessary for the defective image determination is gathered in the workstation 5, and the tile image acquisition and the defective image determination are processed in parallel. Time can be shortened.

  By the way, when such processing is performed, a sequence is performed in which imaging, defective image determination, and re-imaging (if necessary) are performed for each tile, and when a good tile image is obtained, the process proceeds to processing of the next tile. It can also be taken. However, when such a sequence is adopted, the load on the stage 1 that performs step movement increases, and the overall processing time (throughput) may deteriorate. The stage 1 that performs the step movement is a sequence that can move continuously as fast as possible (if the circumstances of the optical system permit), and can perform imaging while maintaining accuracy. Therefore, as in the present embodiment, the sequence in which all tiles are continuously imaged and then only the tiles in which defects are found is re-imaged has the advantage that the overall throughput can be improved. A sequence in which all tiles are divided into several groups and continuous imaging / defective image determination / re-imaging is performed for each group is also preferable. For example, in the example of FIG. 3A, 32 tiles of 4 rows × 8 columns may be divided into 4 groups for each row or 8 groups for each column. By performing processing in units of groups consisting only of tiles in the same row (y position) or only tiles in the same column (x position), the step movement of stage 1 is made in one direction in the x direction or the y direction. Since it can be limited, throughput and positional accuracy can be improved.

  In the present embodiment, as shown in the flow of FIG. 2, continuous imaging (S21), defective image determination (S22), and re-imaging (S24 to S26) are performed while one slide 2 is held on the stage 1. The slide 2 is replaced after the re-imaging is completed. If the slide 2 is once removed from the stage 1 after continuous imaging, it is very difficult to fix the slide 2 at the same position on the stage 1 at the time of re-imaging, and the xy position of the tile between the continuous imaging and the re-imaging. This is because there is a gap between the two. If there is a displacement of the tiles, it is not possible to re-image only the defective tiles, and it becomes necessary to re-image all the tiles on the slide 2. On the other hand, by adopting the sequence as in this embodiment, it becomes possible to selectively re-image only defective tiles, and the time can be greatly reduced compared to re-imaging all the tiles on the slide 2. Can be planned.

(Processing of image judgment unit)
Next, details of the processing of the image determination unit of the workstation 5 will be described.

  The image determination unit determines that a tile image captured under inappropriate imaging conditions is a defective image. Specifically, the image determination unit determines whether or not the image is a defective image by extracting image features from each of the plurality of tile images and evaluating the continuity of the image features between adjacent tile images. Can do. This is because, if the proper imaging conditions are maintained, the image features should be substantially the same between adjacent tiles or change continuously (smoothly). On the other hand, when the difference or change rate of the image feature between adjacent tiles is clearly large (discontinuous), it can be considered that any tile is imaged under an inappropriate imaging condition.

  As the image feature, any image feature can be used as long as the influence of the difference in imaging conditions appears. For example, when the focal position in the optical axis direction changes, the degree of focusing on the specimen changes. Therefore, image features (focus evaluation value) representing the degree of focus, such as the contrast value of the image, the edge strength, and the strength of the high-frequency component, can be used for determining the focus position defect. When the exposure changes, the brightness (brightness distribution) of the entire image changes. Therefore, image features (exposure evaluation values) that evaluate the appropriateness of exposure, such as the percentage of pixels (out-of-white and black-out) in which gradation information has been lost, and the brightness (brightness) histogram of the image, are used for determining defective exposure. Can be used.

(1) Focus position An example of defect determination of the focus position in the optical axis direction will be described. Here, after first describing a general method for determining a focus based on a contrast value, a description will be given of defect determination of a focal position to which the method is applied.

ここで、Ｓ_ｘ（ｘ，ｙ）^２とＳ_ｙ（ｘ，ｙ）^２はSobel法により積分された合成画素であ
る。 <Focus determination by contrast value>
To determine the in-focus state, a method of evaluating the magnitude of the contrast value of an image is common. This method is based on the idea that a focused image has a higher contrast. Examples of contrast evaluation functions include the Tenenbaum Gradient method, the Vollath-5 method, the Brenner Gradient method, the Auto Correlation method, the Normalized Variance method, the Normalized sum of squared intensity method, the Contrast method, the Entropy method, and the Intensity method. Here, the Tenenbaum Gradient method will be described. The Tenenbaum Gradient method (Tenengrad) is an evaluation function using the Sobel method. The Sobel method is a technique of performing weighted averaging in the center for pixel smoothing, and obtains a weighting matrix called a Sobel filter after differentiation and smoothing. The function of Tenenbaum Gradient method is expressed as follows. Tenengrad evaluation value F _Tenengrad is a focus evaluation value indicating the degree of focus of an image.

Here, S _x (x, y) ² and S _y (x, y) ² are synthesized pixels integrated by the Sobel method.

  This focus evaluation value is used for searching for a focus position in the specimen (a focus position in the optical axis direction that is in focus). For example, the z drive system of the stage 1 of the virtual slide scanner is controlled to acquire an image while changing the focal position every 0.5 microns, for example. Since the thickness of the specimen is about 4 to 8 microns, about 9 to 17 images can be obtained. A focus evaluation value is obtained for each acquired image. When the coordinate value in the z direction (height direction) is taken on the horizontal axis and the focus evaluation value is plotted on the vertical axis, a substantially symmetric convex graph is obtained. The in-focus position can be specified by obtaining the z position at which the in-focus evaluation value is maximized from this graph.

FIG. 5 is an example of a graph in which the evaluation value c by the Tenengrad function is plotted. Here, the outputs from the three RGB imaging systems are plotted separately, but the output from the RGB superimposed image may be evaluated. For example, when attention is paid to R, the peak value is around +0.5 microns. As a result, it can be seen that the most focused image can be acquired by imaging with the focal position in the optical axis direction (z direction) in the vicinity of +0.5 microns from the reference value at this imaging position (xy position). Although the description of the Tenenbaum Gradient method has been described here, it is possible to evaluate the degree of focus and search for the focus position using other functions.

<Failure determination for focal position>
Next, a method in which the image determination unit determines whether the focus position of the tile image is appropriate using the focus evaluation value will be described. FIG. 6 is a flowchart showing the processing of the image determination unit.

First, the image determination unit calculates a focus evaluation value for each of the plurality of tile images (step S60). In this embodiment, a Tenenbaum Gradient evaluation value is used as the focus evaluation value. Next, the image determination unit selects one tile image (referred to as a target tile image) to be subjected to defect determination (step S61), and selects a tile image adjacent to the target tile image (referred to as an adjacent tile image). (Step S62). The number of adjacent tile images may be one, two (left and right or top and bottom), four (top and bottom, left and right), or eight (near eight around the target tile). However, tile images that have already been determined to be defective images are excluded from adjacent tile images. Then, the image determination unit evaluates continuity between the focus evaluation value of the target tile image and the focus evaluation value of the adjacent tile image (step S63).

  The continuity can be evaluated based on whether or not the difference (or rate of change) in the focus evaluation value between the target tile image and the adjacent tile image is greater than a predetermined value (threshold value). When using a plurality of adjacent tile images, an average or maximum value of the difference (or change rate) of the focus evaluation values may be used for the evaluation. The threshold value serving as a judgment criterion depends on the type of specimen or staining, the type of focus evaluation value, the resolution of the imaging system, and the like. For example, considering an HE-stained image generally used for pathological specimens, if there is a decrease in evaluation value of about 20%, it should be determined as a defective image.

  Here, it is preferable that the threshold value can be changed by selecting the mode of the virtual slide scanner or changing the setting. For example, standard, high speed (emphasis on throughput) and high quality (emphasis on image quality) are prepared as imaging modes of the virtual slide scanner. When the standard mode is selected, the threshold is 20%, and when the high speed mode is selected, the threshold is 50. %, If the high quality mode is selected, the threshold is set to 5%. Alternatively, the user can directly specify the threshold value on the setting screen. Further, the focus evaluation method (algorithm, type of function or focus evaluation value, number of pixels used for focus evaluation, range, etc.) may be changed by mode selection or setting change. Thereby, usability can be improved.

  If the difference (or rate of change) in the focus evaluation value is greater than the threshold value in step S63, the image determination unit determines that the target tile image is a defective image (step S64). On the other hand, when the difference (or rate of change) in the focus evaluation value is equal to or smaller than the threshold value, the image determination unit determines that the focal position of the target tile image is appropriate (step S65). By performing the above processing on all tile images, it is possible to extract a defective image (re-captured image) having an incorrect focal position.

(2) Judgment of exposure As an exposure evaluation value for evaluating the appropriateness of exposure, the number or ratio of pixels (out-of-white and black-out) where gradation information is lost, the mode value of the luminance (brightness) histogram of the image, An average value or a median value can be used. If the signal from the imaging sensor 4, that is, the pixel value of the image is, for example, 8 bits, it takes a value from 0 (black) to 255 (white). When exposure is appropriate, pixel values are distributed in the range of 0 to 255. When underexposure, the number of pixels with 0 value (blackout) increases or the luminance distribution approaches 0 overall, resulting in overexposure. The number of pixels with 255 values (out-of-white) increases or the luminance distribution approaches 255 as a whole. Therefore, as in the case of the focus evaluation value, the image determination unit obtains the exposure evaluation value of each tile image, and evaluates the continuity of the exposure evaluation value between the tile image of interest and the adjacent tile image, resulting in a defect. Judgment can be made.

  The continuity can be evaluated based on whether or not the difference (or rate of change) in the exposure evaluation value between the target tile image and the adjacent tile image is greater than a predetermined value (threshold value). When using a plurality of adjacent tile images, the average or maximum value of the difference (or change rate) of the exposure evaluation values may be used for the evaluation. The threshold value serving as a judgment criterion depends on the type of specimen or staining, the type of exposure evaluation value, the resolution of the imaging system, and the like. For example, considering an HE-stained image generally used for pathological specimens, if an evaluation value increases or decreases by about 20%, it should be determined as a defective image. As with the focus evaluation value, it is preferable to be able to change the exposure evaluation method and the threshold value by selecting the mode of the virtual slide scanner and changing the setting.

According to the configuration of the present embodiment described above, tile re-imaging can be performed automatically and efficiently when an in-focus or exposure failure occurs during imaging of some tiles due to accidental reasons. it can. Since the defect determination and the re-imaging are performed while holding the slide 2 on the stage 1, only the tile where the defect has occurred can be re-imaged with sufficient positional accuracy (position reproducibility). Therefore, it is possible to realize a highly reliable virtual slide scanner that can acquire a wide-field and high-quality subject image at high speed.

Second Embodiment
In the second embodiment of the present invention, an example will be described in which re-imaging is performed by changing to an appropriate exposure when a defective image (re-captured image required) with an inappropriate exposure is detected.

In slides used for pathological examination and pathological diagnosis, since the entire specimen is composed of the same cell pattern, it is common to photograph all tiles under a constant exposure condition. However, if the thickness of the sample is uneven or the sample is uneven, the image is brighter (overexposed) or darker (exposed) than other tiles, even though the image was taken under the same exposure conditions. Under) may be obtained. Or even when shooting with automatic exposure (AE) for each tile instead of a fixed exposure condition,
There is also a possibility that the exposure adjustment may fail due to the influence of disturbance (for example, stray light) or a sensor or program error. When such an exposure failure occurs, it is necessary to perform re-imaging under an exposure condition different from that when a defective image is obtained.

  FIG. 7 shows a part of the processing flow of the second embodiment. The flow in FIG. 7 shows the details of the processing in steps S24 and S25 in FIG. Since other processes are the same as those of the first embodiment, detailed description thereof is omitted.

  When a poorly exposed tile is detected (step S23; YES), the workstation 5 estimates an appropriate exposure condition for the tile (step S70). For example, the exposure is lowered when overexposed and the exposure is increased when underexposed. At this time, the width for increasing or decreasing the exposure may be adjusted according to the difference in the exposure evaluation value between the poorly exposed tile and the adjacent tile. Alternatively, an appropriate exposure value may be estimated from the exposure values of a plurality of adjacent tiles (left and right tiles, up and down, left and right tiles, etc.). For example, an average value, a mode value, a median value, etc., of exposure values of a plurality of adjacent tiles can be selected as appropriate exposure values.

  Next, the workstation 5 transmits a tile re-imaging request to the control unit 7 (step S71). The re-imaging request includes information such as the xy coordinate value of the tile to be re-imaged and the exposure condition at the time of re-imaging determined in step S70.

  The control unit 7 drives the stage 1 or the imaging system in the xy direction in accordance with the re-imaging request, and adjusts the tile to be re-imaged to the imaging position (field of view) (step S72). And the control part 7 adjusts exposure according to the exposure conditions at the time of re-imaging (step S73). The exposure adjustment is performed by adjusting the light amount of the illumination unit 8, adjusting the aperture of the optical system 3, adjusting the exposure time (electronic shutter speed) of the image sensor 4, and the like. Thereafter, re-imaging of the tile is performed according to the changed exposure condition (step S74). Subsequent processing is the same as the processing in FIG.

  According to the configuration of the present embodiment described above, it is possible to automatically and efficiently adjust the exposure condition and perform re-imaging when an exposure failure occurs during imaging of some tiles. Thereby, the image quality (especially brightness) of all the tiles can be kept uniform, and when the tile images are joined together, a high-quality whole image without unevenness of brightness and pseudo contour can be obtained. In addition, by combining the brightness of tile images, it is possible to increase the accuracy of stitching.

<Third Embodiment>
In the third embodiment of the present invention, an example will be described in which, when a defective image (re-captured image required) whose focus position in the optical axis direction is not appropriate is detected, re-imaging is performed by changing to a proper focus position.

  In a slide used for pathological examination or pathological diagnosis, the focus position (focus position where the focus is most focused) may differ from tile to tile depending on the unevenness or undulation of the specimen or cover glass. Therefore, if the entire slide is photographed at a fixed focal position, an image blurred by some tiles may be obtained. Alternatively, even when tiling is performed by adaptively changing the focal position for each tile instead of a fixed focal position, there is a possibility that focusing may fail due to a sensor or program error. When such an in-focus failure occurs, it is necessary to perform re-imaging at a focal position different from when the defective image is obtained.

  FIG. 7 shows a part of the processing flow of the third embodiment. The flow in FIG. 7 shows the details of the processing in steps S24 and S25 in FIG. Since other processes are the same as those of the first embodiment, detailed description thereof is omitted.

  When a poorly focused tile is detected (step S23; YES), the workstation 5 estimates an appropriate focal position of the tile (step S70). For example, the focus position may be a focus position that is moved by a predetermined distance. For example, the predetermined distance may be set to the same depth as the depth of field of the optical system 3, the size of the contrast value of a poorly focused image (Tenenbaum Gradient evaluation value, etc.), or the contrast with an adjacent tile image. You may change according to the difference of a value. Also, the focal position movement direction is estimated based on whether the in-focus image is a so-called front pin or rear pin, and the focal position is moved downward in the case of the front pin and upward in the case of the rear pin. . Whether it is a front pin or a rear pin can be estimated by analyzing surrounding tile images, for example. The contrast value is not a contrasting peak as in a general Gaussian curve, and taking a sample as an example, the contrast value for the front pin may be significantly lower than for the rear pin . Therefore, if the contrast value of the in-focus image is significantly lower than that of the adjacent tile image, it is determined as a front pin, and if the contrast value is not so low (however, the contrast value is low to determine that it is an in-focus image) ) Can be determined as a rear pin. Alternatively, if tiling shooting is performed with adaptively changing the focus position for each tile, the proper focus position can be estimated from the focus positions of multiple adjacent tiles (left and right tiles, top and bottom tiles, etc.) Good. For example, an average value, a mode value, a median value, and the like of the focal positions (Z coordinates) of a plurality of adjacent tiles can be selected as appropriate focal positions of the in-focus tiles.

  Next, the workstation 5 transmits a tile re-imaging request to the control unit 7 (step S71). The re-imaging request includes information such as the xy coordinate value of the tile to be re-imaged and the focal position at the time of re-imaging determined in step S70.

  The control unit 7 drives the stage 1 or the imaging system in the xy direction in accordance with the re-imaging request, and adjusts the tile to be re-imaged to the imaging position (field of view) (step S72). And the control part 7 adjusts the focus position of an optical axis direction according to the conditions of the focus position at the time of re-imaging (step S73). Thereafter, re-imaging of the tile is performed based on the changed focal position (step S74). Subsequent processing is the same as the processing in FIG.

  According to the configuration of the present embodiment described above, it is possible to automatically and efficiently adjust the focus position and perform re-imaging when focusing failure occurs during imaging of some tiles. Thereby, the image quality (especially sharpness) of all tiles can be kept good, and a high-quality whole image can be obtained when the tile images are joined together. In addition, it is possible to increase the accuracy of stitching by reducing the blur of each tile image.

<Fourth embodiment>
In the second and third embodiments described above, an appropriate imaging condition is estimated based on the image characteristics of the defective image and the adjacent tile image, but in the fourth embodiment, using the functions of automatic focusing and automatic exposure adjustment, An example of determining an appropriate imaging condition at the time of reimaging will be described.

  FIG. 8 shows a part of the processing flow of the fourth embodiment. The flow in FIG. 8 shows the details of the processing in steps S24 and S25 in FIG. Other processes are the same as those in the first embodiment. In FIG. 8, the steps denoted by the same reference numerals as those in FIG. 7 are the same as those in the second or third embodiment.

  When a tile with poor exposure or poor focus is detected (step S23; YES), the workstation 5 estimates an appropriate imaging condition for the tile (step S70). When the imaging condition is successfully estimated (step S80; YES), the same processing (steps S71 to S74) as in the second or third embodiment is performed.

  On the other hand, when the estimation of the imaging condition has failed (step S80; NO), the process proceeds to step S81. Examples of cases where the estimation fails are, for example, when an image of an adjacent tile cannot be acquired (for example, in the case of the tile at the end of the imaging range), when an image of an adjacent tile is also defective, and image characteristics of surrounding tiles There is a case where the estimation accuracy cannot be expected due to a large variation in the number. In step S81, the workstation 5 transmits a tile re-imaging request to the control unit 7 (step S71). The re-imaging request includes information such as the xy coordinate value of the tile to be re-imaged. Information on the type of imaging condition determined to be inappropriate is also included.

  The control unit 7 drives the stage 1 or the imaging system in the xy direction in accordance with the re-imaging request, and adjusts the tile to be re-imaged to the imaging position (field of view) (step S82). And the control part 7 performs the automatic search of the appropriate imaging condition with respect to the said tile, and determines the optimal imaging condition (step S83). For example, in the case of poor exposure, an optimal exposure value is determined by automatic exposure adjustment. At this time, the exposure value may be determined using an exposure meter, but the optimum exposure may be determined by imaging the tile multiple times while changing the exposure and comparing the exposure evaluation values of those images. Good. In the case of poor focusing, an optimum focus position is determined by auto-focusing (Auto-Focus). There are an active method and a passive method for automatic focusing, either of which may be used. In the case of the passive method, the optimum focus position may be determined by imaging the tile a plurality of times while changing the focus position and comparing the focus evaluation values of these images. When the optimum imaging condition is determined in step S83, the subsequent processing is the same as in the above-described embodiment.

  According to the configuration of the present embodiment described above, it is possible to automatically and efficiently determine an imaging condition and perform re-imaging when a defect occurs during imaging of some tiles. In particular, in the present embodiment, even when an appropriate imaging condition cannot be estimated from an image, the optimum imaging condition can be automatically searched, so that recovery when a defect occurs can be performed more reliably. By the way, in the present embodiment, the process of estimating an appropriate imaging condition from the image characteristics of the tile image (step S70) and the process of searching for the optimal imaging condition (step S83) are combined, but only the latter search process. May be configured to execute. However, since the search process generally takes longer than the estimation process, it is preferable to perform the search process only when the estimation process fails as in the present embodiment.

1: Stage 2: Slide 3: Optical system 4: Imaging sensor 5: Workstation 7: Control unit

Claims (15)

A stage to hold the subject,
Imaging means having a smaller field of view than the subject;
Control means for controlling the stage and the imaging means;
Image processing means for inputting an image obtained by the imaging means,
Corresponding to a plurality of regions of the subject by performing the imaging by the imaging unit a plurality of times while changing the position of the visual field with respect to the subject by moving at least one of the stage and the imaging unit. A first operation of acquiring a plurality of tile images to be performed;
A second operation in which the image processing means inspects the plurality of tile images acquired in the first operation and determines whether there is a re-captured image that needs to be retaken;
When it is determined in the second operation that there is a re-captured image, the control unit moves the stage and at least one of the imaging unit and places the field of view in a region corresponding to the re-captured image. In addition, a third operation of acquiring a tile image instead of the re-captured image by performing re-imaging by the imaging unit;
The image acquisition apparatus characterized by performing. The image acquisition apparatus according to claim 1, wherein the first operation, the second operation, and the third operation are executed while the subject is held on the stage. 3. The method according to claim 1, wherein, in the third operation, the control unit performs re-imaging while changing an imaging condition from when the re-captured image is acquired in the first operation. The image acquisition device described. The said control means WHEREIN: The said control means changes the focus position of an optical axis direction, and re-images, when the said re-capturing image required in the said 1st operation | movement is imaged. 4. The image acquisition device according to 3. The image acquisition apparatus according to claim 4, wherein the re-imaging focal position is determined by performing automatic focusing on a region corresponding to the re-captured image. In the third operation, the control means performs re-imaging with different exposure from when the re-captured image is captured in the first operation. The image acquisition device according to any one of the above. The image acquisition apparatus according to claim 6, wherein the re-imaging exposure is determined by performing automatic exposure adjustment on an area corresponding to the re-captured image. The said 3rd operation | movement WHEREIN: The said control means performs a re-imaging on the same imaging conditions as when the said re-capturing image required in the said 1st operation | movement was acquired. Image acquisition device. In the second operation, the image processing means extracts image features from each of the plurality of tile images, and evaluates the continuity of the image features between adjacent tile images, whereby a re-captured image is required. The image acquisition apparatus according to claim 1, wherein it is determined whether or not there is an image. In the second operation, the image processing means obtains a focus evaluation value representing a degree of focus for each of the plurality of tile images, and performs a focus evaluation value of an adjacent tile image and a focus evaluation of a target tile image. 10. The image acquisition device according to claim 1, wherein when the difference or change rate between the values is larger than a predetermined value, the tile image of interest is determined as a re-captured image. . The image acquisition apparatus according to claim 10, wherein the focus evaluation value is a contrast value of a tile image. In the second operation, the image processing unit obtains an exposure evaluation value for evaluating the appropriateness of exposure for each of the plurality of tile images, and between the exposure evaluation value of the adjacent tile image and the exposure evaluation value of the target tile image. The image acquisition apparatus according to any one of claims 1 to 11, wherein the tile image of interest is determined as a re-captured image when the difference or rate of change is greater than a predetermined value. The image acquisition apparatus according to claim 12, wherein the exposure evaluation value is the number or ratio of pixels in which gradation information is lost, or the mode value, average value, or median value of a histogram. A control method for an image acquisition apparatus, comprising: a stage for holding a subject; and an imaging unit having a field of view smaller than the subject,
A plurality of tile images corresponding to a plurality of regions of the subject by performing imaging by the imaging unit a plurality of times while changing the position of the field of view with respect to the subject by moving at least one of the stage and the imaging unit. Obtaining a first step;
Inspecting the plurality of tile images acquired in the first step and determining whether or not there is a re-captured image that needs to be retaken;
When it is determined in the second step that there is a re-captured image, the imaging is performed by moving at least one of the stage and the imaging unit and aligning the field of view with a region corresponding to the re-captured image. Obtaining a tile image instead of the re-captured image by performing re-capturing by means;
A control method for an image acquisition apparatus.   The program for making an image acquisition device perform each step of the control method of the image acquisition device of Claim 14.

Images