JP2006243608A - Microscope system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a microscope system capable of displaying a natural image free from image blurring on a display of reproducing photographed images, regarding multipoint time-lapse photographing. <P>SOLUTION: When the photographed images GaO, Ga1 and Ga6 are displayed so that a photographing point (a) may be located in the center of a monitor 40, the images are displayed shifting from the monitor center due to the stage positioning error at photographing. A rectangular Ga shows a photographing area in the case of no positioning error. The shifting width S1 to the right of the photographed image Ga6 relative to the rectangular area Ga is set as a trimming width on the left side. Similarly, the downward-shifting width S2 of the photographed image Ga1, the upward shifting width S3 of the photographed image Ga6 and the shifting width S4 of the photographed image Gao to the left are separately set as the trimming width on the upper/right/lower side. And each photographed image is trimmed corresponding to the rectangular area Ga based on the trimming widths S1 to S4 and the positional shifting amount of each photographed image with reference to the rectangular area Ga. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a microscope system that captures microscope images at a predetermined time interval with respect to a plurality of observation points.

  Conventionally, as a method of observing a specimen such as a biological cell, a method called time lapse photography in which a microscope image is photographed at regular time intervals is known (for example, see Patent Document 1). When there are a plurality of shooting points, this is called multi-point time lapse, and an electric stage on which a sample is placed is driven to sequentially shoot each shooting point. Such cyclic photographing is performed a plurality of times at regular time intervals, and a plurality of photographed images are acquired for each photographing point. After the photographing is completed, a temporal change in the specimen can be observed by reproducing a plurality of photographed images one after another for each specific photographing point.

JP 2002-277754 A

  By the way, in multipoint evening lapse photography, since a plurality of photography points are cyclically photographed, a high positioning accuracy is required for the stage when moving to the same photography point. When an error occurs in the stage positioning, the shooting area is shifted every time shooting is performed, and there is a problem that when the captured images are continuously reproduced, the images are blurred and become unnatural images.

  Further, when performing high-precision positioning of the stage, feedback control using a linear encoder is adopted for detection of the stage position. However, as the magnification of the optical system increases, the positioning accuracy required for the stage also increases, and so-called hunting that reciprocates before and after the target position occurs when an attempt is made to satisfy it. Therefore, it may take time for positioning, and in the worst case, hunting may not stop.

According to the first aspect of the present invention, a plurality of observation locations set on the observation target are sequentially moved on the optical axis of the objective optical system, and a plurality of observation images of the observation target are sequentially acquired by the imaging device for each observation location. Applied to a microscope system for taking multiple images, a driving means for two-dimensionally driving a stage on which an observation object is placed and an objective optical system in a direction perpendicular to the optical axis, and the stage and the objective optical system Based on the position detection means for detecting one or both positions, and the position information detected by the position detection means, a plurality of photographed images taken over time with respect to the observation location are stored in the same observation region including the observation location. The image processing apparatus is characterized by including correction means for correcting trimming so as to form an image.
According to a second aspect of the present invention, in the microscope system according to the first aspect, a common photographing region included in all of the plurality of photographed images is set to the same observation region.
According to a third aspect of the present invention, in the microscope system according to the first or second aspect, if the position information detected by the position detecting means falls within a predetermined positioning range, the stage of the driving means or the objective optical system Control means for stopping the drive is provided.
According to a fourth aspect of the present invention, in the microscope system according to the third aspect, the correction means sets a trimming amount for trimming correction according to the size of the positioning range.

  According to the present invention, since a plurality of photographed images taken over time with respect to the photographing point are trimmed and corrected so as to be images of the same photographing region including the image of the photographing point, positioning errors occur during photographing. Even if it occurs, it is possible to eliminate blurring of images when the corrected captured images are displayed in order.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an embodiment of a microscope system according to the present invention, and shows an outline of an apparatus configuration. Reference numeral 10 denotes an inverted microscope, and the microscope 10 is provided with an electric stage 30. An electric revolver 11 to which the objective lens 12 can be attached is provided below the electric stage 30, and an illumination device 14 for illuminating the sample is provided above the electric stage 30. A camera 20 for taking an observation image incorporates a CCD image pickup device as an image pickup device.

  The electric stage 30 has an XY stage driven by a DC motor or the like, and the petri dish 13 containing the sample can be moved in the XY direction by driving the XY stage. The xy coordinates of the XY stage are detected by a linear encoder (not shown) provided on the electric stage 30. A stage controller 31 that controls the movement operation of the electric stage 30 is connected to the electric stage 30. An address is assigned to each objective mounting hole of the electric revolver 11, and the magnification of the objective lens 12 arranged on the optical axis L can be detected by a table input in advance. The electric revolver 11 may also be configured to be electrically driven in the XY directions, and the XY coordinates may be acquired by a linear encoder. In short, it is only necessary that the electric stage 30 and the electric revolver 11 can be driven in the XY directions relatively and position information (XY coordinates) of both can be detected.

  A PC (personal computer) 50 controls the entire microscope system, performs image processing, and the like. The stage controller 31 and the microscope 10 are connected to the PC 50, and position information from the encoder, magnification information of the objective lens 12, and Shooting data from the camera 20 is input. The PC 50 is connected to a monitor 40, a pointing device 60 such as a mouse, and an input keyboard (not shown), so that a microscope image taken by the camera 20 can be displayed on the monitor 40. As shown in FIG. 2B, the monitor 40 displays a display area 40Y for displaying the entire image (macro image) of the petri dish 13 and images (micro images) of the shooting points a, b, and c in the petri dish 13. Display area 40X.

<Acquiring captured image data>
When observing live cells, the petri dish 13 is filled with a medium, and images are taken at predetermined time intervals while keeping the cells alive. In the case of multi-point time lapse observation, for example, when there are three observation points, photographing points a, b, and c are set in advance as shown in FIG. FIG. 3 is a flowchart showing a procedure for setting the time lapse and shooting, and the setting and shooting process will be described with reference to this flowchart.

  In step S1 of FIG. 3, it is determined whether or not a multipoint time-lapse shooting condition has been input. If it is determined that there is an input, the process proceeds to step S2. Examples of conditions for time lapse shooting include a shooting point, a shooting time interval Δt, and a time lapse duration ttotal. The operator moves the pointer P of the pointing device 60 to the position to be observed on the macro image 13a on the display screen while viewing the captured images 40a, 40b, and 40c (see FIG. 2) displayed on the monitor 40, and sets the shooting point. specify.

  Hereinafter, a case where three points a, b, and c shown in FIG. 2 are set as shooting points will be described as an example. When the pointing device 60 is operated to move the pointer P over the position a and perform a confirmation operation, the position a is set as a shooting point. The same operation is performed to set shooting points in the order of b and c. With this setting, the xy coordinates of the shooting points a, b, and c are stored in the storage unit of the PC 50, respectively. In time-lapse shooting described later, shooting is performed in the order of the input shooting points a, b, and c.

  Further, a condition input screen is displayed on the monitor 40, and the shooting time interval Δt and the time lapse duration ttotal are input using the pointing device 60 or the keyboard. The shooting time interval Δt is the time interval from the end of the shooting of a, b, c to the next shooting of a, b, c, and the time lapse duration ttotal is the time from the start of the shooting process to the end of the shooting process. . For example, if the shooting time interval Δt = 10 minutes and the time lapse duration ttotal = 60 minutes, shooting is performed seven times for each shooting point including shooting at the start of processing. Below, it demonstrates on this condition. When input of all the conditions is completed, an instruction to start time-lapse shooting is given.

  In step S2, it is determined whether or not there has been an instruction to start time-lapse photography. If it is determined that there has been an instruction, instructions are sequentially issued to the electric stage 30 and the camera 20 in accordance with the input conditions, and step S3. The subsequent series of processing is automatically executed. The process from step S3 to step S8 represents a single shooting process for three shooting points a, b, and c, and a series of shooting processes from step S3 to step S8 are repeatedly executed at the above-described shooting time interval Δt. Is done.

  First, in step S3, based on the xy coordinate data Xa and Ya of the shooting point a stored at the time of setting the shooting point, the electric stage 30 is moved so that the shooting point a is arranged at the center of the shooting screen. Memorize stage position Xai, Yai. Here, i represents the data acquisition order, and the initial value is i = 0. In other words, the stage position data obtained first from step S2 to step S3 are Xa0 and Ya0. In step S4, a microscope image of the shooting point a is taken. The obtained captured image data Ai is stored in association with the stage positions Xai and Yai and the magnification M of the optical system.

  In step S5, the electric stage 30 is moved based on the xy coordinate data Xb and Yb of the shooting point b stored at the time of setting the shooting point so that the shooting point b is arranged at the center of the shooting screen, and the stage after the movement is moved. Memorize the positions Xbi, Ybi. In step S6, a microscope image of the shooting point b is taken. The obtained photographed image data Bi is stored in association with the stage positions Xbi and Ybi and the magnification M of the optical system.

  In step S7, the electric stage 30 is moved based on the xy coordinate data Xc and Yc of the shooting point c stored at the time of setting the shooting point so that the shooting point c is arranged at the center of the shooting screen, and the stage after the movement is moved. Memorize the position Xci, Yci. In step S8, a microscope image at the shooting point c is taken. The obtained photographed image data Ci is stored in association with the stage positions Xci and Yci and the magnification M of the optical system.

  In step S9, it is determined whether or not the elapsed time from the start of the process in step S3 has elapsed, and if it is determined that it has elapsed, the process proceeds to step S10. In step S10, it is determined whether i representing the data acquisition order is i = 6, that is, whether the time lapse duration ttotal has elapsed. According to the conditions set in step S2 described above, since seven shootings are performed at each shooting point and the initial value of i is i = 0, the determination condition as to whether or not the time lapse duration ttotal has elapsed is i. = 6. If it is determined in step S10 that i ≠ 6 (NO), the process proceeds to step S11, and if it is determined that i = 6 (YES), a series of processing operations relating to acquisition of captured image data ends.

  When the process proceeds from step S10 to step S11, the magnitude of i is increased to i + 1 and the process returns to step S3. Then, the processing from step S3 to step S8 is performed in the same manner as the first (i = 0) captured image data acquisition described above, and the second (i = 1) captured image data is acquired. The processing from step S3 to step S8 is repeated until i = 6 in step S10, and as a result, a total of 21 (= 3 × 7) pieces of captured image data A0, A1,..., A6, B0, B1,..., B6, C0, C1,..., C6 are acquired together with the stage position and the optical system magnification M when each is acquired. These data are stored in the storage unit of the PC 50.

  By the way, as a positioning method when moving the electric stage 30, there are open loop control and closed loop control for performing feedback. In the open loop control, the rotation amount of the stage driving motor necessary to reach the target position is calculated in advance, and the motor is rotated based on this. On the other hand, in the case of closed loop control, the stage is moved to the target position while feeding back the stage position detected by the linear encoder. A predetermined range whose center value is the target position is set in advance, and the stage movement is stopped when the detected position of the stage to be detected falls within the predetermined range.

  In the above description, the open-loop control is assumed. However, in the case of closed-loop control, the coordinates (Xa, Ya), (Xb, Yb), (Xc, Yc) at the time of shooting point setting are used as the shooting point a. , B, and c are set as target positions. Also, instead of the coordinates (Xa, Ya), (Xb, Yb), (Xc, Yc), the stage positions (Xa0, Ya0), (Xb0, Yb0) at the time of obtaining the first (i = 0) captured image data , (Xc0, Yc0) may be the target position. Thereafter, for i = 1 to 6, the stage may be moved with the coordinates (Xa0, Ya0), (Xb0, Yb0), (Xc0, Yc0) as targets.

  In this way, for each of the plurality of shooting points a, b, and c, seven shot image data having different shooting times are acquired, and these are reproduced and displayed on the monitor 40. For example, it is easy to understand temporal changes of living cells by sequentially reproducing and displaying seven captured images in order. However, as described above, there is a problem that image blurring occurs when the stage positioning accuracy is poor.

  In particular, when there are many shooting points (several tens to several hundred points), the time that is stopped at the shooting point is shortened depending on the time lapse setting time and the number of shooting points, and the shooting point moves to the next point. Therefore, the stage must be driven at high speed, and the stage positioning accuracy is deteriorated. Therefore, in the present embodiment, the acquired image is corrected, and the corrected image is displayed to eliminate the image blur at the time of continuous reproduction.

<Description of image correction>
Next, a method for correcting captured image data will be described. Since the correction method is the same for any of the photographed image data at the photographing points a, b, and c, correction for the photographed image data at the photographing point a will be described below. As described above, when the seven captured images Ga0, Ga1,..., Ga6 obtained for the photographing point a are simultaneously displayed on the monitor 40 due to the stage positioning error, the display position is shifted as shown in FIG. Occurs. Note that the photographed images Ga0, Ga1,..., Ga6 are images based on the photographed image data A0, A1,..., A6, and in FIG. Only the captured images Ga0, Ga1, and Ga6 are shown. A rectangular area Ga indicated by a one-point difference line shows an ideal photographed image when the positioning error = 0.

  At the time of shooting, the electric stage 30 is driven so that the shooting point a is in the center of the imaging region. Therefore, if there is no positioning error, the shooting point a is an image in all the shot images Ga0, Ga1,. It should be taken in the center. However, since there is actually a positioning error, the shooting point a is shot at a position shifted from the center of the image. Therefore, as shown in FIG. 4, when the captured images Ga0, Ga1,..., Ga6 are displayed so that the shooting point a is at the center of the monitor 40, the centers of the captured images Ga0, Ga1,. It will deviate from.

  The captured images Ga0, Ga1, and Ga6 shown in FIG. 4 are images that are shifted from each other, but the shaded rectangular region R is included in any of the captured images Ga0, Ga1, and Ga6. Therefore, in the present embodiment, the captured images Ga0, Ga1, and Ga6 are trimmed into images of only the rectangular region R, and the trimmed corrected images Ga0 ′, Ga1 ′, and Ga6 ′ are reproduced and displayed on the monitor 40. I tried to do it.

(Calculation of misaligned images)
First, a method for calculating a positioning error necessary for performing trimming correction will be described. As described above, the xy coordinates of the shooting point a stored in the storage unit of the PC 50 by the shooting point setting are Xa and Ya, respectively. The misalignment amount at the center of each captured image Ga0, Ga1,..., Ga6 with respect to the center of the monitor, that is, the misalignment amount on the imaging surface, is obtained by multiplying the misalignment amount of the electric stage 30 by the magnification M of the optical system. . Since the stage position shift amount is the difference between the coordinates at the time of shooting point setting and the coordinates at the time of shooting, the position shift amounts Δxai and Δyai on the imaging surface are expressed by the following equations (1) and (2).
Δxai = (Xai−Xa) × M (1)
Δyai = (Yai−Ya) × M (2)

Here, when the pixel size of the CCD image sensor used in the camera 20 is a square having a dimension p on one side, and the above-described positional deviation amounts Δxai and Δyai are converted into the number of CCD pixels, the converted values ΔPxai and ΔPyai are expressed by the following formulas ( 3) and (4).
ΔPxai = Δxai / p = (Xai−Xa) × M / p (3)
ΔPyai = Δyai / p = (Yai−Ya) × M / p (4)

(Correction process)
Next, an image correction method by trimming will be described. In the following, it is assumed that the rectangular areas Ga and R and the size of the captured image are represented by the number of pixels of the CCD image sensor. The rectangular area R in FIG. 4 shows the size of the image after trimming. In this rectangular area R, the left side area of the rectangular area Ga is trimmed by the width of the pixel number S1, and the upper side area is changed to the pixel number S2. Is obtained by trimming the right side region by the width of the pixel number S3 and trimming the lower side region by the width of the pixel number S4.

As can be seen from FIG. 4, the left side of the rectangular region R coincides with the left side of the captured image Ga6 shifted to the rightmost side. Similarly, the upper side of the rectangular region R coincides with the upper side of the photographed image Ga1 shifted to the lowermost side, the right side of the rectangular region R coincides with the right side of the photographed image Ga0 displaced to the leftmost side, and the lower side of the rectangular region R This coincides with the lower side of the photographed image Ga6 shifted to the uppermost side. Therefore, the trimming widths S1, S2, S3, and S4 are given by S1 = ΔPxa6, S2 = −ΔPya1, S3 = −ΔPxa0, and S4 = ΔPya6. Therefore, when trimming correction is performed on each captured image Gai, trimming widths S1i to S4i of each side of the captured image Gai are set as in the following equations (5) to (8) (see FIG. 4).
Left side: S1i = S1-ΔPxai (5)
Upper side: S2i = S2 + ΔPyai (6)
Right side: S3i = S3 + ΔPxai (7)
Lower side: S4i = S4-ΔPyai (8)

  When the series of photographing operations shown in FIG. 3 is completed and all photographed images have been acquired, the calculation of the positional deviation amounts Δxai and Δyai on the imaging surface and the setting of the rectangular region R described above are performed. Trimming correction of each captured image Gai is performed using the trimming width calculated in (5) to (8). The corrected captured image Gai ′ is stored in the storage unit of the PC 50 in association with the captured image Gai. Alternatively, the captured image Gai may be rewritten and stored with the corrected captured image Gai ′. When the captured images are continuously reproduced and displayed on the monitor 40, the corrected captured images Gai 'are sequentially displayed. Note that the trimming correction processing of the captured images Gbi and Gci related to the captured points b and c is exactly the same as that of the captured image Gai, and thus description thereof is omitted here.

  In the above description, the trimming widths S1 to S4 are set by comparing the rectangular area R and the rectangular area Ga, and each photographing is performed based on the converted values ΔPxai and ΔPyai of the positional deviations of the expressions (3) and (4). Although the image is trimmed, for example, a captured image Ga0 may be used instead of the rectangular area Ga. In this case, the positional shift amounts are considered based on the coordinates (Xa0, Ya0) of the captured image Ga0 for Δxai and Δyai.

[Modification]
In the above-described embodiment, the trimming width is set after all the captured images Gai, Gbi, and Gci are acquired, and the captured images Gai, Gbi, and Gci are corrected for trimming using the trimming width. However, in the closed loop control in which the electric stage 30 is positioned by feeding back the detected stage position, the trimming width is set first, and the electric stage 30 is positioned within an error range in which the trimming width is possible. Anyway.

  First, a trimming width S is set as shown in FIG. In FIG. 5, the xy coordinates of the shooting point a are coordinates Xa and Ya read when the shooting point a is set on the monitor 40. A rectangular area Ga indicates a shooting area set around the coordinates Xa and Ya, and a rectangular area R indicates a shooting image area after trimming correction. The electric stage 30 is positioned by feedback control with the coordinates Xa and Ya as target positions.

  Here, a case where the trimming width of the upper, lower, left and right sides of the rectangular area Ga is S will be described. The positioning allowable range when the trimming width S is set in this way will be described. The captured image Ga1 of FIG. 5 shows a case where the positioning error is too large with respect to the target position (Xa, Ya). In this case, on the left side of the rectangular region R, a part of the region is outside the captured image Ga1.

  That is, in the case of the photographed image Ga1, since there is no image data near the left side of the rectangular area R, the photographed image Ga1 cannot be trimmed to the size of the rectangular area R. On the contrary, when the captured image Ga1 is greatly shifted to the left with respect to the rectangular area R, trimming is similarly impossible. Therefore, it is necessary to control the positioning of the electric stage 30 so that the horizontal positioning error ΔPxai on the imaging surface satisfies | ΔPxai | ≦ S. Considering the same for the vertical direction, the positioning error ΔPyai in both the vertical direction on the imaging surface must be | ΔPyai | ≦ S.

When the positioning error on the imaging surface is converted into the positioning error of the electric stage 30 using the equations (3) and (4), the following equations (9) and (10) are obtained. Then, the closed loop control is performed so that the stage coordinates X and Y satisfy the conditions of Expressions (9) and (10), and the electric stage 30 is stopped when the conditions are satisfied. In this case, by setting the trimming width S to a size that does not cause hunting during closed loop control, the occurrence of hunting can be prevented and the trimming width S can be reduced.
| X−Xa | ≦ (S · p / M) (9)
| Y−Ya | ≦ (S · p / M) (10)

  As described above, in the present invention, by correcting the trimming of the acquired captured image, the influence of the stage positioning error in the captured image can be removed, and a natural image without image blur is displayed when continuously reproduced. be able to. In addition, since a method of correcting a photographed image in software is adopted, it can be easily applied without changing the hardware configuration of a conventional microscope system.

  In the above-described embodiment, the microscope 10, the camera 20, and the electric stage 30 as the microscope system have been described as being connected to the PC main body 50 as independent products, but the present invention is not limited to this. . For example, it can be realized with a microscope system in which a microscope, a camera, and an electric stage are integrated. Further, although a microscope system using the inverted microscope 10 has been described as an example, the present invention can be similarly applied to a microscope system using an upright microscope.

  In the correspondence between the embodiment described above and the elements of the claims, the camera 20 is an imaging device, the rectangular region R is the same observation region, the PC 50 is a correction means, the electric stage 30 is a stage, the PC 50 and the stage. The controller 31 constitutes control means, and the linear encoder provided on the electric stage 30 constitutes position detection means. The above description is merely an example, and when interpreting the invention, there is no limitation or restriction on the correspondence between the items described in the above embodiment and the items described in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure which shows one Embodiment of the microscope system by this invention, and shows the outline of an apparatus structure. It is a figure explaining the imaging | photography point a, b, c displayed on the display monitor 40, (A) is a conceptual diagram which shows the relationship between a macro image and a micro image, (B) shows the display content of the monitor 40. . It is a flowchart which shows the procedure of a time lapse setting and imaging | photography. It is a figure which shows the picked-up image Ga0, Ga1, and Ga6 displayed on the display monitor 40. FIG. It is a figure explaining the setting of the trimming width S in a modification.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Microscope 12 Objective lens 13 Petri dish 20 Camera 31 Stage controller 40 Monitor 50 PC
a, b, c Shooting points
Ga, R rectangular area

Claims (4)

A microscope system that sequentially moves a plurality of observation spots set on the observation target on the optical axis of the objective optical system, and images the observation image of the observation target multiple times over time by the imaging device for each observation spot. In
Driving means for two-dimensionally driving the stage on which the observation object is placed and the objective optical system in a direction perpendicular to the optical axis;
Position detecting means for detecting a position in one direction or both of the stage and the objective optical system;
Based on the position information detected by the position detection means, a correction means for correcting trimming of a plurality of photographed images taken over time with respect to the observation location so as to be images of the same observation area including the observation location. And a microscope system. The microscope system according to claim 1, wherein
A microscope system, wherein a common imaging region included in all of the plurality of captured images is set in the same observation region. The microscope system according to claim 1 or 2,
A microscope comprising control means for stopping the driving of the stage or the objective optical system by the driving means when the position information detected by the position detecting means falls within a predetermined positioning range. system. The microscope system according to claim 3,
The microscope system according to claim 1, wherein the correction unit sets a trimming amount for the trimming correction in accordance with a size of the positioning range.

Images