JP2010079221A - Living body observation apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a living body observation apparatus having improved detection accuracy. <P>SOLUTION: The living body observation device includes: an objective lens disposed in close contact with a specimen; an image pickup device that captures an image of the specimen; an image processing section that sets a search area in the image captured by the image pickup device and processes the image in the search area; and an objective lens drive device that drives the objective lens in the direction of correcting the displacement of the specimen, based on a signal from the image processing section. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a living body observation apparatus.

In recent years, in biological research, so-called in vivo observation has been performed in which an individual experimental animal is used as a specimen and its organs and the like are observed with an optical microscope while alive. In in-vivo observation, the observation object has movements such as pulsation and respiratory movement, so that the observation position is shifted and the focus is likely to be blurred. As a method for eliminating such observation position deviation and out-of-focus blur, an apparatus (for example, Patent Document 1) that tracks the entire microscope including an imaging unit in accordance with the movement of the observation target, or the movement of the observation target. An apparatus (for example, Patent Document 2) that causes an objective lens to follow is also known. In Patent Document 2, a detection camera is provided separately from the observation camera in order to detect the movement of the observation target.
JP-A-7-222754 JP 2007-187810 A

  Usually, a detection camera for detecting a displacement of a living body has a high frame rate (about 1000 fps). Therefore, the image processing time may exceed the frame rate interval. In addition, if the image processing time is long, a detection delay time with respect to the actual displacement of the living body occurs, and there is a problem that the accuracy of the follow-up control is lowered. Furthermore, if the image processing time is long, the dead time for the visual servo system will increase, which will affect the stability of the tracking control system and the control gain cannot be increased so high that accurate tracking control cannot be performed. was there.

  The present invention has been made in view of the above problems, and in a living body observation apparatus for observing a living body alive, by reducing the image processing time for detecting the movement of the living body, the present invention is more accurate than before. It is an object of the present invention to provide a living body observation apparatus that can perform follow-up control and thereby improve detection accuracy.

In order to achieve the above object, the present invention provides the following means.
The present invention provides an objective lens that is arranged close to a sample, an imaging device that images the sample via the objective lens, a search region set in an image captured by the imaging device, and an image in the search region And an objective lens driving device that drives the objective lens in a direction to correct the displacement of the sample based on a signal from the image processing unit.

  According to the present invention, the sample is imaged by the imaging device with respect to the sample that causes the observation position to shift to show the dynamic behavior, the displacement is detected from the obtained image, and based on the detection result. By operating the objective lens driving device, the objective lens is driven in a direction to correct the image shift.

  The displacement of the living body is mainly due to respiration, pulsation, and pulsation, and is detected as a regular vibration as shown in FIG. In the present invention, attention is paid to the movement unique to the living body, that is, vibration in a certain direction, and the search area is set using this property.

  Specifically, a search area is set in the image acquired by the imaging device, only the image in the search area is processed, and the displacement of the living body is detected. By limiting the image area to be processed in this way, the image processing time can be shortened, the detection delay time with respect to the actual biological vibration can be shortened, and the accuracy of the follow-up control can be improved.

In the above-mentioned invention, it can further be provided with a control device which controls the above-mentioned imaging device so that only a predetermined field is imaged.
With such a configuration, the area to be captured is limited, and the time for transferring the captured image to the image processing unit can also be shortened.

In the above invention, as the imaging device, a detection imaging device and an observation imaging device for detecting a displacement of a sample and capturing an image in which a search region is set by the image processing unit are used. be able to.
By using the two imaging devices for detection and observation, an image for detecting a biological displacement can be acquired separately from the image for observation. For this reason, it is possible to set the imaging region in the imaging device for detection without considering the acquisition of the observation image.

  Moreover, in the said invention, the said search area | region may be a rectangular area | region which makes one side the length exceeding the maximum value of the difference of the displacement between continuous image frames. By doing so, the search area can be extremely limited, and the image processing area can be significantly reduced.

  Moreover, in the said invention, the setting of the said search area is performed before and after the drive of the said objective lens drive device is started. By doing in this way, a search area | region is set after the drive of a drive device, ie, after the start of follow-up control, and a search area | region can be reduced further.

  Further, the living body observation apparatus may be a living body observation microscope, or may be an endoscope or other observation apparatus.

  The sample observed in the living body observation apparatus may be a cell, tissue, or internal organ in the living body, and the minute behavior in the living sample is observed by correcting the dynamic behavior of the living body. It is possible.

  According to the present invention, by limiting the image processing area for detecting vibration, the image processing time can be shortened, and the accuracy of the follow-up control can be improved. Thereby, a living body observation apparatus capable of highly accurate detection can be provided.

  The biological observation apparatus of the present invention is suitably used as a microscope observation apparatus, but is not limited to this, and can be used as, for example, an endoscope observation apparatus or other appropriate apparatus. Hereinafter, for convenience of explanation, a microscope observation apparatus including two imaging apparatuses, that is, a detection imaging apparatus and an observation imaging apparatus will be described in detail as an example.

  A microscope observation apparatus 1 according to a first embodiment of the present invention will be described with reference to FIG.

  As shown in FIG. 1, the microscope observation apparatus 1 according to the present embodiment guides the excitation light E from the stage 2 on which the sample S is mounted, the excitation light source 3 that emits the excitation light E, and the excitation light source 3. The illumination lens (illumination optical system) 4 and the excitation light E guided by the illumination lens 4 are guided to the sample S, while the objective lens 5 that collimates the fluorescence F generated in the sample S is collimated by the objective lens 5 A dichroic mirror 6 for branching the fluorescent light F from the excitation light E, a beam splitter 7 for further branching the branched fluorescent light F, and an imaging device such as two CCDs for detecting the branched fluorescent light F, that is, for observation An imaging device 8 and a detection imaging device 9 are provided. Reference numeral 20 in the figure denotes an imaging lens.

  The excitation light source 3, the illumination lens 4, the dichroic mirror 6, the beam splitter 7, and the two imaging devices 8 and 9 are provided in the microscope body 10. On the other hand, the objective lens 5 is separated from the microscope body 10 and supported by the objective lens driving device 11.

  The objective lens driving device 11 includes an arm 12 that arranges the objective lens 5 between the stage 2 and the microscope main body 10, and a first drive mechanism 13 that moves the objective lens 5 held by the arm 12 in the vertical direction. And second and third drive mechanisms 14 and 15 for moving the objective lens 5 in two horizontal directions, and swinging the objective lens 5 about an axis along two directions orthogonal to the optical axis of the objective lens 5. The fourth and fifth drive mechanisms 16 and 17 are provided. The first to third drive mechanisms 13 to 15 include, for example, motors 13a to 15a and a linear movement mechanism such as a ball screw (not shown) connected to the motors 13a to 15a. The fourth and fifth drive mechanisms 16 and 17 include, for example, motors 16a and 17a and speed reduction mechanisms 16b and 17b connected to the motors 16a and 17a.

  In the imaging device 9, an image processing unit 21 that sets a search region in an image acquired by the imaging device 9 and processes an image in the search region to calculate a displacement amount and a direction of the sample S; Based on the information from the image processing unit 21, the objective lens 5 is driven in a direction to correct the deviation of the optical axis of the objective lens 5 due to the displacement of the sample S with respect to the driving mechanisms 13 to 17 of the objective lens driving device 11. A drive control device 18 for outputting a drive command is connected. Further, the objective lens driving device is provided with an encoder for detecting the position of the objective lens.

  In the microscope observation apparatus 1 according to the present embodiment configured as described above, the objective lens 5 is moved in the Z-axis direction along the optical axis by the operation of the objective lens driving device 11, as shown in FIG. Can be moved in the X-axis and Y-axis directions orthogonal to each other, and in the A and B directions around the X-axis and Y-axis.

The operation of the microscope observation apparatus 1 according to the present embodiment configured as described above will be described with reference to FIGS.
In the microscope observation apparatus 1 according to the present embodiment, an observation image of the sample is captured by the observation imaging apparatus 8. On the other hand, the detection imaging device 9 captures an image for detecting the displacement of the sample. Image information captured and acquired by the imaging device 9 for detection is sent to the image processing unit 21, a search area is set, and the image is processed in the area to calculate the displacement amount and direction of the sample S. The The result is sent to the drive control device 18, and command signals sent to the drive mechanisms 13 to 17 of the objective lens drive device 11 are calculated in the drive control device 18, and the drive mechanisms 13 to 17 are operated based on the command signals. Driven.

  Here, the setting of the search area in the image processing unit 21 will be described. For the detection imaging device 9, a high-speed camera can be used, for example, a high-speed camera that takes an image of the observation area at horizontal 256 pixels × vertical 256 pixels at 1000 fps. In general, since a high-speed camera has a short exposure time and low sensitivity, it is difficult to photograph with a brightness that allows observation of the living body itself. Therefore, a marker is placed on the surface of the living body, and the movement of the marker is detected by the detection image pickup device 9, thereby detecting the vibration of the living body.

  In this embodiment, fluorescent beads are used as markers as shown in FIG. As shown in FIG. 5, a fluorescent bead having a size that is photographed with a width of a plurality of pixels or more in an area photographed by the detection imaging device 9 will be described below. The calculation accuracy of the center of gravity position to be improved can be improved.

  Detection of biological vibration using fluorescent beads will be described with reference to the flowchart of FIG. First, an image is picked up by the detection image pickup device 9 (S101), two-dimensional image information is stored in the memory 22 in the image processing unit 21 (S102), and the position of the fluorescent beads is determined from the data representing the luminance in the image information. Calculate. First, a pixel block (labeling region) having a luminance value higher than a predetermined value that does not detect other than fluorescent beads (for example, organs or simple image noise) is detected (S103). On the other hand, the position of the center of gravity is obtained (S104). Then, the calculation result of the center of gravity is stored in the memory 21 as time series information (S105).

  The movement trajectory of the fluorescent beads is calculated in time series by continuously performing the above operation at the frame rate interval of the detection imaging device 9.

  Next, the objective lens control system will be described with reference to the control block diagram of FIG.

  The difference between the target position signal (that is, the marker position signal) for moving the objective lens 5 and the objective lens position signal from the encoder is used as a control error, and the drive control device 18 is set to reduce this control error. Thus, for example, control calculation such as PID control is performed, and the objective lens 5 is driven to an arbitrary position by the objective lens driving device 11 based on the calculation result. Here, it is assumed that the objective lens 5 has a movement range equal to or greater than the maximum amplitude of biological vibration.

Next, a search area setting method will be described.
Now, in the state where the objective lens 5 is stopped, it is the biological vibration itself that can be seen in the observation region. FIG. 8A is a schematic diagram when the vibration is observed, and an arrow in the figure represents a locus of movement of the fluorescent beads. In FIG. 8A, a rectangular area is an observation area. Here, the horizontal axis of the observation area is assumed to be the x-axis and the vertical axis is assumed to be the y-axis. FIG. 8B is a schematic diagram showing the movement of the fluorescent beads in the y direction in time series.

  First, the image processing unit 21 performs image processing on the entire observation image to detect the position of the fluorescent beads. Then, for each of the x-axis and the y-axis, time series data for at least one cycle of the biological vibration is stored in the memory 22 in the image processing unit 21 as shown in FIG. In the time-series data stored in the memory 22, the difference in the position of the fluorescent beads between successive image frames is calculated. That is, the position of the fluorescent beads at a certain imaging time point (frame) is compared with the position of the fluorescent beads at the next imaging time point (frame), and the difference is calculated. By this calculation, the movement amount P of the fluorescent beads between the image frames as shown in FIG. 9B is obtained. Next, as shown in FIG. 9B, the maximum value Pmax and the minimum value Pmin of the movement amount P are detected, and the maximum amplitude value PP is calculated and stored in the memory 22.

  Next, as shown in FIG. 10, a rectangular area having one side larger than the maximum movement amount PP of the fluorescent beads around the position of the fluorescent beads is set as the search area P. In the present embodiment, for convenience, the x-axis and y-axis of the search area P are set to be parallel to the x-axis and y-axis of the observation area, respectively. However, the setting method of the search area P is not limited to this, and can be set in an arbitrary direction. Further, the length of one side of the rectangular area is equal to or longer than PP. In order to reduce the area of the search region P, the length of one side of the rectangular region is preferably shorter. For example, the length of one side of the search region P is 6 times that of PP, more preferably 3 times, and even more preferably 2 times. When the vibration of the living body is stable, PP can be the length of one side of the search region P.

  After the search area P is set, an image only in the search area P centered on the position where the current fluorescent bead is detected is processed to detect fluorescent beads. The state at this time is shown in FIG. An arrow in FIG. 11 represents the locus of vibration of the fluorescent beads, and a dotted square represents the search region P. The search region P at a certain time point is set as described above with the position of the fluorescent bead detected one sampling time before, that is, the immediately preceding frame as the center. Since the fluorescent beads vibrate, when this procedure is repeated, as shown in FIG. 11, the search region P also moves according to the vibration. Since image processing is performed only within each search region P, the area of the image to be processed is significantly reduced, and therefore the image processing time can be greatly shortened.

  After the search area P is set in this way, the follow-up control of the objective lens 5 is started. At this time, as shown in FIG. 12, it is preferable to start the tracking control when the center of the vibration amplitude of the fluorescent bead is at the center position of the observation region. For this reason, the fluorescent beads are detected in time series, and the follow-up control of the objective lens 5 is started when the position comes near the center of the observation region. These will be described with reference to FIG.

13A shows biological vibration (movement of fluorescent beads), FIG. 13B shows movement of the objective lens, and FIG. 13C shows movement of fluorescent beads on the observation screen. The fluorescent beads are adjusted in advance so as to pass through the origin Po, which is the central portion, at any point of vibration.
In FIG. 13A, at time t1, the amplitude of the biological vibration passes through the origin Po at the center of the screen. When this is detected, as shown in FIG. 13 (b), the objective lens driving device 11 which has been in a stopped state until t1 starts driving to follow the vibration according to the control sequence of FIG. Start following vibration. As a result, as shown in FIG. 13C, the biological vibration is substantially canceled after time t1, and an image with less blurring effect is output on the observation screen. As a result, the fluorescent beads can be observed near the center of the observation region.

After the start of the follow-up control (after t1), a search area is further set.
First, as shown in FIG. 13C, time series data for at least one cycle of the vibration of the fluorescent beads is stored in the memory 22 after the control is started. In the time-series data stored in the memory 22, the difference in the position of the fluorescent beads is calculated between successive image frames. By this calculation, the movement amount Q of the fluorescent beads between the image frames as shown in FIG. 13D is obtained. Next, the maximum value Qmax and the minimum value Qmin of the movement amount Q are detected, and the maximum amplitude value QQ is calculated and stored in the memory 22.

  Then, as shown in FIG. 14, a rectangular area having a size equal to or larger than the maximum movement amount QQ of the fluorescent beads is set as the search area Q with the position of the fluorescent beads as the center. The length of one side of the search area Q is set as appropriate in the same manner as the search area P.

  As described above, after the tracking control of the objective lens 5 is started, the search area Q is set again, thereby further narrowing down the search area and further reducing the image processing area. Therefore, the image processing time can be further shortened.

  The search area is set at least twice for the search area P and the search area Q. The search area Q may be maintained after the start of tracking, or may be reset when observation conditions are different, such as stimulating a biological reaction or anesthesia. Or you may update a setting for every fixed time.

  As another embodiment, the search areas P and Q are not limited to a rectangle, but may be a circle as shown in FIG. Alternatively, as shown in FIG. 16, the rectangle may have a long side parallel to the vibration direction of the fluorescent beads. In this way, the search area can be further reduced.

  As another embodiment, when the tracking control of the objective lens 5 is started after the search region P is set, the tracking may be started when the fluorescent bead is located at a position other than the center position of the vibration amplitude. As shown in FIG. 17, since the amount of displacement is small in the portion other than the breathing portion, if the control is started in this portion, it can smoothly follow. Thereafter, the position of the fluorescent beads may be set to the center manually or automatically.

  By the way, as shown in FIG. 18, when the fluorescent bead exceeds the search area, the position of the fluorescent bead detected in the immediately preceding frame is output as the current position, and the tracking control system is disrupted due to missing data. prevent. In this case, the objective lens 5 remains at the immediately preceding position. If the fluorescent beads return to the search region in the next frame, the detection may be continued as it is. In the next frame, if the fluorescent beads are present outside the search area, the search area setting may be updated.

  Further, when the search area P is set, as shown in FIG. 19, the set search area may protrude outside the observation area. In such a case, as shown in FIG. 20, by setting an area (net-line portion) in the observation area as a search area, it is possible to prevent malfunction of the system.

  By the way, the calculation time varies depending on the image area processed in the image processing unit 21, that is, the number of pixels. For this reason, the stability of the control system is affected by the setting of the search area. The calculation delay due to image processing becomes a dead time of the feedback loop system. FIG. 21 shows a block diagram of a visual servo system that expresses this dead time. The open loop frequency characteristics of this feedback system are represented by a Bode diagram shown in FIG. At the gain crossover frequency where the gain characteristic crosses 0 dB, the stability of the control system is determined by the phase margin indicating how much the phase has a margin from -180 degrees. Therefore, depending on the amount of dead time, the phase characteristic changes as shown in FIG. For example, as the number of pixels increases and the processing time increases, the dead time increases and the phase margin decreases. Therefore, the stability of the control system is deteriorated. Therefore, by reducing the control gain, the gain crossover frequency at which the gain characteristic crosses 0 dB moves, and a phase margin can be secured. This maintains the stability of the control system.

  From the above, it is preferable that the time required for image processing is measured in advance according to the number of pixels to be processed and stored in the memory 23 as a lookup table. In consideration of the phase state of the control system according to the calculation time, when the processing time is long, that is, when the search region is wide, the drive control device 18 lowers the control gain to ensure the stability of the control system. Then, by increasing the control gain in accordance with the shortening of the calculation time, the accuracy of the control system can be improved as much as the dead time is shortened. Thus, the control system can be stabilized by appropriately setting the control gain according to the length of the calculation time.

Next, a microscope observation apparatus 100 according to the second embodiment of the present invention will be described with reference to FIGS.
In the description of the present embodiment, the same reference numerals are given to portions having the same configuration as the microscope observation apparatus 1 according to the first embodiment described above, and the description thereof is omitted.

  In the present embodiment, an X-Y address type image sensor such as a CMOS as shown in FIG. An imaging region is set in advance by the control device 24 for the detection imaging device 9, and only the imaging region is imaged. As the imaging area, an area where the fluorescent beads move can be arbitrarily set. For example, the search area may be set as the imaging area. The captured image data is transferred to the image processing unit 21. By limiting the imaging area, not only the image processing time in the image processing unit 21 but also the transfer time of the image data can be shortened. Therefore, the delay time in the follow-up control is further shortened. The setting of the imaging area in the present embodiment is preferably performed after the setting of the search area Q.

  In the first and second embodiments described above, the search area is set in the image processing unit 21 and the displacement amount and the direction of the sample S are calculated and sent to the objective lens driving device 11 in the drive control device 18. However, the image processing unit 21 may only set the search area, and the displacement amount and direction of the sample may be calculated by the drive control device 18.

  As described above, the configuration and operation of the present invention have been described using a microscope as an example. However, the scope of the present invention is not limited to a microscope. Any one is included in the present invention.

  In the above embodiment, the case where there is only one fluorescent bead in the observation region has been described. However, as shown in FIG. 25, when there are a plurality of fluorescent beads in the observation region, which fluorescent bead should be tracked. Set the operation. In this case, the angle of view is manually adjusted, or the range is designated in the initial setting of the tracking area.

The typical whole block diagram which shows the microscope observation apparatus which concerns on embodiment of this invention. The schematic diagram which shows the vibration of a biological body in time series. The perspective view which shows the relationship between the objective lens and coordinate axis in description of the microscope observation apparatus of FIG. The schematic diagram which shows the marker in an observation visual field. The schematic diagram which shows the area | region which a marker occupies on an image sensor. The figure which shows the flowchart of a biological vibration detection. The control block diagram of an objective-lens control system. The schematic diagram when a biological vibration is observed without tracking control. (A) represents time-series data of biological vibration for one period, and (b) represents the amount of movement of fluorescent beads between image frames. The schematic diagram of the search area | region set based on Pmax. The schematic diagram which shows a response | compatibility with the displacement of a fluorescent bead, and a search area | region. The schematic diagram which shows the state which has a fluorescent bead in the center position of an observation area | region. The schematic diagram in the case of starting tracking control at the center of vibration amplitude. The schematic diagram which set the search area | region Q. FIG. The schematic diagram whose search area | region is circular. The schematic diagram whose search area is a rectangle. The schematic diagram at the time of starting tracking control in parts other than the center of vibration amplitude. The figure which shows the case where a fluorescent bead exists outside a search area | region. The figure which shows the case where a search area | region protrudes out of the observation area | region. The figure which shows the area | region searched when a search area | region protrudes outside an observation area | region. The block diagram which shows a visual servo system. The Bode diagram which shows the relationship between stability of a control system, and a gain characteristic. The typical whole block diagram which shows the microscope observation apparatus which concerns on 2nd Embodiment. 1 is a diagram illustrating an embodiment of an image sensor used in an imaging apparatus. The schematic diagram which shows the case where a some fluorescent bead exists in an observation area | region.

Explanation of symbols

  S ... sample, E ... excitation light (light), F ... fluorescence (return light), 1,100 ... microscope observation device, 3 ... excitation light source (light source), 4 ... illumination lens (illumination optical system), 5 ... objective lens , 8 ... Observation imaging device, 9 ... Detection imaging device, 10 ... Microscope main body, 11 ... Objective lens drive device, 18 ... Drive control device, 21 ... Image processing unit, 22, 23 ... Memory, 24 ... Control device.

Claims (7)

An objective lens placed close to the sample;
An imaging device for imaging a sample;
An image processing unit that sets a search region in an image captured by the imaging device and processes an image in the search region;
A biological observation apparatus comprising: an objective lens driving device that drives the objective lens in a direction in which the displacement of the sample is corrected based on a signal from the image processing unit.   The living body observation apparatus according to claim 1, further comprising a control device that controls the imaging device so as to capture only a predetermined region.   The imaging apparatus includes a detection imaging apparatus and an observation imaging apparatus for detecting a displacement of a sample and capturing an image in which a search region is set by the image processing unit. 2. The biological observation apparatus according to 2.   The living body observation apparatus according to any one of claims 1 to 3, wherein the search area is a rectangular area having a length that exceeds a maximum displacement difference between successive image frames.   The biological observation apparatus according to claim 1, wherein the search area is set before and after the driving of the objective lens driving apparatus is started.   The living body observation apparatus according to claim 1, wherein the living body observation apparatus is a microscope for living body observation.   The living body observation apparatus according to any one of claims 1 to 6, wherein the sample is a cell, tissue, or viscera in a living body.

Images