JP5893315B2 - Microscope system - Google Patents

Description

  The present invention relates to a microscope system for displaying an image corresponding to image data generated by imaging a specimen sample placed on a stage using an objective lens.

  2. Description of the Related Art Conventionally, a technique for generating a single combined image wider than the observation region of an imaging device by combining a plurality of still images taken while changing the field of view for observing a specimen is known (patent) Literature 1 and Patent literature 2). In this technique, a live image continuously generated by the imaging device is superimposed on a specimen sample bonded image displayed on a display monitor, and the positional relationship of the specimen sample to be bonded next is displayed using this live image. This makes it possible to create a bonded image of a desired specimen.

JP 2010-141697 A JP 2010-141698 A

  However, in the above-described technique, when the user creates a composite image of a specimen sample, the entire composite image including the live image fits in the display area in order to determine the position of the live image in the entire composite image. In order to display the entire image at a reduced size and to grasp the detailed position of the live image on the composite image, the display magnification of the composite image including the live image is fixed and the live image is the center. When displaying an image, it is necessary to switch the display each time and display it on a display monitor, which causes a problem that the operation becomes complicated.

  The present invention has been made in view of the above, and an object of the present invention is to provide a microscope system that can easily create a desired bonded image by a simple operation.

  In order to solve the above-mentioned problems and achieve the object, a microscope system according to the present invention mounts a specimen sample, images a stage that can be moved in a horizontal direction, and the specimen specimen placed on the stage. An imaging device that continuously generates image data of the specimen sample, a display unit that can display an image corresponding to the image data generated by the imaging device, and the image data that is continuously generated by the imaging device A live image generation unit that sequentially generates live images to be displayed on the display unit, and a relative position of the observation region of the imaging apparatus with respect to the stage based on the temporally continuous image data generated by the imaging apparatus. Based on the movement amount calculated by the movement amount calculation unit and the movement amount calculation unit that calculates the movement amount, the view of the imaging device is obtained from the image data continuously generated by the imaging device. A combined image generation unit that generates a combined image data having an observation region having a wider outer edge than the region; a live frame generation unit that generates a live position frame indicating the observation region of the imaging device in the display region of the display unit; A live image corresponding to the live image data and a composite image corresponding to the composite image data are displayed in different display areas, respectively, and based on the travel amount calculated by the travel amount calculation unit, And a display control unit that superimposes the live position frame on the display position on the composite image corresponding to the observation area of the imaging device and displays the frame on the display unit.

  In the above invention, the microscope system according to the present invention is located around the live image in the display area of the live image displayed by the display unit based on the movement amount calculated by the movement amount calculation unit. The display unit further includes a peripheral image generation unit that generates a peripheral image by cutting out the region of the combined image, and the display control unit combines the peripheral image with the periphery of the live image displayed by the display unit. It is characterized by being displayed.

  The microscope system according to the present invention further includes a movement amount determination unit that determines whether or not the movement amount calculation unit was able to calculate the movement amount in the above invention, and the display control unit includes the display control unit, When the movement amount determination unit determines that the movement amount calculation unit has not been able to calculate the movement amount, the display unit displays information indicating that the movement amount has not been calculated. It is characterized by being displayed.

  Further, in the microscope system according to the present invention, in the above invention, when the display control unit determines that the movement amount calculation unit cannot calculate the movement amount by the movement amount determination unit, the live control unit The display mode of the position frame is changed and displayed on the display unit.

  The microscope system according to the present invention further includes a display size switching unit that receives an input of an instruction signal for switching a display size of the composite image displayed by the display unit in the above invention, and the display control unit includes the display control unit, The composite image and the live position frame are each displayed on the display unit with the display size corresponding to an instruction signal input from a display size switching unit.

  The microscope system according to the present invention further includes a scale display selection unit that receives an input of an instruction signal for selecting whether or not to display a scale for measuring the actual size of the bonded image in the above-described invention, and the display control unit Is characterized in that the scale is displayed on the display unit in response to an instruction signal input from the scale display selection unit.

  In the microscope system according to the present invention, in the above invention, the indication signal for selecting whether or not to display grid lines indicating grids formed at predetermined intervals on the display area where the display unit displays the bonded image. A grid display selection unit that receives the input, and the display control unit displays the grid line on the display unit in response to an instruction signal input from the grid display selection unit.

  In the microscope system according to the present invention, in the above invention, a display position selection unit that receives an input of an instruction signal for selecting a reference position of the live position frame on a display area where the display unit displays the composite image. In addition, the display control unit causes the display unit to display the live position frame at a reference position corresponding to an instruction signal input from the display position selection unit.

  According to the microscope system of the present invention, the display control unit displays the live image and the composite image in different display regions, and the relative movement of the observation region of the imaging device with respect to the stage calculated by the movement amount calculation unit Based on the amount, the live position frame is superimposed on the display position on the composite image corresponding to the observation area of the current imaging device and displayed on the display unit. Accordingly, the user can easily identify the region to be pasted on the composite image and position the live image by confirming the composite image and the live image displayed on the display unit. As a result, it is possible to efficiently create a bonded image of the specimen sample.

FIG. 1 is a conceptual diagram showing an example of the configuration of a microscope system according to an embodiment of the present invention. FIG. 2 is a block diagram showing a functional configuration of the microscope system according to the embodiment of the present invention. FIG. 3 is a diagram illustrating an example of an operation screen displayed by the display unit of the microscope system according to the embodiment of the present invention. FIG. 4 is a flowchart showing an outline of processing performed by the microscope system according to the embodiment of the present invention. FIG. 5 is a diagram schematically illustrating processing performed by the microscope system according to the embodiment of the present invention. FIG. 6 is a diagram illustrating another example of the operation screen displayed by the display unit of the microscope system according to the second modification of the embodiment of the present invention. FIG. 7 is a diagram illustrating another example of the operation screen displayed by the display unit of the microscope system according to the third modification of the embodiment of the present invention. FIG. 8A is a diagram illustrating another example of the operation screen displayed on the display unit of the microscope system according to the fourth modification of the embodiment of the present invention. FIG. 8B is a diagram illustrating another example of the operation screen displayed on the display unit of the microscope system according to the fourth modification of the embodiment of the present invention. FIG. 9 is a diagram illustrating another example of the operation screen displayed by the display unit of the microscope system according to the fifth modification of the embodiment of the present invention.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention (hereinafter referred to as “embodiments”) will be described with reference to the drawings. The present invention is not limited to the embodiments described below. Further, in the description of the drawings, the same portions will be described with the same reference numerals.

  FIG. 1 is a conceptual diagram showing an example of the configuration of a microscope system according to an embodiment of the present invention. FIG. 2 is a block diagram showing a functional configuration of the microscope system according to the embodiment of the present invention. 1 and 2, the plane on which the microscope system 1 is placed will be described as an XY plane, and a direction perpendicular to the XY plane will be described as a Z direction.

  As shown in FIG. 1 and FIG. 2, the microscope system 1 images the specimen sample S through the microscope apparatus 2 that observes the specimen sample S, the microscope control unit 3 that drives and controls the microscope apparatus 2, and the microscope apparatus 2. The image pickup apparatus 4 that generates image data, the image pickup control unit 5 that controls the drive of the image pickup apparatus 4, and the image corresponding to the image data picked up by the image pickup apparatus 4 via the control terminal 7 are displayed, and the microscope system The display input part 6 which receives the input of 1 various operation, and the control terminal 7 which controls each part of the microscope system 1 are provided. The microscope device 2, the microscope control unit 3, the imaging device 4, the imaging control unit 5, the display input unit 6, and the control terminal 7 are connected by wire or wireless so that data can be transmitted and received.

  The microscope apparatus 2 has a stage 21 on which a specimen S is placed, a substantially C-shape in side view, a stage body 24 that supports the stage 21 and holds an objective lens 23 via a revolver 22; And an epi-illumination light source 25 that irradiates the specimen S with light.

  The stage 21 is configured to be movable in the XYZ directions. The stage 21 is movable in the XY plane by the stage driving unit 211. Under the control of the microscope control unit 3, the stage 21 detects a predetermined origin position on the XY plane by an XY position origin sensor (not shown), and the drive amount of the stage drive unit 211 is controlled based on this origin position. Thus, the observation location (observation region) on the specimen sample S is moved. The stage 21 outputs position signals (XY coordinates) relating to the X position and the Y position during observation to the microscope control unit 3. Further, the stage 21 is movable in the Z direction by a motor 212. Under the control of the microscope control unit 3, the stage 21 detects a predetermined origin position in the Z direction of the stage 21 by a Z position origin sensor (not shown), and the driving amount of the motor 212 is controlled based on this origin position. As a result, the specimen sample S is moved to an arbitrary Z position within a predetermined height range. The stage 21 outputs a position signal related to the Z position during observation to the microscope control unit 3.

  The revolver 22 is provided so as to be slidable or rotatable with respect to the microscope main body 24, and the objective lens 23 is disposed above the specimen S. The revolver 22 is configured using a nose piece, a swing revolver, or the like. The revolver 22 holds a plurality of objective lenses 23 having different magnifications (observation magnifications) by the mounter 221. The revolver 22 is inserted in the optical path of the observation light, and the connection between the revolver 22 and the revolver driving unit 222 that slides or rotates the mounter 221 to selectively switch the objective lens 23 used for observing the specimen S. And a revolver detector 223 for detecting a state and the like.

  The revolver driving unit 222 slides or rotates the mounter 221 under the control of the microscope control unit 3. The revolver detection unit 223 includes a revolver connection sensor (not shown) that detects that the revolver 22 is connected to the microscope main body 24, and the type of the objective lens 23 in which the objective lens 23 is inserted in the optical path of the observation light. And a movement completion sensor (not shown) for detecting that the objective lens 23 is inserted on the optical path of the observation light. The revolver detection unit 223 outputs detection results detected by various sensors to the microscope control unit 3.

  The objective lens 23 includes, for example, an objective lens 231 with relatively low magnification of 1 ×, 2 ×, and 4 × (hereinafter referred to as “low magnification objective lens 231”), and a 10 ×, 20 ×, and 40 × low magnification objective lens. At least one objective lens 232 (hereinafter referred to as “high-magnification objective lens 232”) having a high magnification relative to the magnification of 231 is attached to the mounter 221. Note that the magnifications of the low-magnification objective lens 231 and the high-magnification objective lens 232 are examples, and it is sufficient that the high-magnification objective lens 232 is higher than the low-magnification objective lens 231.

  The microscope main body 24 includes an illumination lens 241 that collects the illumination light L1 emitted from the epi-illumination light source 25 through the fiber 251 (hereinafter referred to as “epi-illumination light L1”), and an optical path of the epi-illumination light L1. The half mirror 242 that deflects along the optical axis of the objective lens 23, the zoom lens unit 243 that magnifies the sample S, and the sample sample S incident through the objective lens 23, the zoom lens unit 243, and the half mirror 242. An imaging lens 244 that focuses the reflected light to form an observation image is provided inside.

  The zoom lens unit 243 includes one or a plurality of lenses, and includes a zoom optical system 243a that can zoom the specimen sample S, and a zoom drive unit 243b that drives the zoom optical system 243a along the optical axis. The zoom drive unit 243b changes the zoom magnification of the zoom lens unit 243 by moving the zoom optical system 243a along the optical axis under the control of the microscope control unit 3.

  The epi-illumination light L1 is applied to the specimen S through the illumination lens 241, the half mirror 242, the zoom optical system 243a, and the objective lens 23. The reflected light L2 reflected by the specimen S (hereinafter referred to as “observation light L2”) enters the imaging device 4 through the objective lens 23, the zoom optical system 243a, the half mirror 242, and the imaging lens 244.

  The epi-illumination light source 25 includes a halogen lamp, a xenon lamp, an LED (Light Emitting Diode), or the like. The epi-illumination light source 25 emits epi-illumination light L <b> 1 for forming an observation image of the specimen S through the fiber 251 to the microscope main body 24.

  The microscope control unit 3 is configured using a CPU (Central Processing Unit) or the like, and comprehensively controls the operation of each unit configuring the microscope apparatus 2 under the control of the control terminal 7. Specifically, the microscope control unit 3 drives the revolver driving unit 222 to rotate the mounter 221 to switch the objective lens 23 arranged on the optical path of the observation light L2, the stage driving unit 211 or the motor By driving 212, the stage 21 driving process, the adjustment process for adjusting each part of the microscope apparatus 2 accompanying the observation of the specimen S, and the like are performed. In addition, the microscope control unit 3 informs the control terminal 7 of the state of each part constituting the microscope apparatus 2, for example, the position information (XY position, Z position) of the stage 21 and the type information of the objective lens 23 attached to the revolver 22. Output.

  The imaging device 4 is configured using an imaging device 41 such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). Under the control of the imaging control unit 5, the imaging device 4 captures the observation image of the specimen sample S incident through the imaging lens 244 and continuously generates image data of the specimen sample S. The imaging device 4 outputs the image data of the specimen sample S generated via the camera cable to the control terminal 7.

  The imaging control unit 5 is configured using a CPU or the like, and controls the operation of the imaging device 4. Specifically, the imaging control unit 5 controls the photographing operation of the imaging device 4 by performing ON / OFF switching processing of automatic gain control of the imaging device 4, gain setting processing, frame rate setting processing, and the like. The imaging control unit 5 includes an AE processing unit 51 and an AF processing unit 52.

  The AE processing unit 51 automatically sets the exposure condition of the imaging device 4 based on the image data generated by the imaging device 4. Specifically, the AE processing unit 51 calculates the luminance from the image data acquired via the control terminal 7, and determines the exposure condition of the imaging device 4, for example, the exposure time based on the calculated luminance. AE processing for automatically adjusting 4 is performed.

  The AF processing unit 52 automatically adjusts the focus of the imaging device 4 based on the image data generated by the imaging device 4. Specifically, the AF processing unit 52 automatically adjusts the focus of the imaging device 4 by evaluating the contrast included in the image data and detecting the in-focus position (focal position). AF processing is performed. Note that the AF processing unit 52 may detect a focused focus position (Z position) by evaluating the contrast of the image at each Z position of the stage 21 based on the image data.

  The display input unit 6 includes a display communication unit 61 that performs communication with the control terminal 7, a display unit 62 that displays an image, and a touch panel 63 that outputs a position signal corresponding to an external object contact.

  The display communication unit 61 is a communication interface for performing communication with the control terminal 7. The display communication unit 61 outputs the image data output from the control terminal 7 to the display unit 62.

  The display unit 62 is configured using a display panel made of liquid crystal, organic EL (Electro Luminescence), or the like. The display unit 62 displays an image corresponding to image data input via the display communication unit 61 and various operation information of the microscope system 1. Specifically, the display unit 62 combines the live image corresponding to the image data continuously generated by the imaging device 4 and the specimen sample S having an observation region whose outer edge is wider than the observation region of the imaging device 4. The combined image corresponding to the data is displayed in a different display area. Details of the display area of the live image and the combined image displayed on the display unit 62 will be described later.

  The touch panel 63 is provided on the display screen of the display unit 62 and accepts an input according to the contact position of the object from the outside. Specifically, the touch panel 63 detects a position where the user touches (contacts) according to an operation icon displayed on the display unit 62, and outputs a position signal corresponding to the detected touch position to the control terminal 7. The touch panel 63 functions as a graphical user interface (GUI) when the display unit 62 displays various operation information of the microscope system 1 in the display area. In general, the touch panel includes a resistance film method, a capacitance method, an optical method, and the like. In the present embodiment, any type of touch panel is applicable.

  The control terminal 7 includes a control communication unit 71 that communicates with the microscope control unit 3, the imaging control unit 5, and the display input unit 6, a storage unit 73 that stores various types of information of the microscope system 1, and each unit of the microscope system 1. An input unit 72 that receives an input of a drive instruction signal that instructs driving, and a control unit 74 that controls each unit of the microscope system 1 are provided.

  The control communication unit 71 is a communication interface for communicating with each of the microscope control unit 3, the imaging control unit 5, and the display input unit 6. Further, the control communication unit 71 outputs the image data output from the imaging device 4 to the control unit 74 via the camera cable.

  The input unit 72 is configured using a keyboard, a mouse, a joystick, various switches, and the like, and outputs operation signals corresponding to operation inputs of the various switches to the control unit 74.

  The storage unit 73 is realized by using a semiconductor memory such as a flash memory and a RAM (Random Access Memory) fixedly provided inside the control terminal 7. The storage unit 73 stores various programs to be executed by the microscope system 1 and various data used during execution of the programs. The storage unit 73 temporarily stores information being processed by the control unit 74. The storage unit 73 includes an image data storage unit 731 that stores image data captured by the imaging device 4. In addition, the memory | storage part 73 may be comprised using the memory card etc. which are mounted | worn from the outside.

  The control unit 74 is configured using a CPU or the like, and performs a command or data transfer corresponding to each unit constituting the microscope system 1 in accordance with an operation signal from the input unit 72 and a position signal from the touch panel 63, and the microscope system. The operation of 1 is comprehensively controlled.

  A detailed configuration of the control unit 74 will be described. The control unit 74 includes a live image generation unit 741, a movement amount calculation unit 742, a movement amount determination unit 743, a live frame generation unit 744, a captured image generation unit 745, a combined image generation unit 746, and a peripheral image. A generation unit 747, a drive control unit 748, and a display control unit 749 are included.

  The live image generation unit 741 sequentially generates live image data to be displayed on the display unit 62 from the image data continuously generated by the imaging device 4. The live image generation unit 741 outputs live image data to the image data storage unit 731. The live image generation unit 741 also performs predetermined image processing, such as optical black subtraction processing, white balance adjustment processing, synchronization processing, color matrix calculation processing, γ correction processing, color reproduction processing, and edge enhancement processing, on image data. Image processing including the above is performed.

  The movement amount calculation unit 742 is based on image data corresponding to temporally continuous image data generated by the imaging device 4 input via the control communication unit 71 in the observation region of the imaging device 4 with respect to the stage 21. The relative movement amount is calculated. Specifically, the movement amount calculation unit 742 extracts feature points included in each of the live images corresponding to the two pieces of image data moving forward and backward in time, and extracts a plurality of corresponding points that are the same between the two live images. By calculating the amount of movement (number of pixels), the amount of movement (shift amount) relative to the stage 21 in the observation region of the imaging device 4 is calculated. Here, the feature point is luminance information, color information, or edge information. Note that the movement amount calculation unit 742 may calculate the relative movement amount in the observation region of the imaging device 4 using a known technique such as pattern matching.

  The movement amount determination unit 743 determines whether or not the movement amount calculation unit 742 can calculate the relative movement amount in the observation region of the imaging device 4 with respect to the stage 21 based on the two live images that move back and forth in time. . Specifically, the movement amount determination unit 743 determines whether or not the movement amount calculation unit 742 has been able to extract a corresponding feature point between two live images. For example, when the movement amount calculation unit 742 cannot extract feature points corresponding to each other between two live images, the movement amount determination unit 743 observes the imaging apparatus 4 with respect to the stage 21. It is determined that the relative movement amount in the region could not be calculated.

  The live frame generation unit 744 generates a live position frame indicating the observation area of the imaging device 4. Specifically, the live frame generation unit 744 generates a rectangular frame as a live position frame indicating the observation area of the imaging device 4.

  The captured image generation unit 745 performs predetermined image processing on the image data input via the control communication unit 71 and outputs the image data to the composite image generation unit 746 and the image data storage unit 731. Specifically, the captured image generation unit 745 performs optical black subtraction processing, white balance adjustment processing, synchronization processing, color matrix calculation processing, γ correction processing, color reproduction processing, edge enhancement processing, and the like on the image data. Including image processing. The photographed image generation unit 745 compresses the image data by a predetermined method, for example, JPEG (Joint Photographic Experts Group) method, and outputs the compressed image data to the combined image generation unit 746 and the image data storage unit 731. Also good.

  Based on the relative movement amount of the observation area of the imaging device 4 with respect to the stage 21 calculated by the movement amount calculation unit 742, the bonded image generation unit 746 uses the imaging device 4 from image data continuously generated by the imaging device 4. Bonded image data having an observation area whose outer edge is wider than the observation area is generated. Specifically, the composite image generation unit 746 acquires the composite image data stored in the image data storage unit 731, and the movement amount calculation unit 742 calculates the captured image data output from the captured image generation unit 745. The composite image data is generated by combining the display position of the composite image determined by the relative movement amount of the observation region of the imaging device 4 with respect to the stage 21. For example, the composite image generation unit 746 performs blending processing of pixel values on the overlapping area between the composite image and the captured image in order to make the joint inconspicuous when the captured image is combined with the composite image. A stitched image is generated. Here, the blending process is an image process for calculating a pixel value of a composite image by weighted averaging of pixel values between images. Note that the composite image generation unit 746 may change the weight at the time of the weighted average in the blending process according to the pixel position.

  The peripheral image generation unit 747 is based on the relative movement amount of the observation area of the imaging device 4 with respect to the stage 21 calculated by the movement amount calculation unit 742, and the peripheral area of the live image is displayed in the display area of the live image displayed by the display unit 62. The peripheral image corresponding to is cut out from the combined image data to generate the peripheral image data.

  The drive control unit 748 controls the driving of the microscope device 2 and the imaging device 4 based on the operation signal input from the input unit 72 or the position signal and instruction signal input from the touch panel 63. Specifically, when a shooting instruction signal for instructing shooting is input from the input unit 72, the drive control unit 748 outputs a command signal for instructing shooting to the imaging control unit 5, thereby shooting the imaging device 4. Run the action.

  The display control unit 749 controls the display mode of the display unit 62. Specifically, the display control unit 749 differs between the live image generated by the live image generation unit 741 and the composite image generated by the composite image generation unit 746 via the control communication unit 71 and the display communication unit 61. Each is displayed in the display area of the display unit 62. Based on the movement amount of the observation area of the imaging device 4 with respect to the stage 21 calculated by the movement amount calculation unit 742, the display control unit 749 displays the display position on the composite image corresponding to the observation area of the current imaging device 4. The live position frame generated by the live frame generation unit 744 is superimposed and displayed on the display unit 62.

  Here, an operation screen of the microscope system 1 displayed on the display unit 62 under the control of the display control unit 749 will be described. FIG. 3 is a diagram illustrating an example of an operation screen of the microscope system 1 displayed on the display unit 62.

  As shown in FIG. 3, the operation screen W1 displayed by the display unit 62 includes a bonded image display unit W10, a live image display unit W11, a photographing instruction unit W12, a completion instruction unit W13, and a reference position selection unit W14. And a scale display selection unit W15, a display size switching unit W16, and a grid display selection unit W17.

  The composite image display unit W10 is controlled by the display control unit 749, and the live position generated by the composite image P1 and the live frame generation unit 744 corresponding to the composite image data generated by the composite image generation unit 746. A frame B1 is displayed.

  The live image display unit W11 corresponds to the live image P2 corresponding to the live image data generated by the live image generation unit 741 and the peripheral image data generated by the peripheral image generation unit 747 under the control of the display control unit 749. The surrounding image P3 to be displayed is displayed. The live image display unit W11 displays the live image P2 fixed in the central area. The peripheral image P3 is moved and displayed with respect to the live image P2 displayed in a fixed manner.

  The photographing instruction unit W12 receives an input of a photographing instruction signal that instructs photographing by the imaging device 4. Specifically, the photographing instruction unit W12 moves the mouse pointer K1 over the display area of the photographing instruction unit W12 by the user operating the mouse or the like of the input unit 72, and performs a drag operation (hereinafter, “drag operation”). The input of the shooting instruction signal is accepted.

  The completion instruction unit W13 receives an input of an end instruction signal for ending generation of the bonded image of the specimen sample S when the user performs a drag operation with the mouse or the like of the input unit 72 and is selected.

  The reference position selection unit W14 is a reference position selection list box for selecting a display position where the live position frame B1 is displayed in the display area of the bonded image display unit W10. The reference position selection unit W14 specifies the display position of the live position frame B1 by the user dragging the reference position selection list box W141 with the mouse or the like of the input unit 72 to select the display position of the live position frame B1. Accepts input of display position designation signal.

  The scale display selection unit W15 is a selection button for setting display selection of a scale for measuring the actual size of the combined image P1 on the display area of the combined image display unit W10. The scale display selection unit W15 receives an input of an instruction signal for causing the scale to be displayed in the display area of the bonded image display unit W10 when the user drags and selects the ON switch W151. On the other hand, when the user performs a drag operation and selects the OFF switch W152, an input of an instruction signal to hide the scale in the display area of the combined image display unit W10 is received.

  The display size switching unit W16 is a display size switching button for switching the display size of the composite image P1 displayed on the display area of the composite image display unit W10. The display size switching unit W16 receives an input of an instruction signal for switching the display size of the composite image P1 when the user performs a drag operation to select the list box W161 and select a desired display size.

  The grid display selection unit W17 is a grid setting button for setting display selection of grid lines (guidelines) formed at a predetermined interval in the display area of the bonded image display unit W10. The grid display selection unit W17 is a guide division number selection list box that switches the division number of the grid lines. The grid display selection unit W17 accepts an input of an instruction signal for the number of grid line divisions when the user drags and selects the list box W171.

  The microscope system 1 configured in this manner displays an image of the sample sample S on the display unit 62 by displaying the image data of the sample sample S imaged by the imaging device 4 under the control of the control unit 74. Can be observed. Furthermore, in the microscope system 1, the microscope device 2 and the imaging device 4 are driven when the control unit 74 outputs an instruction signal or a drive signal to each unit of the microscope system 1 based on an operation signal input from the input unit 72. To do.

  Next, operations performed by the microscope system 1 will be described. FIG. 4 is a flowchart showing an outline of processing performed by the microscope system 1 according to the present embodiment. In the following, a bonded image generation mode for generating a bonded image of the specimen sample S performed by the microscope system 1 will be described.

  As shown in FIG. 4, the live frame generation unit 744 generates a live position frame B1 corresponding to the current observation area of the imaging device 4 displayed on the bonded image display unit W10 (step S101).

  Subsequently, the display control unit 749 displays the live position frame B1 generated by the live frame generation unit 744 at the reference position of the bonded image display unit W10 set by the reference position selection unit W14 (step S102). Specifically, as illustrated in FIG. 5A, the display control unit 749 displays the live position frame B1 on the combined image display unit W10 based on the position selected by the reference position selection unit W14.

  Thereafter, the live image generation unit 741 generates live image data to be displayed on the display unit 62 from the image data generated by the imaging device 4 (step S103).

  Subsequently, the display control unit 749 displays a live image corresponding to the live image data generated by the live image generation unit 741 on the display area of the live image display unit W11 (step S104). Specifically, as shown in FIG. 5A, the display control unit 749 displays the live image P2 of the specimen sample S generated by the live image generation unit 741 on the upper left on the display area of the live image display unit W11. Let

  Thereafter, the control unit 74 determines whether or not the live image display unit W11 displays the live image for the first time (step S105). When the control unit 74 determines that the live image display by the live image display unit W11 is the first time (step S105: Yes), the microscope system 1 returns to step S103. On the other hand, when the control unit 74 determines that the live image display by the live image display unit W11 is not the first time (step S105: No), the microscope system 1 proceeds to step S106.

  After step S105, the movement amount calculation unit 742 calculates the relative movement amount of the observation area of the imaging device 4 with respect to the stage 21 based on the latest live image and the immediately preceding live image (step S106). Specifically, the movement amount calculation unit 742 is an imaging device for the stage 21 based on the latest live image generated by the live image generation unit 741 and the previous live image stored by the image data storage unit 731. The relative movement amount in the four observation areas is calculated.

  Subsequently, the movement amount determination unit 743 determines whether or not the movement amount calculation unit 742 has been able to calculate the relative movement amount of the observation area of the imaging device 4 with respect to the stage 21 (step S107). Specifically, the movement amount determination unit 743 extracts the corresponding points corresponding to each of a plurality of feature points extracted from the previous live image by the movement amount calculation unit 742, thereby capturing the stage 21. It is determined whether or not the movement amount of the observation area of the device 4 has been calculated. For example, when the movement amount by which the stage 21 moves is large, the movement amount determination unit 743 cannot extract the corresponding points corresponding to the feature points extracted from the previous live image by the movement amount calculation unit 742 from the latest live image. The movement amount calculation unit 742 determines that the relative movement amount of the observation region of the imaging device 4 with respect to the stage 21 could not be calculated. When the movement amount determination unit 743 determines that the movement amount calculation unit 742 can calculate the relative movement amount of the observation region of the imaging device 4 with respect to the stage 21 (step S107: Yes), the microscope system 1 is described later. The process proceeds to step S108. On the other hand, when the movement amount determination unit 743 determines that the movement amount calculation unit 742 could not calculate the relative movement amount of the observation region of the imaging device 4 with respect to the stage 21 (step S107: No), the microscope system 1 Shifts to step S120 to be described later.

  In step S <b> 108, the drive control unit 748 stores the latest live image data generated by the live image generation unit 741 in the image data storage unit 731.

  Subsequently, the control unit 74 determines whether or not the combined image data is stored in the image data storage unit 731 (step S109). When the control unit 74 determines that the combined image data is stored in the image data storage unit 731 (step S109: Yes), the microscope system 1 proceeds to step S110 described later. On the other hand, when the control unit 74 determines that the combined image data is not stored in the image data storage unit 731 (step S109: No), the microscope system 1 proceeds to step S113 described later.

  In step S110, the peripheral image generation unit 747 generates a peripheral image corresponding to the periphery of the live image P2 in the display area of the live image display unit W11 based on the movement amount calculated by the movement amount calculation unit 742. To generate peripheral image data by cutting out from the combined image data stored. Specifically, the peripheral image generation unit 747 acquires the combined image data stored in the image data storage unit 731 and then captures the image of the stage 21 calculated by the movement amount calculation unit 742 for the combined image data. The peripheral area of the live image P2 is specified from the relative movement amount of the four observation areas, and the specified peripheral area is cut out from the combined image data (trimming process) to generate the peripheral image data.

  Thereafter, the display control unit 749 displays a peripheral image corresponding to the peripheral image data generated by the peripheral image generation unit 747 around the live image P2 displayed in the display area of the live image display unit W11 (step S111). Based on the relative movement amount of the observation area of the imaging device 4 calculated by the movement amount calculation unit 742, the display position of the live position frame B1 on the display area of the bonded image display unit W10 is updated (step S112). . Specifically, as shown in FIGS. 5B and 5C, the display control unit 749 performs the pasting corresponding to the periphery of the live image P2 displayed on the display area of the live image display unit W11. The peripheral image P3 of the image P1 is virtually pasted and displayed, and the display position of the live position frame B1 on the display area of the composite image display unit W10 is updated. As a result, the user can confirm the live image P2 and the peripheral image P3 with a sufficient display size. Therefore, when the composite image P1 is created, the accuracy of the joint between the joints is increased and the composite image P1 is displayed. Can be created.

  Subsequently, the control unit 74 determines whether or not a shooting instruction signal is input by operating the shooting instruction unit W12 via the input unit 72 (step S113). When the control unit 74 determines that an imaging instruction signal has been input (step S113: Yes), the microscope system 1 proceeds to step S114. On the other hand, when the control unit 74 determines that the imaging instruction signal is not input from the imaging instruction unit W12 via the input unit 72 (step S113: No), the microscope system 1 returns to step S103.

  In step S114, the drive control unit 748 stops the live image data generation by the live image generation unit 741.

  Subsequently, the drive control unit 748 outputs a shooting drive signal to the imaging control unit 5 and causes the imaging device 4 to perform a shooting operation to generate image data (step S115).

  Thereafter, the captured image generation unit 745 generates captured image data from the image data generated by the imaging device 4 (step S116). At this time, the captured image generation unit 745 generates captured image data from one frame of image data generated by the imaging device 4. Further, when the imaging device 4 generates a plurality of frames of image data, the captured image generation unit 745 may generate omnifocal image data and 3D image data from the plurality of frames of image data.

  Subsequently, the composite image generation unit 746 acquires the composite image data from the image data storage unit 731 and the captured image data from the captured image generation unit 745, and then combines the composite image from the movement amount calculated by the movement amount calculation unit 742. The display position of the captured image data on the image P1 is calculated, and the captured image data generated by the captured image generation unit 745 is combined with the calculated display position, thereby generating the combined image data (step S117). . Specifically, as illustrated in FIG. 5D, the composite image generation unit 746 combines the captured images on the composite image P <b> 1 based on the movement amount calculated by the movement amount calculation unit 742. The position is calculated, and the captured image P4 is superimposed on the calculated display position so as to overlap a part of the region of the combined image P1 (see FIG. 5E). Thereby, it is possible to generate a bonded image P1 of the specimen sample S in which the joint between the bonded image P1 and the captured image P4 is smooth.

  Thereafter, the display control unit 749 displays a composite image corresponding to the composite image data generated by the composite image generation unit 746 on the display area of the composite image display unit W10 (step S118). Specifically, as illustrated in FIG. 5C or FIG. 5E, the display control unit 749 pastes the composite image P1 corresponding to the composite image data generated by the composite image generation unit 746. The image is displayed on the display area of the image display unit W10.

  Subsequently, the control unit 74 determines whether or not a completion instruction signal is input by operating the completion instruction unit W13 via the input unit 72 (step S119). When the control unit 74 determines that the completion instruction signal has been input (step S119: Yes), the microscope system 1 ends this process. On the other hand, when the control unit 74 determines that the completion instruction signal has not been input via the input unit 72 (step S119: No), the microscope system 1 returns to step S103.

  In step S <b> 120, the display control unit 749 causes the display unit 62 to display a warning that the movement amount calculation unit 742 has not been able to calculate the relative movement amount of the observation region of the imaging device 4 with respect to the stage 21. Specifically, the display control unit 749 changes the display mode of the live position frame B1 displayed on the display area of the bonded image display unit W10, for example, by changing from “green” to “red”. The movement amount calculation unit 742 causes the display unit 62 to display a warning that the relative movement amount of the observation region of the imaging device 4 could not be calculated. Accordingly, the user can intuitively grasp that the bonded image P1 cannot be generated because the movement amount of the stage 21 is large. After step S120, the microscope system 1 proceeds to step S119.

  According to the embodiment of the present invention described above, the display control unit 749 displays the live image P2 and the composite image P1 in the display areas of the different display units 62, and is calculated by the movement amount calculation unit 742. Based on the relative movement amount of the observation area of the imaging device 4 with respect to the stage 21, the live position frame B1 is superimposed on the display position on the composite image P1 corresponding to the observation area of the current imaging device 4 and displayed. 62 is displayed. Thus, the user can easily identify the area on the composite image P1 to which the live image P2 is to be combined and position the live image P2 by confirming the composite image P1 and the live image P2 displayed on the display unit 62. Can be done. As a result, it is possible to efficiently create a bonded image of the specimen sample S with high accuracy.

  In addition, according to the embodiment of the present invention, the display control unit 749 pastes the peripheral image generated by the peripheral image generation unit 747 around the live image P2 displayed by the display unit 62, and displays the display unit 62. To display. Accordingly, the user can intuitively grasp the live image P2 and the image of the specimen sample S around the live image P2 regardless of the display size of the composite image P1, and thus the connection of the composite image P1. A bonded image with high eye accuracy can be generated.

(Modification 1)
In the embodiment of the present invention, the display size of the composite image P1 displayed on the display area of the composite image display unit W10 is controlled by operating the display size switching unit W16 via the input unit 72. Can be switched to automatic or manual.

  In FIG. 3, when the display size switching unit W16 selects automatic (Auto) via the input unit 72, the display control unit 749 reduces the composite image P1 within the display area of the composite image display unit W10. The combined image P1 is displayed while being automatically reduced at a rate. On the other hand, when manual is selected by the display size switching unit W16 via the input unit 72, the display control unit 749 displays the composite image P1 with a specified size on the display area of the composite image display unit W10. The designated size may be a ratio to the actual size, for example, the number of image pixels or the length of the image.

  Further, when manual is selected by the display size switching unit W16 via the input unit 72, the display control unit 749 determines that the input of the input unit 72 when the composite image P1 is larger than the display area of the composite image display unit W10. The combined image P1 corresponding to the scroll operation may be reduced. Further, when automatic is selected by the display size switching unit W16 via the input unit 72, the display control unit 749 sets a reduction ratio for reducing the composite image P1 to the end portion of the display area of the composite image display unit W10. You may make it reduce with a margin.

  As described above, according to the first modification according to the embodiment of the present invention, when automatic is selected by the display size switching unit W16, the user does not need to operate the display size, and the actual image of the composite image P1 is displayed. Even if the size is increased, the entire bonded image P1 can be confirmed. As a result, it is possible to improve the work efficiency of creating the bonded image P1. On the other hand, when the user selects manual in the display size switching unit W16, the combined image P1 is displayed in the display size desired by the user. Thereby, the user can confirm the detail of the bonded image P1. As a result, the accuracy of the combined image P1 can be improved.

(Modification 2)
Further, in one embodiment of the present invention, the scale can be displayed on the display area of the bonded image display unit W10 by operating the scale display selection unit W15 via the input unit 72.

  FIG. 6 is a diagram illustrating another example of the operation screen displayed by the display unit 62 of the microscope system 1 according to the second modification of the embodiment of the present invention.

  As shown in FIG. 6, when the ON switch W151 is selected and set by the scale display selection unit W15 via the input unit 72, the display control unit 749 has the scale S1 on the display area of the bonded image display unit W10. Is displayed. The scale S1 has X-axis and Y-axis lines, and a scale is provided on these lines.

  The display control unit 749 also displays the scale (X: 5 μm, Y: 5 μm) and angle (Angle: 90) of the scale S1 on the bonded image display unit W10. It is displayed near S1. Further, the display control unit 749 may display the scale S1 by changing the lengths and angles of the X axis and the Y axis according to the operation content of the input unit 72. Furthermore, the display control unit 749 may display the scale S1 by moving the display position according to the operation content of the input unit 72. Note that the scale S1 may be only in one direction of the X axis or the Y axis.

  According to the modification 2 according to the embodiment of the present invention described above, when the display control unit 749 selects the ON switch W151 by the scale display selection unit W15 via the input unit 72, the bonded image display unit Since the scale S1 for measuring the composite image P1 is displayed on the display area of W10, the actual size of the composite image P1 can be measured by using the scale S1. As a result, the composite image P1 can be created without waste, and the work efficiency of creating the composite image P1 is improved.

(Modification 3)
In one embodiment of the present invention, a grid line (guideline) can be displayed on the display area of the bonded image display unit W10 by operating the grid display selection unit W17 via the input unit 72. .

  FIG. 7 is a diagram illustrating another example of the operation screen W1 displayed on the display unit 62 of the microscope system 1 according to the third modification of the embodiment of the present invention.

  As shown in FIG. 7, when the division number of the grid lines is selected and set by the grid display selection unit W17 via the input unit 72, the display control unit 749 is displayed on the display area of the bonded image display unit W10. The grid line G1 is displayed. Under the situation illustrated in FIG. 7, the display control unit 749 displays the grid line G <b> 1 on the display area of the bonded image display unit W <b> 10 in 3 × 3. The designated type of the division number of the grid line G1 can be set as appropriate, or the number of grid lines G1 may be manually moved via the input unit 72.

  According to Modification 3 according to the embodiment of the present invention described above, the display control unit 749 pastes based on the division number of the grid line G1 selected by the grid display selection unit W17 via the input unit 72. A grid line G1 to be displayed on the display area of the combined image display unit W10 is displayed. Thereby, the grid line G1 is displayed on the display area of the combined image display unit W10, so that the user can easily perform the positioning of the bonding when the captured image is combined with the combined image P1. The work efficiency of creating the bonded image P1 is improved.

(Modification 4)
Further, in one embodiment of the present invention, by operating the reference position selection unit W14 via the input unit 72, the display of the live position frame B1 that is initially displayed on the display area of the bonded image display unit W10. The position can also be changed.

  FIG. 8 is a diagram illustrating another example of the operation screen W1 displayed on the display unit 62 of the microscope system 1 according to the fourth modification of the embodiment of the present invention.

  As shown in FIG. 8, the display control unit 749 displays the live position on the display area of the bonded image display unit W10 according to the set display position operated by the reference position selection unit W14 via the input unit 72. The display position of the frame B1 is displayed. Specifically, as shown in FIG. 8A, when the upper right is selected and set by the reference position selection unit W14 via the input unit 72, the display control unit 749 is displayed on the display area of the combined image display unit W10. First, the display position for displaying the live position frame B1 is set at the upper right and displayed. Further, as shown in FIG. 8B, the display control unit 749 sets and displays the display position of the live position frame B1 to be displayed first according to the display position selected by the reference position selection unit W14. Note that the display control unit 749 may display the live position frame B1 by changing the display position in accordance with the operation of the input unit 72.

  According to the modification 4 according to the embodiment of the present invention described above, the display control unit 749 performs the pasting according to the display position set by operating the reference position selection unit W14 via the input unit 72. Since the display position of the live position frame B1 on the display area of the image display unit W10 is displayed, it is possible to specify a reference position when creating the composite image P1. As a result, since the user can change the order of pasting in accordance with the specimen S, work efficiency when creating the pasted image P1 can be improved.

(Modification 5)
In one embodiment of the present invention, a warning can be issued when the live position frame B1 moves outside the display area of the composite image P1.

  FIG. 9 is a diagram illustrating another example of the operation screen displayed by the display unit 62 of the microscope system 1 according to the fifth modification of the embodiment of the present invention.

  As shown in FIG. 9, when the display position of the live position frame B1 is displayed outside the display area of the composite image P1 (FIG. 9A), the display control unit 749 displays the input unit 72. When the shooting instruction signal of the imaging device 4 is input via the change, the display mode of the live position frame B1 is changed, for example, by changing from “green” to “red”, the shape of the composite image P1 becomes invalid. Is warned (FIG. 9B). Note that the display control unit 749 may display a message or the like on the display unit 62 instead of changing the display mode of the live position frame B1 to warn that the shape of the composite image P1 is incorrect. . In FIG. 9, the solid line is expressed in green and the dotted line is expressed in red.

  According to the fifth modification of the embodiment of the present invention described above, when the display control unit 749 displays the display position of the live position frame B1 outside the display area of the composite image P1 (see FIG. 9 (a)), when the shooting instruction signal of the imaging device 4 is input via the input unit 72, the display mode of the live position frame B1 is changed. Thereby, the user can intuitively grasp that the shape of the combined image P1 is distorted.

  In the above-described embodiment, the microscope system including the microscope device, the imaging device, the display input unit, and the control terminal has been described as an example. However, for example, an objective lens for enlarging a specimen sample, and a specimen sample via an objective lens The present invention can also be applied to an imaging apparatus having an imaging function for imaging and a display function for displaying an image, such as a video microscope.

  In the above-described embodiment, the upright microscope apparatus is described as an example of the microscope apparatus. However, the present invention can be applied to an inverted microscope apparatus, for example. Furthermore, the present invention can be applied to various systems such as a line apparatus incorporating a microscope apparatus.

  In the embodiment described above, various settings of the microscope system are performed via the input unit. However, various settings of the microscope system may be performed via the touch panel.

  In the above-described embodiment, the electric stage has been described as an example of the stage. However, the present invention can be applied to a manual stage that is manually moved.

  In the above-described embodiment, the display input unit and the control terminal are separately configured. However, for example, a portable terminal in which the display input unit and the control terminal are integrally formed may be used.

  In the above-described embodiment, the stage 21 is movable in the Z direction by the motor 212. However, the objective lens 23 or the entire observation optical system including the objective lens 23 may be movable in the Z direction.

DESCRIPTION OF SYMBOLS 1 Microscope system 2 Microscope apparatus 3 Microscope control part 4 Imaging apparatus 5 Imaging control part 6 Display input part 7 Control terminal 21 Stage 22 Revolver 23 Objective lens 24 Microscope main body 25 Light source for epi-illumination 41 Image pick-up element 51 AE process part 52 AF process Unit 61 Display communication unit 62 Display unit 63 Touch panel 71 Control communication unit 72 Input unit 73 Storage unit 74 Control unit 211 Stage drive unit 212 Motor 221 Mounter 222 Revolver drive unit 241 Illumination lens 242 Half mirror 243 Zoom lens unit 243a Zoom optical system 243b Zoom drive unit 244 Imaging lens 731 Image data storage unit 741 Live image generation unit 742 Movement amount calculation unit 743 Movement amount determination unit 744 Live frame generation unit 745 Captured image generation unit 746 Bonded image generation unit 747 Image generation unit 748 Drive control unit 749 Display control unit W1 Operation screen W10 Bonded image display unit W11 Live image display unit W14 Reference position selection unit W15 Scale display selection unit W16 Display size switching unit W17 Grid display selection unit

Claims (10)

A stage on which a specimen can be placed and moved horizontally,
An imaging device that images the specimen sample placed on the stage and continuously generates image data of the specimen sample;
A display unit capable of displaying an image corresponding to the image data generated by the imaging device;
A live image generation unit that sequentially generates live images to be displayed on the display unit from the image data continuously generated by the imaging device;
A movement amount calculation unit that calculates a relative movement amount of the observation region of the imaging device with respect to the stage based on the temporally continuous image data generated by the imaging device;
Based on the movement amount calculated by the movement amount calculation unit, the image data continuously generated by the imaging device is used to generate bonded image data having an observation region whose outer edge is wider than the observation region of the imaging device. A composite image generation unit;
A live frame generation unit that generates a live position frame indicating an observation region of the imaging device in a display region of the display unit;
The live image corresponding to the live image data is displayed in the live image display area of the display unit, the composite image corresponding to the composite image data is displayed in the composite image display area of the display unit , and the movement amount Based on the movement amount calculated by the calculation unit, the live position frame is superimposed on the display position on the composite image corresponding to the current observation area of the imaging device, and the composite image display region of the display unit A display control unit to be displayed on
Equipped with a,
The display control unit fixes the live image to a central region of the live image display region, displays the live image on the display unit, and surrounds the live image specified by the movement amount calculated by the movement amount calculation unit A microscope system , wherein a peripheral image that is a part of the composite image corresponding to a region is displayed in the live image display region in accordance with a position of the live image . Based on the movement amount calculated by the movement amount calculation unit, a region image is generated by cutting out the region of the combined image located around the live image in the display region of the live image displayed by the display unit. the microscope system according to claim 1, further comprising a peripheral image generation unit. The movement amount calculation unit extracts feature points respectively included in the two live images corresponding to the two temporally continuous image data generated by the imaging device, and is identical between the two live images. The microscope system according to claim 1, wherein the movement amount relative to the observation area of the imaging apparatus with respect to the stage is calculated based on movement amounts of a plurality of corresponding points. A movement amount determination unit for determining whether or not the movement amount calculation unit was able to calculate the movement amount;
When the movement amount determination unit determines that the movement amount calculation unit cannot calculate the movement amount, the display control unit indicates that the movement amount cannot be calculated on the display unit. The information which shows is displayed on the said display part, The microscope system as described in any one of Claims 1-3 characterized by the above-mentioned. When the movement amount determination unit determines that the movement amount calculation unit cannot calculate the movement amount, the display control unit changes the display mode of the live position frame and displays the display on the display unit. The microscope system according to claim 4 , wherein: A display size switching unit for receiving an input of an instruction signal for switching the display size of the composite image displayed by the display unit;
The display control unit causes the display unit to display the composite image and the live position frame at the display size according to an instruction signal input from the display size switching unit, respectively. 5. The microscope system according to any one of 5 above. An input unit for performing input to the display size switching unit;
The microscope system according to claim 6, wherein the display control unit changes the display size of the composite image and the live position frame in accordance with an input of the input unit. A scale display selection unit for receiving an input of an instruction signal for selecting whether or not to display a scale for measuring the actual size of the combined image;
The display controller, the microscope system according to any one of claims 1-7, characterized in that for displaying the scale in response to an instruction signal input from the scale display selection unit on the display unit. The display unit further includes a grid display selection unit that receives an input of an instruction signal for selecting whether or not to display grid lines indicating a grid formed at a predetermined interval on a display area where the combined image is displayed,
The display controller, the microscope system according to any one of claims 1-7, characterized in that to display the grid line in response to an instruction signal input from the grid display selection unit on the display unit . The display unit further includes a display position selection unit that receives an input of an instruction signal for selecting a reference position of the live position frame on a display area where the combined image is displayed,
The display controller, to claim 1-7, characterized in that for displaying the live position frame to a reference position corresponding to the instruction signal input from the display position selection section on the display section The described microscope system.

Images