JP5881356B2 - Microscope system - Google Patents

Description

  The present invention relates to a microscope system for displaying an image corresponding to image data generated by imaging a specimen sample placed on a stage using an objective lens.

  2. Description of the Related Art Conventionally, a technique for generating a single combined image wider than an observation region of an imaging device by combining a plurality of still images taken while changing an observation region for observing a specimen is known ( Patent Document 1). In this technique, a live image continuously generated by the imaging device is displayed on a bonded image of a specimen sample displayed on a display monitor, and the positional relationship of the specimen sample to be bonded next is displayed using this live image. This makes it possible to create a bonded image of a desired specimen sample.

JP 2010-141697 A

  However, in the above-described technique, the position for the next pasting is determined by manual operation while confirming at the display position of the live image, and the pasted images are sequentially created. When the user makes various measurements using the image after pasting, it is difficult to measure because the shape of the stitched image is distorted, and accurate measurement values cannot be obtained. There was a point.

  The present invention has been made in view of the above, and an object of the present invention is to provide a microscope system capable of realizing accurate measurement using a bonded image.

In order to solve the above-described problems and achieve the object, the microscope system according to the present invention is an image obtained by imaging a specimen sample placed on a stage and pasting a plurality of images corresponding to the image data of the specimen sample. In the microscope system capable of generating a combined image, the effective region determining unit that determines the maximum region in which the rectangular effective region including the pixel data in the combined image is maximized, and the determination result of the effective region determining unit Accordingly, an image shaping unit that cuts out the maximum area from the combined image and shapes the shape of the combined image is provided.

  The microscope system according to the present invention is characterized in that, in the above invention, the effective area determination unit determines the effective area for a reduced image obtained by reducing the combined image.

Furthermore, the microscope system according to the present invention, in the invention, the effective region judging section, the maximum area in which the effective area within the image bonded using said plurality of patterns having a rectangular shape is maximized determine Teisu It is characterized by that.

In the microscope system according to the present invention, in the above invention, the effective area determination unit determines the effective area including the pixel data by performing a binarization process on the combined image. Features.
In the microscope system according to the present invention as set forth in the invention described above, the imaging of the specimen sample placed on the stage is performed by manually moving the stage.

  According to the microscope system of the present invention, the effective area determination unit determines a rectangular effective area including pixel data in the combined image, and the image shaping unit pastes the image according to the determination result of the effective area determination unit. The effective area is cut out from the combined image and the shape of the combined image is shaped. As a result, there is an effect that when measurement is performed using the bonded image, measurement is easy and accurate measurement can be realized.

FIG. 1 is a conceptual diagram showing an example of the configuration of the microscope system according to the first embodiment of the present invention. FIG. 2 is a block diagram illustrating a functional configuration of the microscope system according to the first embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a display unit of the microscope system according to the first embodiment of the present invention. FIG. 4 is a flowchart showing an outline of processing performed by the microscope system according to the first embodiment of the present invention. FIG. 5 is a diagram schematically illustrating the four-direction expansion of the first rectangle determination process executed by the effective area determination unit. FIG. 6 is a diagram schematically illustrating a situation in which the first region determination process performed by the effective region determination unit is not good at the four-direction expansion. FIG. 7 is a diagram schematically illustrating an outline of the lateral priority extension of the second rectangle determination process executed by the effective area determination unit. FIG. 8 is a diagram schematically illustrating a situation in which the second region determination process executed by the effective area determination unit is not good at the priority in the horizontal direction. FIG. 9 is a diagram schematically illustrating an overview of the vertical priority extension of the third rectangle determination process executed by the effective area determination unit. FIG. 10 is a diagram schematically illustrating a situation in which the third region determination process executed by the effective region determination unit is not good at the vertical priority extension. FIG. 11 is a diagram schematically illustrating an outline of 4-direction expansion + vertical direction priority expansion of the fourth rectangle determination process executed by the effective area determination unit. FIG. 12 is a diagram schematically illustrating a situation in which it is difficult to perform the 4-direction extension + vertical direction priority extension of the fourth rectangle determination process executed by the effective area determination unit. FIG. 13 is a diagram schematically illustrating an outline of 4-direction expansion + horizontal priority expansion of the fifth rectangle determination process executed by the effective area determination unit. FIG. 14 is a diagram schematically illustrating an outline of 4-direction expansion + horizontal priority expansion of the fifth rectangle determination process executed by the effective area determination unit. FIG. 15 is a flowchart illustrating an outline of processing performed by the microscope system according to the second embodiment of the present invention. FIG. 16 is a diagram schematically illustrating an overview of the extension determination process for a binary image having a single size executed by the effective area determination unit.

  Hereinafter, a form for carrying out this embodiment (hereinafter referred to as “embodiment”) will be described with reference to the drawings. The present invention is not limited to the embodiments described below. Further, in the description of the drawings, the same portions will be described with the same reference numerals.

(Embodiment 1)
FIG. 1 is a conceptual diagram showing an example of the configuration of the microscope system according to the first embodiment of the present invention. FIG. 2 is a block diagram illustrating a functional configuration of the microscope system according to the first embodiment of the present invention. 1 and 2, the plane on which the microscope system 1 is placed will be described as an XY plane, and a direction perpendicular to the XY plane will be described as a Z direction.

  As shown in FIG. 1 and FIG. 2, the microscope system 1 images the specimen sample S through the microscope apparatus 2 that observes the specimen sample S, the microscope control unit 3 that drives and controls the microscope apparatus 2, and the microscope apparatus 2. The image pickup apparatus 4 that generates image data, the image pickup control unit 5 that controls the drive of the image pickup apparatus 4, and the image corresponding to the image data picked up by the image pickup apparatus 4 via the control terminal 7 are displayed, and the microscope system The display input part 6 which receives the input of 1 various operation, and the control terminal 7 which controls each part of the microscope system 1 are provided. The microscope device 2, the microscope control unit 3, the imaging device 4, the imaging control unit 5, the display input unit 6, and the control terminal 7 are connected by wire or wireless so that data can be transmitted and received.

  The microscope apparatus 2 has a stage 21 on which a specimen S is placed, a substantially C-shape in side view, a stage body 24 that supports the stage 21 and holds an objective lens 23 via a revolver 22; And an epi-illumination light source 25 that irradiates the specimen S with light.

  The stage 21 is configured to be movable in the XYZ directions. The stage 21 is movable in the XY plane by the stage driving unit 211. Under the control of the microscope control unit 3, the stage 21 detects a predetermined origin position on the XY plane by an XY position origin sensor (not shown), and the drive amount of the stage drive unit 211 is controlled based on this origin position. Thus, the observation location (observation region) on the specimen sample S is moved. The stage 21 outputs position signals (XY coordinates) relating to the X position and the Y position during observation to the microscope control unit 3. Further, the stage 21 is movable in the Z direction by a motor 212. Under the control of the microscope control unit 3, the stage 21 detects a predetermined origin position in the Z direction of the stage 21 by a Z position origin sensor (not shown), and the driving amount of the motor 212 is controlled based on this origin position. As a result, the specimen sample S is moved to an arbitrary Z position within a predetermined height range. The stage 21 outputs a position signal related to the Z position during observation to the microscope control unit 3.

  The revolver 22 is provided so as to be slidable or rotatable with respect to the microscope main body 24, and the objective lens 23 is disposed above the specimen S. The revolver 22 is configured using a nose piece, a swing revolver, or the like. The revolver 22 holds a plurality of objective lenses 23 having different magnifications (observation magnifications) by the mounter 221. The revolver 22 is inserted in the optical path of the observation light, and the connection between the revolver 22 and the revolver drive unit 222 that slides or rotates the mounter 221 in order to selectively switch the objective lens 23 used for observing the specimen S. And a revolver detector 223 for detecting the like.

  The revolver driving unit 222 slides or rotates the mounter 221 under the control of the microscope control unit 3. The revolver detection unit 223 includes a revolver connection sensor (not shown) that detects that the revolver 22 is connected to the microscope main body 24, and the type of the objective lens 23 in which the objective lens 23 is inserted in the optical path of the observation light. And a movement completion sensor (not shown) for detecting that the objective lens 23 is inserted on the optical path of the observation light. The revolver detection unit 223 outputs detection results detected by various sensors to the microscope control unit 3.

  The objective lens 23 includes, for example, an objective lens 231 with relatively low magnification of 1 ×, 2 ×, and 4 × (hereinafter referred to as “low magnification objective lens 231”), and a 10 ×, 20 ×, and 40 × low magnification objective lens. At least one objective lens 232 (hereinafter referred to as “high-magnification objective lens 232”) having a high magnification relative to the magnification of 231 is attached to the mounter 221. Note that the magnifications of the low-magnification objective lens 231 and the high-magnification objective lens 232 are examples, and it is sufficient that the high-magnification objective lens 232 is higher than the low-magnification objective lens 231.

  The microscope main body 24 includes an illumination lens 241 that collects the illumination light L1 emitted from the epi-illumination light source 25 through the fiber 251 (hereinafter referred to as “epi-illumination light L1”), and an optical path of the epi-illumination light L1. The half mirror 242 that deflects along the optical axis of the objective lens 23, the zoom lens unit 243 that magnifies the sample S, and the sample sample S incident through the objective lens 23, the zoom lens unit 243, and the half mirror 242. An imaging lens 244 that focuses the reflected light to form an observation image is provided inside.

  The zoom lens unit 243 includes one or a plurality of lenses, and includes a zoom optical system 243a that can zoom the specimen sample S, and a zoom drive unit 243b that drives the zoom optical system 243a along the optical axis. The zoom drive unit 243b changes the zoom magnification of the zoom lens unit 243 by moving the zoom optical system 243a along the optical axis under the control of the microscope control unit 3.

  The epi-illumination light L1 is applied to the specimen S through the illumination lens 241, the half mirror 242, the zoom optical system 243a, and the objective lens 23. The reflected light L2 reflected by the specimen S (hereinafter referred to as “observation light L2”) enters the imaging device 4 through the objective lens 23, the zoom optical system 243a, the half mirror 242, and the imaging lens 244.

  The epi-illumination light source 25 includes a halogen lamp, a xenon lamp, an LED (Light Emitting Diode), or the like. The epi-illumination light source 25 emits epi-illumination light L <b> 1 for forming an observation image of the specimen S through the fiber 251 to the microscope main body 24.

  The microscope control unit 3 is configured using a CPU (Central Processing Unit) or the like, and comprehensively controls the operation of each unit configuring the microscope apparatus 2 under the control of the control terminal 7. Specifically, the microscope control unit 3 drives the revolver driving unit 222 to rotate the mounter 221 to switch the objective lens 23 arranged on the optical path of the observation light L2, the stage driving unit 211 or the motor By driving 212, the stage 21 driving process, the adjustment process for adjusting each part of the microscope apparatus 2 accompanying the observation of the specimen S, and the like are performed. In addition, the microscope control unit 3 informs the control terminal 7 of the state of each part constituting the microscope apparatus 2, for example, the position information (XY position, Z position) of the stage 21 and the type information of the objective lens 23 attached to the revolver 22. Output.

  The imaging device 4 is configured using an imaging device 41 such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). Under the control of the imaging control unit 5, the imaging device 4 captures the observation image of the specimen sample S incident through the imaging lens 244 and continuously generates image data of the specimen sample S. The imaging device 4 outputs the image data of the specimen sample S generated via the camera cable to the control terminal 7.

  The imaging control unit 5 is configured using a CPU or the like, and controls the operation of the imaging device 4. Specifically, the imaging control unit 5 controls the photographing operation of the imaging device 4 by performing ON / OFF switching processing of automatic gain control of the imaging device 4, gain setting processing, frame rate setting processing, and the like. The imaging control unit 5 includes an AE processing unit 51 and an AF processing unit 52.

  The AE processing unit 51 automatically sets the exposure condition of the imaging device 4 based on the image data generated by the imaging device 4. Specifically, the AE processing unit 51 calculates the luminance from the image data acquired via the control terminal 7, and determines the exposure condition of the imaging device 4, for example, the exposure time based on the calculated luminance. AE processing for automatically adjusting the exposure of 4 is performed.

  The AF processing unit 52 automatically adjusts the focus of the imaging device 4 based on the image data generated by the imaging device 4. Specifically, the AF processing unit 52 automatically adjusts the focus of the imaging device 4 by evaluating the contrast included in the image data and detecting the in-focus position (focal position). AF processing is performed. Note that the AF processing unit 52 may detect a focused focus position (Z position) by evaluating the contrast of the image at each Z position of the stage 21 based on the image data.

  The display input unit 6 includes a display communication unit 61 that performs communication with the control terminal 7, a display unit 62 that displays an image, and a touch panel 63 that outputs a position signal corresponding to an external object contact.

  The display communication unit 61 is a communication interface for performing communication with the control terminal 7. The display communication unit 61 outputs the image data output from the control terminal 7 to the display unit 62.

  A display panel made of liquid crystal or organic EL (Electro Luminescence) is used. The display unit 62 displays an image corresponding to image data input via the display communication unit 61 and various operation information of the microscope system 1. Specifically, the display unit 62 is a live image corresponding to the image data continuously generated by the imaging device 4 or a bonded image of the specimen sample S having an observation region whose outer edge is wider than the observation region of the imaging device 4. Display the combined image corresponding to the data.

  The touch panel 63 is provided on the display screen of the display unit 62 and accepts an input according to the contact position of the object from the outside. Specifically, the touch panel 63 detects a position where the user touches (contacts) according to an operation icon displayed on the display unit 62, and outputs a position signal corresponding to the detected touch position to the control terminal 7. The touch panel 63 functions as a graphical user interface (GUI) when the display unit 62 displays various operation information of the microscope system 1 in the image display area A1 (see FIG. 3). In general, the touch panel includes a resistance film method, a capacitance method, an optical method, and the like. In the first embodiment, any type of touch panel is applicable.

  The control terminal 7 includes a control communication unit 71 that communicates with the microscope control unit 3, the imaging control unit 5, and the display input unit 6, a storage unit 73 that stores various types of information of the microscope system 1, and each unit of the microscope system 1. An input unit 72 that receives an input of a drive instruction signal that instructs driving, and a control unit 74 that controls each unit of the microscope system 1 are provided.

  The control communication unit 71 is a communication interface for communicating with each of the microscope control unit 3, the imaging control unit 5, and the display input unit 6. Further, the control communication unit 71 outputs the image data output from the imaging device 4 to the control unit 74 via the camera cable.

  The input unit 72 is configured using a keyboard, a mouse, a joystick, various switches, and the like, and outputs operation signals corresponding to operation inputs of the various switches to the control unit 74.

  The storage unit 73 is realized by using a semiconductor memory such as a flash memory and a RAM (Random Access Memory) fixedly provided inside the control terminal 7. The storage unit 73 stores various programs to be executed by the microscope system 1 and various data used during execution of the programs. The storage unit 73 temporarily stores information being processed by the control unit 74. The storage unit 73 has an image data storage unit 731 that stores image data captured by the imaging device 4 and a live image corresponding to the image data continuously generated by the imaging device 4 or an outer edge of the observation region of the imaging device 4. A combined image data storage unit 732 that stores the combined image data of the specimen sample S having a wide observation area. In addition, the memory | storage part 73 may be comprised using the memory card etc. which are mounted | worn from the outside.

  The control unit 74 is configured using a CPU or the like, and performs a command or data transfer corresponding to each unit constituting the microscope system 1 in accordance with an operation signal from the input unit 72 and a position signal from the touch panel 63, and the microscope system. The operation of 1 is comprehensively controlled.

  A detailed configuration of the control unit 74 will be described. The control unit 74 includes a live image generation unit 741, a movement amount calculation unit 742, a movement amount determination unit 743, a captured image generation unit 744, a composite image generation unit 745, an effective area determination unit 746, and an image shaping. A unit 747, a drive controller 748, and a display controller 749.

  The live image generation unit 741 sequentially generates live image data to be displayed on the display unit 62 from the image data continuously generated by the imaging device 4. The live image generation unit 741 outputs live image data to the image data storage unit 731. The live image generation unit 741 also performs predetermined image processing, such as optical black subtraction processing, white balance adjustment processing, synchronization processing, color matrix calculation processing, γ correction processing, color reproduction processing, and edge enhancement processing, on image data. Image processing including the above is performed.

  The movement amount calculation unit 742 observes the imaging device 4 with respect to the stage 21 based on two images corresponding to two image data whose time is generated by the imaging device 4 input via the control communication unit 71. The relative movement amount in the area is calculated. Specifically, the movement amount calculation unit 742 extracts feature points included in each of the live images corresponding to the two pieces of image data moving forward and backward in time, and extracts a plurality of corresponding points that are the same between the two live images. By calculating the amount of movement (number of pixels), the amount of movement (shift amount) relative to the stage 21 in the observation region of the imaging device 4 is calculated. Here, the feature point is luminance information, color information, or edge information. Note that the movement amount calculation unit 742 may calculate the relative movement amount in the observation region of the imaging device 4 using a known technique such as pattern matching.

  The movement amount determination unit 743 determines whether or not the movement amount calculation unit 742 can calculate the relative movement amount in the observation region of the imaging device 4 with respect to the stage 21 based on the two live images that move back and forth in time. . Specifically, the movement amount determination unit 743 determines whether or not the movement amount calculation unit 742 has been able to extract a corresponding feature point between two live images. For example, when the movement amount calculation unit 742 cannot extract feature points corresponding to each other between two live images, the movement amount determination unit 743 observes the imaging apparatus 4 with respect to the stage 21. It is determined that the relative movement amount in the region could not be calculated.

  The captured image generation unit 744 performs predetermined image processing on the image data input via the control communication unit 71 and outputs the image data to the composite image generation unit 745 and the image data storage unit 731. Specifically, the captured image generation unit 744 performs optical black subtraction processing, white balance adjustment processing, synchronization processing, color matrix calculation processing, γ correction processing, color reproduction processing, edge enhancement processing, and the like on the image data. Including image processing. The captured image generation unit 744 compresses the image data by a predetermined method, for example, JPEG (Joint Photographic Experts Group) method, and outputs the compressed image data to the combined image generation unit 745 and the image data storage unit 731. Also good.

  Based on the relative movement amount of the observation area of the imaging device 4 with respect to the stage 21 calculated by the movement amount calculation unit 742, the bonded image generation unit 745 uses the observation area of the imaging device 4 from the image data generated by the imaging device 4. Bonded image data having an observation region with a wider outer edge than that is generated. Specifically, the composite image generation unit 745 acquires the composite image data stored in the image data storage unit 731, and the movement amount calculation unit 742 calculates the captured image data output from the captured image generation unit 744. The composite image data is generated by combining the display position of the composite image determined by the relative movement amount of the observation region of the imaging device 4 with respect to the stage 21. For example, the composite image generation unit 745 performs blending processing of pixel values on the overlapping area between the composite image and the captured image in order to make the joint inconspicuous when the captured image is combined with the composite image. A stitched image is generated. Here, the blending process is an image process for calculating a pixel value of a composite image by weighted averaging of pixel values between images. Note that the composite image generation unit 745 may change the weight at the time of the weighted average in the blending process according to the pixel position.

  The effective area determination unit 746 determines a rectangular effective area that includes pixel data in the combined image corresponding to the combined image data generated by the combined image generation unit 745. Specifically, the effective area determination unit 746 performs a binarization process on the combined image data, so that a rectangular effective area including pixel data in the combined image and an invalid pixel data are not present. Determine the area. In addition, the effective area determination unit 746 determines a maximum area in which the effective area is maximized in the composite image using a plurality of rectangular patterns.

  The image shaping unit 747 cuts out the effective area from the composite image and shapes the shape of the composite image according to the determination result of the effective area determination unit 746. Specifically, the image shaping unit 747 shapes the shape of the combined image into a rectangular shape by cutting out an area where the effective area of the combined image determined by the effective area determining unit 746 is maximized (trimming process). Generate a shaped image.

  The drive control unit 748 controls the driving of the microscope device 2 and the imaging device 4 based on the operation signal input from the input unit 72 or the position signal input from the touch panel 63. Specifically, when a shooting instruction signal for instructing shooting is input from the input unit 72, the drive control unit 748 outputs a command signal for instructing shooting to the imaging control unit 5, thereby shooting the imaging device 4. Run the action. In addition, the drive control unit 748 drives the stage 21 at a moving speed at which two images generated by the imaging device 4 that have different times are overlapped. For example, the drive control unit 748 drives the stage drive unit 211 at a moving speed at which ends including one side of the image overlap each other. Furthermore, when the observation mode of the microscope system 1 is set from the normal observation mode to the bonded image observation mode by the input unit 72, the drive control unit 748 receives an instruction signal that instructs the movement of the stage 21 from the input unit 72. Then, the stage 21 is driven at a moving speed that is slower than the moving speed in the normal observation mode.

  The display control unit 749 controls the display mode of the display unit 62. Specifically, the display control unit 749 displays the live image generated by the live image generation unit 741 and the composite image generated by the composite image generation unit 745 via the control communication unit 71 and the display communication unit 61. This is displayed on the unit 62. Based on the movement amount of the observation area of the imaging device 4 calculated by the movement amount calculation unit 742, the display control unit 749 adds a live position frame to the display position on the composite image corresponding to the current observation area of the imaging device 4. The images are superimposed and displayed in the image display area A1 of the display unit 62.

  The microscope system 1 configured in this manner displays an image of the sample sample S on the display unit 62 by displaying the image data of the sample sample S imaged by the imaging device 4 under the control of the control unit 74. Can be observed. Furthermore, in the microscope system 1, the microscope device 2 and the imaging device 4 are driven when the control unit 74 outputs an instruction signal or a drive signal to each unit of the microscope system 1 based on an operation signal input from the input unit 72. To do.

  Next, operations performed by the microscope system 1 will be described. FIG. 4 is a flowchart illustrating an outline of processing performed by the microscope system 1 according to the first embodiment. In the following, a bonded image generation mode in which the microscope system 1 generates a bonded image of the specimen S will be described.

  As shown in FIG. 4, the live image generation unit 741 generates live image data to be displayed on the display unit 62 from the image data generated by the imaging device 4 (step S101).

  Subsequently, the display control unit 749 displays a live image corresponding to the live image data generated by the live image generation unit 741 on the display unit 62 (step S102).

  Thereafter, the movement amount calculation unit 742 calculates the relative movement amount of the observation region of the imaging device 4 with respect to the stage 21 based on the latest live image and the previous live image (step S103). Specifically, the movement amount calculation unit 742 is an imaging device for the stage 21 based on the latest live image generated by the live image generation unit 741 and the previous live image stored by the image data storage unit 731. The relative movement amount in the four observation areas is calculated.

  Subsequently, the movement amount determination unit 743 determines whether or not the movement amount calculation unit 742 has been able to calculate the relative movement amount of the observation area of the imaging device 4 with respect to the stage 21 (step S104). Specifically, the movement amount determination unit 743 extracts the corresponding points corresponding to each of a plurality of feature points extracted from the previous live image by the movement amount calculation unit 742, thereby capturing the stage 21. It is determined whether or not the movement amount of the observation area of the device 4 has been calculated. For example, when the movement amount by which the stage 21 moves is large, the movement amount determination unit 743 cannot extract the corresponding points corresponding to the feature points extracted from the previous live image by the movement amount calculation unit 742 from the latest live image. The movement amount calculation unit 742 determines that the relative movement amount of the observation region of the imaging device 4 with respect to the stage 21 could not be calculated. When the movement amount determination unit 743 determines that the movement amount calculation unit 742 can calculate the relative movement amount of the observation region of the imaging device 4 with respect to the stage 21 (step S104: Yes), the microscope system 1 is described later. The process proceeds to step S106. On the other hand, when the movement amount determination unit 743 determines that the movement amount calculation unit 742 cannot calculate the relative movement amount of the observation region of the imaging device 4 with respect to the stage 21 (step S104: No), the microscope system 1 Shifts to step S105 to be described later.

  In step S <b> 105, the display control unit 749 causes the display unit 62 to display a warning that the movement amount calculation unit 742 has not been able to calculate the relative movement amount of the observation region of the imaging device 4 with respect to the stage 21. Specifically, the display control unit 749 changes the display mode of the live image displayed on the display area of the display unit 62, for example, displays it in a highlight display, so that the movement amount calculation unit 742 can A warning that the relative movement amount of the observation area of the imaging device 4 could not be calculated is displayed on the display unit 62. Accordingly, the user can intuitively understand that the live image cannot be smoothly combined with the composite image because the moving amount of the stage 21 is large. After step S105, the microscope system 1 proceeds to step S106.

  Subsequently, the control unit 74 determines whether or not a shooting instruction signal is input via the input unit 72 or the touch panel 63 (step S106). When the control unit 74 determines that an imaging instruction signal has been input (step S106: Yes), the microscope system 1 proceeds to step S107. On the other hand, when the control unit 74 determines that the photographing instruction signal is not input via the input unit 72 or the touch panel 63 at a predetermined time (step S106: No), the microscope system 1 returns to step S101.

  In step S107, the drive control unit 748 stops the live image data generation by the live image generation unit 741.

  Subsequently, the drive control unit 748 outputs a shooting drive signal to the imaging control unit 5 and causes the imaging device 4 to perform a shooting operation to generate image data (step S108).

  Thereafter, the captured image generation unit 744 generates captured image data from the image data generated by the imaging device 4 (step S109). At this time, the captured image generation unit 744 generates captured image data from one frame of image data generated by the imaging device 4. Further, when the imaging device 4 generates a plurality of frames of image data, the captured image generation unit 744 may generate omnifocal image data and 3D image data from the plurality of frames of image data.

  Subsequently, the composite image generation unit 745 generates composite image data (step S110). Specifically, the combined image generation unit 745 acquires the combined image data from the combined image data storage unit 732 and the captured image data from the captured image generation unit 744, and then calculates the movement calculated by the movement amount calculation unit 742. The display position of the captured image data on the composite image is calculated from the amount, and the composite image data is generated by synthesizing the captured image data generated by the captured image generation unit 744 with the calculated display position. For example, the composite image generation unit 745 calculates a display position where the captured image is combined on the composite image based on the movement amount calculated by the movement amount calculation unit 742, and pastes the captured image on the calculated display position. The combined images are superimposed so that a part of the region overlaps. Thereby, it is possible to generate a bonded image of the specimen sample S in which the joint between the bonded image and the captured image is smooth.

  Thereafter, the display control unit 749 causes the display unit 62 to display a composite image corresponding to the composite image data generated by the composite image generation unit 745 (step S111).

  Subsequently, the control unit 74 determines whether or not a completion instruction signal for completing creation of a composite image has been input via the input unit 72 or the touch panel 63 (step S112). When the control unit 74 determines that the completion instruction signal for completing the creation of the composite image has been input via the input unit 72 or the touch panel 63 (step S112: Yes), the microscope system 1 proceeds to step S113. On the other hand, when the control unit 74 determines that the completion instruction signal for completing the creation of the bonded image is not input via the input unit 72 or the touch panel 63 within a predetermined time (step S112: No), the microscope system 1 Return to step S101.

  In step S <b> 113, the drive control unit 748 stores the composite image data generated by the composite image generation unit 745 in the composite image data storage unit 732.

  Subsequently, the effective area determination unit 746 acquires the composite image data from the composite image data storage unit 732 and performs a binary process on the acquired composite image data, thereby performing a binary image of the composite image. Is generated (step S114). Specifically, the effective area determination unit 746 generates a map image that determines an image area and a blank area when color images are pasted together.

  Thereafter, the effective area determination unit 746 calculates the barycentric coordinates of the binary image of the combined image (step S115). Specifically, the effective area determination unit 746 extracts the outline of the image area from the binary image of the composite image, and calculates the barycentric coordinate (center) based on the extracted outline.

  Subsequently, the effective area determination unit 746 determines whether or not the center of gravity of the binary image of the composite image is within the image area (step S116). When the effective area determination unit 746 determines that the center of gravity of the binary image of the combined image is within the image area (step S116: Yes), the microscope system 1 proceeds to step S117 described later. On the other hand, when the effective area determination unit 746 determines that the center of gravity of the binary image of the composite image is not within the image area (step S116: No), the microscope system 1 proceeds to step S120 described later.

  In step S117, the effective area determination unit 746 executes rectangular extension determination processing on the binary image of the combined image (step S117).

  Here, the rectangular extension determination process executed by the effective area determination unit 746 will be described in detail. The effective area determination unit 746 performs five patterns of rectangular expansion determination processing. FIG. 5 is a diagram schematically illustrating the four-direction expansion of the first rectangle determination process executed by the effective area determination unit 746. In the following, the effective area of the binary image of the composite image is expressed in white and the ineffective area is expressed in black.

  As shown in FIG. 5, the effective area determination unit 746 simultaneously expands the rectangular area R1 from the center of gravity G1 of the binary image P1 in the vertical and horizontal directions with respect to the binary image P1 of the composite image (FIG. 5 ( a)). Thereafter, the effective area determination unit 746 further expands the rectangular area R1 from the center of gravity G1 in the vertical and horizontal directions with respect to the binary image P1 (FIG. 5B). Subsequently, the effective area determination unit 746 further expands the rectangular area R1 from the center of gravity G1 in the vertical and horizontal directions with respect to the binary image P1 (FIG. 5C). At this time, when the rectangular area R1 reaches the blank area N1 of the binary image P1, the effective area determination unit 746 stops the extension in the vertical direction. Thereafter, the effective area determination unit 746 expands the rectangular area R1 from the center of gravity G1 in the horizontal direction with respect to the binary image P1 (FIG. 5D). In FIG. 5, the description has been given with priority given to the extension in the vertical direction, but the extension in the horizontal direction may be given priority.

  By the way, in the four-direction expansion of the first rectangle determination process, the effective region determination unit 746 may end the expansion with a small rectangular region R1 unlike the predicted rectangular region. FIG. 6 is a diagram schematically illustrating a situation in which it is difficult to expand the four directions of the first rectangle determination process executed by the effective area determination unit 746.

  As shown in FIG. 6, the effective area determination unit 746 simultaneously extends the rectangular area R1 from the center of gravity G1 in the vertical and horizontal directions with respect to the binary image P1 (FIG. 6A). Thereafter, the effective area determination unit 746 further expands the rectangular area R1 from the center of gravity G1 in the vertical and horizontal directions with respect to the binary image P1 (FIG. 6B). At this time, when the rectangular area R1 reaches the blank area N1 of the binary image P1, the effective area determination unit 746 stops the downward and leftward expansion. Subsequently, the effective area determination unit 746 expands the rectangular area R1 upward and rightward from the center of gravity G1 with respect to the binary image P1 (FIG. 5C). In this case, the effective area determination unit 746 ends the expansion of the rectangular area R1 in an area smaller than the predicted effective area R2 of the binary image P1.

  Next, the second rectangle determination process performed by the effective area determination unit 746 will be described. FIG. 7 is a diagram schematically illustrating an outline of the horizontal priority extension of the second rectangle determination process performed by the effective area determination unit 746.

  As shown in FIG. 7, the effective area determination unit 746 searches for the end of the image area in the lateral direction (left-right direction) from the center of gravity G1 with respect to the binary image P1 (FIG. 7A). Thereafter, the effective area determination unit 746 searches the edge of the image area in the vertical direction (vertical direction) from the center of gravity G1 based on the lateral expansion result for the binary image P1 (FIG. 7B). . In this case, when the search in the vertical direction reaches the end of the image, the effective area determination unit 746 determines the rectangular area R1 as the maximum area of the effective area in the composite image.

  In addition, the effective area determination unit 746 may end expansion in a small rectangular area R1, unlike the predicted rectangular area, in the horizontal direction priority expansion of the second rectangular determination process. FIG. 8 is a diagram schematically illustrating a situation in which the second region determination process executed by the effective area determination unit 746 is not good at the lateral priority extension.

  As illustrated in FIG. 8, the effective area determination unit 746 predicts a binary image when the rectangular area R1 is preferentially expanded from the center of gravity G1 in the horizontal direction (left-right direction) with respect to the binary image P1. Expansion of the rectangular area R1 ends in an area smaller than the effective area R2 of P1.

  Next, a third rectangle determination process performed by the effective area determination unit 746 will be described. FIG. 9 is a diagram schematically illustrating an outline of the vertical priority extension of the third rectangle determination process executed by the effective area determination unit 746.

  As shown in FIG. 9, the effective area determination unit 746 searches the binary image P1 for the end of the image area in the vertical direction (vertical direction) from the center of gravity G1 (FIG. 9A). Thereafter, the effective area determination unit 746 searches the edge of the image area in the horizontal direction (left and right direction) from the center of gravity G1 with respect to the binary image P1 based on the extension result in the vertical direction (FIG. 9B). . In this case, when the search in the left-right direction reaches the end of the image, the effective area determination unit 746 determines this rectangular area R1 as the maximum area of the effective area in the composite image.

  In addition, in the vertical direction priority extension of the third rectangle determination process, the effective area determination unit 746 may end the expansion in a small rectangular area R1 unlike the predicted rectangular area. FIG. 10 is a diagram schematically illustrating a situation in which the third region determination process executed by the effective region determination unit is not good at the vertical priority extension.

  As shown in FIG. 10, the effective area determination unit 746 predicts the effective area of the binary image P1 when the binary image P1 is preferentially expanded from the center of gravity G1 in the horizontal direction (left-right direction). Expansion of the rectangular area R1 ends in an area smaller than R2.

  Next, a fourth rectangle determination process performed by the effective area determination unit 746 will be described. FIG. 11 is a diagram schematically illustrating an outline of 4-direction expansion + vertical direction priority expansion of the fourth rectangle determination process executed by the effective area determination unit 746.

  As shown in FIG. 11, the effective area determination unit 746 simultaneously expands the rectangular area R1 from the center of gravity G1 in four directions (up and down, left and right directions) with respect to the binary image P1 (FIG. 11A). Thereafter, the effective area determination unit 746 expands the rectangular area R1 with priority in the vertical direction from the center of gravity G1 with respect to the binary image P1 (FIG. 11B). In this case, when the rectangular area R1 reaches the blank area N1 of the binary image P1, the effective area determination unit 746 stops the vertical expansion and determines the rectangular area R1 as the maximum area of the effective area in the composite image. To do.

  Also, the effective area determination unit 746 may end the expansion in a small rectangular area R1 unlike the predicted rectangular area in the four-direction expansion + horizontal priority expansion of the fourth rectangular determination process. FIG. 12 is a diagram schematically illustrating a situation in which it is difficult to perform the four-direction extension + vertical direction priority extension of the fourth rectangle determination process executed by the effective area determination unit 746.

  As shown in FIG. 12, the effective area determination unit 746 predicts the effective area of the binary image P1 when the binary image P1 is preferentially expanded from the center of gravity G1 in the vertical direction (vertical direction). Expansion of the rectangular area R1 ends in an area smaller than R2.

  Next, a fifth rectangle determination process performed by the effective area determination unit 746 will be described. FIG. 13 is a diagram schematically illustrating an outline of 4-direction expansion + lateral direction priority expansion of the fifth rectangle determination process performed by the effective area determination unit 746.

  As shown in FIG. 13, the effective area determination unit 746 simultaneously expands the rectangular area R1 from the center of gravity G1 in four directions (up, down, left, and right directions) with respect to the binary image P1 (FIG. 13A). Thereafter, the effective area determination unit 746 expands the rectangular area R1 with priority from the center of gravity G1 in the horizontal direction with respect to the binary image P1 (FIG. 13B). In this case, when the rectangular area R1 reaches the blank area N1 of the binary image P1, the effective area determination unit 746 stops the lateral expansion and determines the rectangular area R1 as the maximum area of the effective area in the composite image. To do.

  In addition, the effective area determination unit 746 may end the expansion in a small rectangular area R1 unlike the predicted rectangular area in the four-direction extension + horizontal priority extension of the fifth rectangle determination process. FIG. 14 is a diagram schematically illustrating a situation in which the fifth rectangle determination process is not good at the four-direction extension + lateral direction priority extension.

  As illustrated in FIG. 14, the effective area determination unit 746 predicts the effective area of the binary image P1 when the binary image P1 is preferentially expanded from the center of gravity G1 in the horizontal direction (left-right direction). The expansion of the rectangular area R1 is finished in an area smaller than R2.

  Thus, the effective area determination unit 746 determines the area of the effective area of the combined image in each rectangular expansion determination process by sequentially performing the above-described five patterns of rectangular expansion determination processes.

  Returning to FIG. 4, the description of step S118 and subsequent steps will be continued. In step S118, the effective area determination unit 746 determines that the effective area of the composite image has the maximum area based on the determination result determined in the five-pattern rectangular expansion determination process. For example, the effective area determination unit 746 determines that the rectangular area R1 of the binary image P1 determined in the first rectangular expansion determination process is that the effective area of the composite image has the maximum area.

  Subsequently, based on the determination result of the effective area determination unit 746, the image shaping unit 747 generates a shaped image data having a rectangular shape by cutting out the area having the maximum effective area from the combined image (step S119). .

  Thereafter, the display control unit 749 causes the display unit 62 to display a shaped image corresponding to the shaped image data generated by the image shaping unit 747.

  Subsequently, the drive control unit 748 stores the shaped image data generated by the image shaping unit 747 in the image data storage unit 731 (step S121). Thereafter, the microscope system 1 ends this process.

  In step S122, the display control unit 749 causes the display unit 62 to display a warning that the effective region determination unit 746 has not been able to determine the effective region of the composite image. After step S122, the microscope system 1 ends this process.

  According to the first embodiment of the present invention described above, the effective area determination unit 746 determines an effective area including image data in the composite image, and the image shaping unit 747 determines the determination result of the effective area determination unit 746. In response, the effective area is cut out from the combined image to shape the shape of the combined image. As a result, it is easy for the user to measure the specimen S using the bonded image, and highly accurate measurement can be performed.

  Further, according to the first embodiment of the present invention, the effective area determination unit 746 determines an effective area having a rectangular shape in the combined image, and the image shaping unit 747 determines the rectangular shape determined by the effective area determination unit 746. Cut out the effective area to shape the combined image into a rectangular shape. As a result, when the user outputs a combined image in a report or the like, the combined image that shows only the effective area is obtained, so that the appearance is excellent.

  Furthermore, according to Embodiment 1 of the present invention, the effective area determination unit 746 determines the maximum area where the effective area is maximized in the composite image using each of a plurality of rectangular patterns, and the image shaping unit In step 747, the maximum area in which the effective area is maximized in the composite image is cut out from the composite image, and the composite image is shaped. Thereby, the effective area in the combined image can be used to the maximum extent.

  Furthermore, according to Embodiment 1 of the present invention, the display control unit 749 causes the display unit 62 to display a combined image (shaped image) corresponding to the combined image data shaped by the image shaping unit 747. As a result, the user can intuitively grasp the shaped combined image.

(Embodiment 2)
Next, the second embodiment of the present invention will be described. The second embodiment of the present invention differs only in the operation of the microscope system 1 according to the first embodiment described above, and the configuration of the microscope system is the first embodiment described above. It has the same configuration as the microscope system according to the above. For this reason, only the operation of the microscope system according to the second embodiment of the present invention will be described below.

  FIG. 15 is a flowchart illustrating an outline of processing performed by the microscope system 1 according to the second embodiment.

  As shown in FIG. 15, steps S201 to S214 correspond to steps S101 to S114 in FIG. 4, respectively.

  In step S215, the effective area determination unit 746 reduces the binary image. Specifically, the effective area determination unit 746 reduces the binary image by thinning out the pixels of the binary image so that the boundary between the image area and the blank area does not become ambiguous. For example, the effective area determination unit 746 performs reduction by thinning a predetermined number of times regardless of the image size of the binarized image. As a result, the same error (pixel) can be generated regardless of the image size. Further, the effective area determination unit 746 may reduce the binary image by thinning out the short side of the binary image until a predetermined reference size is reached. In this case, the larger the image, the smaller the error.

  Steps S216 to S219 correspond to steps S115 to S118 in FIG. 4, respectively.

  In step S220, the effective area determination unit 746 restores coordinates corresponding to the reduction ratio. Specifically, since the rectangular area R1 is a coordinate at a reduced size, the coordinates for the reduction ratio are restored. For example, the effective area determination unit 746 restores the rectangular area R1 to the size before reduction by restoring the reduced binary image P1 according to the reduction magnification.

  Subsequently, the effective area determination unit 746 performs an expansion determination process on the 1-size binary image (step S221). Specifically, the effective area determination unit 746 performs an expansion determination process according to the pixels of the blank area N1 included in the rectangular area R1. For example, in the case where the image size of the binary image is reduced from 10000 × 10000 to 5000 × 5000, the effective area determination unit 746 restores the reduced rectangular area R1 to the image size before the reduction, and the maximum in the rectangular area R1. Since the error is two pixels, the rectangular extension determination process may be performed for the two pixels. As shown in FIG. 16, the effective area determination unit 746 may not perform the rectangular extension determination process when the rectangular area R11 has no error with respect to the rectangular area R10 of the binary image P11 including the error. .

  Steps S222 to S225 correspond to steps S119 to S122 of FIG. 4, respectively.

  According to the second embodiment of the present invention described above, the effective area determination unit 746 determines the effective area including the image data in the composite image for the reduced image obtained by reducing the composite image. The effective area can be determined at a higher speed than in the embodiment. As a result, the time from when the image shaping unit 747 shapes the shape of the combined image to when the combined image is displayed on the display unit 62 can be instantaneously performed.

  In the first and second embodiments described above, the effective area determination unit 746 determines the rectangular area so that the effective area is the maximum area for the binary image of the combined image. Image data may be supplemented for invalid areas in the image. In this case, the effective area determination unit 746 determines the position of the invalid area with respect to the composite image, and the drive control unit 748 corresponds the stage 21 to the invalid area according to the determination result of the effective area determination unit 746. The image data in the invalid area of the composite image may be complemented by moving to the observation area of the imaging apparatus 4 and causing the imaging apparatus 4 to capture the image. Further, the combined image generation unit 745 may complement the invalid area from the peripheral image of the invalid area.

  In the first and second embodiments described above, the effective area determination unit 746 determines the rectangular area so that the effective area is the maximum area for the binary image of the combined image. Image data may be supplemented for invalid areas in the image. In this case, the effective area determination unit 746 determines the position of the invalid area with respect to the composite image, and the drive control unit 748 drives the revolver driving unit 222 according to the determination result of the effective area determination unit 746. By switching to the low magnification objective lens 231, the entire specimen sample S is photographed by the imaging device 4 to generate image data of the entire specimen sample S, and the image shaping unit 747 uses the image data to generate image data in the invalid area. It may be supplemented.

  In the first and second embodiments described above, a microscope system including a microscope apparatus, an imaging apparatus, a display input unit, and a control terminal has been described as an example. For example, via an objective lens and an objective lens for enlarging a specimen sample The present invention can also be applied to an imaging apparatus having an imaging function for imaging a specimen and a display function for displaying an image, such as a video microscope.

  In the first and second embodiments described above, the upright microscope apparatus has been described as an example of the microscope apparatus. However, the present invention can be applied to an inverted microscope apparatus, for example. Furthermore, the present invention can be applied to various systems such as a line apparatus incorporating a microscope apparatus.

  In the first and second embodiments described above, various settings of the microscope system are performed via the input unit. However, various settings of the microscope system may be performed via the touch panel.

  Moreover, although Embodiment 1 and 2 mentioned above demonstrated the electric stage as an example as a stage, even if it is a manual stage moved manually, for example, this invention is applicable.

  In the first and second embodiments described above, the display input unit and the control terminal are separately configured. However, for example, even a portable terminal in which the display input unit and the control terminal are integrally formed. Good.

  In the first and second embodiments, the stage 21 is movable in the Z direction by the motor 212. However, the objective lens 23 or the entire observation optical system including the objective lens 23 may be movable in the Z direction. .

DESCRIPTION OF SYMBOLS 1 Microscope system 2 Microscope apparatus 3 Microscope control part 4 Imaging apparatus 5 Imaging control part 6 Display input part 7 Control terminal 21 Stage 22 Revolver 23 Objective lens 24 Microscope main body 25 Light source for epi-illumination 41 Image pick-up element 51 AE process part 52 AF process Unit 61 Display communication unit 62 Display unit 63 Touch panel 71 Control communication unit 72 Input unit 73 Storage unit 74 Control unit 211 Stage drive unit 212 Motor 221 Mounter 222 Revolver drive unit 241 Illumination lens 242 Half mirror 243 Zoom lens unit 243a Zoom optical system 243b Zoom drive unit 244 Imaging lens 731 Image data storage unit 732 Bonded image data storage unit 741 Live image generation unit 742 Movement amount calculation unit 743 Movement amount determination unit 744 Captured image generation unit 745 Bonded image generation 746 effective region judging section 747 image shaping section 748 the drive control unit 749 display control unit

Claims (5)

In a microscope system capable of generating a bonded image in which a plurality of images corresponding to image data of the sample sample are bonded together by imaging the sample sample placed on the stage,
An effective area determination unit that determines a maximum area in which a rectangular effective area including pixel data is maximized in the combined image;
According to the determination result of the effective area determination unit, an image shaping unit that cuts out the maximum region from the combined image and shapes the shape of the combined image;
A microscope system comprising:   The microscope system according to claim 1, wherein the effective area determination unit determines the effective area for a reduced image obtained by reducing the combined image. The effective area determination unit, according to the maximum area in which the effective area is maximized within the bonded image with a plurality of patterns having a rectangular shape in claim 1 or 2, characterized in determine Teisu Rukoto Microscope system.   The effective area determination unit determines the effective area including the pixel data by performing a binarization process on the composite image. The described microscope system. The microscope system according to any one of claims 1 to 4, wherein the specimen sample placed on the stage is imaged by manually moving the stage.

Images