EP3382439B1 - Mikroskop, beobachtungsverfahren und bildverarbeitungsprogramm - Google Patents

Mikroskop, beobachtungsverfahren und bildverarbeitungsprogramm

Info

Publication number
EP3382439B1
EP3382439B1 EP15909327.7A EP15909327A EP3382439B1 EP 3382439 B1 EP3382439 B1 EP 3382439B1 EP 15909327 A EP15909327 A EP 15909327A EP 3382439 B1 EP3382439 B1 EP 3382439B1
Authority
EP
European Patent Office
Prior art keywords
region
fluorescent substance
image
objective lens
position information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP15909327.7A
Other languages
English (en)
French (fr)
Other versions
EP3382439A4 (de
EP3382439A1 (de
Inventor
Ichiro Sase
Yasutoshi Kaneko
Tatsuo Fukui
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nikon Corp
Original Assignee
Nikon Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nikon Corp filed Critical Nikon Corp
Publication of EP3382439A1 publication Critical patent/EP3382439A1/de
Publication of EP3382439A4 publication Critical patent/EP3382439A4/de
Application granted granted Critical
Publication of EP3382439B1 publication Critical patent/EP3382439B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365—Control or image processing arrangements for digital or video microscopes
    • G02B21/367—Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G01N21/645—Specially adapted constructive features of fluorimeters
    • G01N21/6456—Spatial resolved fluorescence measurements; Imaging
    • G01N21/6458—Fluorescence microscopy
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/02—Objectives
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/16—Microscopes adapted for ultraviolet illumination ; Fluorescence microscopes
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/24—Base structure
    • G02B21/26—Stages; Adjusting means therefor
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/368—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements details of associated display arrangements, e.g. mounting of LCD monitor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G06T15/08—Volume rendering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10056—Microscopic image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10064—Fluorescence image

Definitions

  • the present invention relates to a microscope, an observation method, and an image processing program.
  • STORM stochastic optical reconstruction microscopy
  • STORM is used to observe a specimen with fluorescent substances.
  • STORM activates a fluorescent substance at low density and causes only the activated fluorescent substance to emit light by irradiation of excitation light, thereby acquiring a fluorescence image in which images of fluorescence are sparsely distributed.
  • the images of fluorescence are individually separated, and the centroid positions of the individual images are able to be determined by Gaussian fitting or the like.
  • STORM generates (constructs) a high-resolution image by arranging point images at the centroid positions of a large number of fluorescent substances obtained from a large number of fluorescence images.
  • the structure of a specimen is able to be acquired three-dimensionally by determining the position of a fluorescent substance for each of a plurality of layers of the specimen and integrating the positions of fluorescent substances for the layers.
  • a first aspect of the present invention provides a microscope as recited in claim 1 below.
  • a second aspect of the present invention provides an observation method as recited in claim 8 below.
  • a third aspect of the present invention provides an image processing program as recited in claim 9 below.
  • a microscope according to the embodiment is, for example, a microscope using single-molecule localization microscopy, such as STORM and PALM.
  • a microscope 1 is usable for both fluorescence observation of a specimen labeled with one kind of fluorescent substance and fluorescence observation of a specimen labeled with two or more kinds of fluorescent substances.
  • the number of kinds of fluorescent substances (for example, reporter dyes) used for labelling is one.
  • the microscope 1 is able to generate three-dimensional super-resolution images.
  • the microscope 1 has a mode to generate a two-dimensional super-resolution image and a mode to generate a three-dimensional super-resolution image, and is able to switch the two modes.
  • the specimen may be the one that contains a live cell, the one that contains a cell fixed by a tissue fixing solution, such as a formaldehyde solution, or a tissue.
  • a fluorescent substance may be a fluorescent dye such as a cyanine dye or a fluorescent protein.
  • the fluorescent dye includes a reporter dye that emits fluorescence when irradiated with excitation light in a state of being activated (hereinafter referred to as "activated state").
  • the fluorescent dye may include an activator dye that activates the reporter dye when irradiated with activation light. When the fluorescent dye does not contain the activator dye, the reporter dye becomes activated when irradiated with the activation light.
  • the fluorescent dye is a dye pair, for example, in which two types of cyanine dye are combined with each other, examples of the dye pair including a Cy3-Cy5 dye pair (Cy3 and Cy5 are registered trademarks), a Cy2-Cy5 dye pair (Cy2 and Cy5 are registered trademarks), a Cy3-Alexa Fluor 647 dye pair (Cy3 and Alexa Fluor are registered trademarks), and a single type of dye, such as Alexa Fluor 647 (Alexa Fluor is a registered trademark).
  • the fluorescent protein is PA-GFP or Dronpa, for example.
  • Fig. 1 is a diagram illustrating a microscope 1 according to the first embodiment.
  • the microscope 1 includes a stage 2, a light source device 3, an illumination optical system 4, a first observation optical system 5, an imager 6, an image processor 7, and a control device 8.
  • the control device 8 includes a controller 42 that comprehensively controls the parts in the microscope 1.
  • the image processor 7 is provided in the control device 8, for example.
  • the stage 2 holds a specimen W to be observed.
  • the stage 2 is able to place the specimen W on its top surface.
  • the stage 2 may have a mechanism that moves the specimen W, such as an XY stage, and is not necessarily required to have a mechanism that moves the specimen W, such as a desk.
  • the microscope 1 is not necessarily required to include the stage 2.
  • the light source device 3 includes an activation light source 10a, an excitation light source 10b, a shutter 11a, and a shutter 11b.
  • the activation light source 10a emits activation light L to activate a part of a fluorescent substance included in the specimen W. It is assumed here that the fluorescent substance contains a reporter dye and does not contain an activator dye. When irradiated with the activation light L, the reporter dye in the fluorescent substance becomes an activated state capable of emitting fluorescence.
  • the fluorescent substance may contain a reporter dye and an activator dye. In this case, when receiving the activation light L, the activator dye activates the reporter dye.
  • the fluorescent substance may be a fluorescent protein such as PA-GFP and Dronpa, for example.
  • the excitation light source 10b emits excitation light L1 to excite at least a part of a fluorescent substance activated in the specimen W.
  • the fluorescent substance When irradiated with the excitation light L1 in the state of being activated, the fluorescent substance emits fluorescence or becomes inactivated.
  • the activation light L in the state of being inactivated hereinafter referred to as "inactivated state"
  • the fluorescent substance becomes activated state again.
  • the activation light source 10a and the excitation light source 10b include solid-state light sources such as laser light sources, for example, and each emit laser light having a wavelength corresponding to the type of fluorescent substance.
  • the emission wavelength of the activation light source 10a and the emission wavelength of the excitation light source 10b are selected from about 405 nm, about 457 nm, about 488 nm, about 532 nm, about 561 nm, about 640 nm, and about 647 nm.
  • the emission wavelength of the activation light source 10a is about 405 nm
  • the emission wavelength of the excitation light source 10b is selected from about 488 nm, about 561 nm, and about 647 nm.
  • the shutter 11a is controlled by the controller 42, and is able to switch between the state in which the activation light L from the activation light source 10a is allowed to pass and the state in which the activation light L is blocked.
  • the shutter 11b is controlled by the controller 42, and is able to switch between the state in which the excitation light L1 from the excitation light source 10b is allowed to pass and the state in which the excitation light L1 is blocked.
  • the light source device 3 includes a mirror 12, a dichroic mirror 13, an acousto-optic element 14, and a lens 15.
  • the mirror 12 is provided on the exit side of the excitation light source 10b.
  • the excitation light L1 from the excitation light source 10b is reflected by the mirror 12 to enter the dichroic mirror 13.
  • the dichroic mirror 13 is provided on the exit side of the activation light source 10a.
  • the dichroic mirror 13 has characteristics that transmit the activation light L and reflect the excitation light L1.
  • the activation light L that has transmitted through the dichroic mirror 13 and the excitation light L1 that has been reflected by the dichroic mirror 13 enter the acousto-optic element 14 through the same optical path.
  • the acousto-optic element 14 is an acousto-optic filter.
  • the acousto-optic element 14 is controlled by the controller 42, and is able to adjust the light intensity of the activation light L and the light intensity of the excitation light L1.
  • the acousto-optic element 14 is controlled by the controller 42, and is able to switch the states of the activation light L and the excitation light L1 between the state in which the light passes through the acousto-optic element 14 (hereinafter referred to as "light passing state”) and the state in which the light is blocked or its intensity is reduced by the acousto-optic element 14 (hereinafter referred to as "light blocking state").
  • the controller 42 controls the acousto-optic element 14 to irradiate a specimen with the activation light L and the excitation light L1 simultaneously.
  • the controller 42 controls the acousto-optic element 14 to irradiate a specimen with the activation light L and then the excitation light L1.
  • the lens 15 is a coupler, for example, and condenses the activation light L and the excitation light L1 from the acousto-optic element 14 onto a light guide member 16.
  • the microscope 1 does not necessarily include at least a part of the light source device 3.
  • the light source device 3 may be unitized and provided in the microscope 1 in a replaceable (attachable, detachable) manner.
  • the light source device 3 may be attached to the microscope 1 when the microscope 1 is used for observation.
  • the illumination optical system 4 emits the activation light L to activate a part of fluorescent substances included in the specimen W and the excitation light L1 to excite at least a part of activated fluorescent substances.
  • the illumination optical system 4 irradiates the specimen W with the activation light L and the excitation light L1 from the light source device 3.
  • the illumination optical system 4 includes the light guide member 16, a lens 17, a lens 18, a filter 19, a dichroic mirror 20, and an objective lens 21.
  • the light guide member 16 is an optical fiber, for example, and guides the activation light L and the excitation light L1 to the lens 17.
  • the lens 17 is a collimator, for example, and converts the activation light L and the excitation light L1 into parallel light.
  • the lens 18 condenses the activation light L and the excitation light L1 to the position of a pupil plane of the objective lens 21.
  • the filter 19 has characteristics that transmit the activation light L and the excitation light L1 and block at least a part of light beams having other wavelengths.
  • the dichroic mirror 20 has characteristics that reflect the activation light L and the excitation light L1 and transmit light (for example, fluorescence) in a predetermined wavelength band among light beams from the specimen W.
  • Light from the filter 19 is reflected by the dichroic mirror 20 to enter the objective lens 21.
  • the specimen W is disposed on a front focus plane of the objective lens 21 for observation.
  • the above-mentioned illumination optical system 4 irradiates specimen W with the activation light L and the excitation light L1.
  • the above-mentioned illumination optical system 4 is an example, and is changeable as appropriate. For example, a part of the above-mentioned illumination optical system 4 may be omitted.
  • the illumination optical system 4 may include at least a part of the light source device 3.
  • the illumination optical system 4 may include an aperture stop or an illumination stop.
  • the first observation optical system 5 forms an image of light from the specimen W.
  • the first observation optical system 5 forms an image of fluorescence from a fluorescent substance included in the specimen W.
  • the first observation optical system 5 includes an objective lens 21, a dichroic mirror 20, a filter 24, a lens 25, an optical path switching member 26, a lens 27, and a lens 28.
  • the first observation optical system 5 shares the objective lens 21 and the dichroic mirror 20 with the illumination optical system 4.
  • the optical path between the specimen W and the imager 6 is illustrated by solid lines. Fluorescence from the specimen W passes through the objective lens 21 and the dichroic mirror 20 to enter the filter 24.
  • the filter 24 has characteristics that selectively transmit light in a predetermined wavelength band among light beams from the specimen W.
  • the filter 24 blocks illumination light reflected by the specimen W, external light, stray light, and the like.
  • the filter 24 is unitized with the filter 19 and the dichroic mirror 20, and this filter unit 23 is provided so as to be replaceable.
  • the filter unit 23 may be replaced depending on the wavelength of light (for example, the wavelength of the illumination light L, the wavelength of excitation light L1) emitted from the light source device 3 or the wavelength of fluorescence exiting from the specimen W, or may use a single filter unit supporting a plurality of excitation wavelengths and fluorescence wavelengths.
  • the optical path switching member 26 is a prism, and is provided so as to be insertable to and removable from the optical path of the first observation optical system 5.
  • the optical path switching member 26 is inserted to and removed from the optical path of the first observation optical system 5 by a driver (not shown) controlled by the controller 42. In the state in which the optical path switching member 26 is inserted to the optical path of the first observation optical system 5, the optical path switching member 26 guides fluorescence from the specimen W to an optical path toward the imager 6 by internal reflection.
  • the lens 27 converts fluorescence that has exited from an intermediate image (fluorescence that has passed through the intermediate image plane 5b) into parallel light, and the lens 28 condenses light that has passed through the lens 27.
  • the first observation optical system 5 includes an astigmatic optical system (for example, a cylindrical lens 29).
  • the cylindrical lens 29 acts on at least a part of fluorescence from the specimen W, and generates astigmatism for at least a part of the fluorescence.
  • the astigmatic optical system such as the cylindrical lens 29 generates an astigmatic difference by generating astigmatism for at least part of the fluorescence.
  • the astigmatism is used to calculate the position of a fluorescent substance in the depth direction of the specimen W (the optical axis direction of the objective lens 21).
  • the cylindrical lens 29 is provided to be insertable to and removable from an optical path between the specimen W and the imager 6 (for example, an imaging sensor 40).
  • the cylindrical lens 29 is insertable to and removable from an optical path between the lens 27 and the lens 28.
  • the cylindrical lens 29 is disposed on the optical path in the mode to generate a three-dimensional super-resolution image, and is retracted from the optical path in the mode to generate a two-dimensional super-resolution image.
  • the microscope 1 includes a second observation optical system 30.
  • the second observation optical system 30 is used to set an observation range, for example.
  • the second observation optical system 30 includes, in order from the specimen W to a viewpoint Vp of an observer, an objective lens 21, a dichroic mirror 20, a filter 24, a lens 25, a mirror 31, a lens 32, a mirror 33, a lens 34, a lens 35, a mirror 36, and a lens 37.
  • the second observation optical system 30 shares the configurations from the objective lens 21 to the lens 25 with the first observation optical system 5.
  • Light from the specimen W passes through the lens 25, and then enters the mirror 31 in a state in which the optical path switching member 26 is retracted from the optical path of the first observation optical system 5.
  • the second observation optical system 30 forms an intermediate image of the specimen W on the optical path between the lens 35 and the lens 37.
  • the lens 37 is an eyepiece, and the observer is able to set an observation range by observing the intermediate image.
  • the imager 6 takes an image formed by the first observation optical system 5.
  • the imager 6 includes an imaging sensor 40 and a controller 41.
  • the imaging sensor 40 is a CMOS image sensor, and may be a CCD image sensor.
  • the imaging sensor 40 has a plurality of pixels two-dimensionally arranged, and has a structure in which a photoelectric conversion element such as a photodiode is disposed in each pixel.
  • charges accumulated in the photoelectric conversion element are read by a readout circuit.
  • the imaging sensor 40 converts the read charges into digital data, and outputs data (for example, image data) in the digital form in which pixel positions and grayscale values are associated with one another.
  • the controller 41 controls the imaging sensor 40 to operate based on a control signal input from the controller 42 for the control device 8, and outputs data on taken images to the control device 8.
  • the controller 41 outputs a charge accumulation period and a charge reading period to the control device 8.
  • the control device 8 generally controls the parts in the microscope 1.
  • the control device 8 includes a controller 42 and an image processor 7.
  • the controller 42 supplies, based on a signal indicating a charge accumulation period and a charge reading period (information on imaging timing) supplied from the controller 41, a control signal for switching between the light passing state that transmits light from the light source device 3 and the light blocking state that blocks light from the light source device 3 to the acousto-optic element 14.
  • the acousto-optic element 14 switches between the light passing state and the light blocking state based on the control signal.
  • the controller 42 controls the acousto-optic element 14 to control a period in which the specimen W is irradiated with the activation light L and a period in which the specimen W is not irradiated with the activation light L.
  • the controller 42 controls the acousto-optic element 14 to control a period in which the specimen W is irradiated with the excitation light L1 and a period in which the specimen W is not irradiated with the excitation light L1.
  • the controller 42 controls the acousto-optic element 14 to control the light intensity of the activation light L and the light intensity of the excitation light L1 that irradiate the specimen W.
  • the controller 41 may supply a control signal for switching between the light blocking state and the light passing state to the acousto-optic element 14 to control the acousto-optic element 14 based on the signal indicating the charge accumulation period and the charge reading period (information on imaging timing).
  • the controller 42 controls the imager 6 such that the imaging sensor 40 executes imaging.
  • the controller 42 acquires imaging results (data on taken images) from the imager 6.
  • the image processor 7 uses the imaging results of the imager 6 to perform image processing. Through this image processing, the position of a fluorescent substance is three-dimensionally acquired.
  • the controller 42 changes the relative positions of the objective lens 21 and the specimen W (stage 2) in the optical axis direction of the objective lens 21, thereby changing the imaging position in the optical axis direction of the objective lens 21, and the image processor 7 three-dimensionally acquires the positions of a fluorescent substance at a plurality of the imaging positions different in the optical axis direction of the objective lens 21.
  • the image processor 7 represents (replaces) the respective positions of fluorescent substances by (with) bright points, thereby acquiring a stereoscopic image (Z-Stack image, 3D-Stack image) spread in the optical axis direction of the objective lens 21.
  • FIG. 2 is a diagram conceptually illustrating the first observation optical system 5 and the imaging sensor 40.
  • an optical axis 5a of the first observation optical system 5 is an optical system that is bent by an optical path switch member 26.
  • the first observation optical system 5 is conceptually illustrated as a straight optical system (an optical system in which the optical axis 5a of the first observation optical system 5 is not bent).
  • the illustration of the configuration between the objective lens 21 and the cylindrical lens 29 is omitted.
  • a direction parallel to the optical axis 5a of the first observation optical system 5 is a Z direction, and two directions that are perpendicular to the Z direction and perpendicular to each other are an X direction and a Y direction.
  • the X direction is a horizontal direction of the imaging sensor 40
  • the Y direction is a vertical direction of the imaging sensor 40.
  • the cylindrical lens 29 is an optical member having power (refractive power) in only one of the vertical direction and the horizontal direction. It is assumed that the cylindrical lens 29 has the power in the XZ plane but does not have the power in the YZ plane.
  • the astigmatic optical system may use, instead of the cylindrical lens 29, a toroidal lens having the power in both the vertical direction and the horizontal direction and the power is different in the two directions.
  • Fig. 3(A) is a diagram illustrating optical paths of fluorescence from positions different in the Z direction in the specimen W.
  • Figs. 3(B) to 3(D) are conceptual diagrams illustrating respective images of fluorescence from a position Z1, a position Z2, and a position Z3 in the Z direction.
  • an optical path of fluorescence (indicated by solid lines in Fig. 3 ) from the position Z2 is described.
  • fluorescence from a fluorescent substance present at the position Z2 in the specimen W enters the cylindrical lens 29 as parallel light beams. The fluorescence is refracted by the cylindrical lens 29 and condensed, and is directed to the imaging sensor 40 (see Fig.
  • the width of the fluorescence image in the X direction becomes minimum at a position Z4 corresponding to a focus point on the XZ plane, and becomes larger as being apart from the position Z4 on the -Z side.
  • fluorescence from the fluorescent substance present at the position Z2 in the specimen W enters the cylindrical lens 29 as parallel light beams. This fluorescence is not substantially refracted by the cylindrical lens 29, and enters the lens 28 as parallel light. This fluorescence is directed to the imaging sensor 40 (see Fig. 2 ) while being condensed by being refracted by the lens 28.
  • the width of the fluorescence image in the Y direction becomes minimum at a position Z6 corresponding to a focus point on the YZ plane, and becomes larger as being apart from the position Z6 on the +Z side.
  • a fluorescence image Im2 at an intermediate position Z5 between the position Z4 and the position Z6 becomes an exact circle in which a width Wx of the fluorescence image in the X direction and a width Wy of the fluorescence image in the X direction are equal to each other as illustrated in Fig. 3(C) .
  • FIG. 3(B) is a diagram illustrating an optical path of fluorescence from a fluorescent substance present at the position Z1 in the specimen W and its fluorescence image.
  • the position Z1 is located on the opposite side (+Z side) of the imaging sensor 40 (see Fig. 2 ) across the position Z2.
  • the width of a fluorescence image in the X direction becomes minimum at a position apart from the position Z4 on the +Z side.
  • the width of the fluorescence image in the Y direction becomes minimum at a position Z5.
  • a fluorescence image Im1 at the position Z5 becomes an ellipse whose major axis is the X direction as illustrated in Fig. 3(B) .
  • an elliptic image whose major axis is the X direction is acquired as a fluorescence image from the fluorescent substance present at the position Z1 in the specimen W.
  • the position Z3 is located on the same side (-Z side) of the imaging sensor 40 (see Fig. 2 ) with respect to the position Z2.
  • the width of the image of fluorescence in the X direction becomes minimum at the position Z5.
  • the width of the image of fluorescence in the Y direction becomes minimum at a position away from the position Z6 on the side (-Z side) opposite to the specimen W.
  • An image Im3 of fluorescence at the position Z5 becomes an ellipse whose major axis is the Y direction as illustrated in Fig. 3(D) .
  • the image processor 7 illustrated in Fig. 1 performs elliptical Gaussian fitting to calculate the centroid position of the image of fluorescence, thereby calculating the position (XY coordinates) of the fluorescent substance in the XY plane.
  • the oblateness (ellipticity) of the fluorescence image changes depending on the position of the fluorescent substance in the Z direction, and the image processor 7 calculates the position (Z coordinate) of the fluorescent substance in the Z direction based on the oblateness of the image of fluorescence.
  • Fig. 4 is a diagram illustrating an example of a sequence of illumination and imaging.
  • the controller 42 irradiates the specimen W with the activation light L and the excitation light L1.
  • the controller 42 controls the imager 6 to take images in a plurality of frame periods Tf.
  • the controller 42 sets the focus position of the objective lens 21 to Zc1.
  • Symbols Z1-P1 to Z1-P represent respective taken images (imaging results) obtained in the frame periods Tf in the first image generation period T1.
  • the imaging position is changed, and an image generation period T1 corresponding to the next imaging position is started.
  • the imaging position is changeable by the controller 42 controlling the stage 2 to change the distance between the specimen W on the stage 2 and the objective lens 21 in the Z direction.
  • the imaging position is a front focus position of the objective lens 21 in the specimen W.
  • a second image generation period T1 following the first image generation period T1 the controller 42 sets the focus position of the objective lens 21 to Zc2.
  • Symbols Z2-P1 to Z2-Pn represent respective taken images (imaging results) obtained in frame periods Tf in the second image generation period T1.
  • the image generation period T1 is repeatedly provided until the imaging at a predetermined imaging position is completed.
  • n (n is a natural number of 2 or more) image generation periods T1 are provided, and in the n-th image generation period T1, the controller 42 sets the focus position of the objective lens 21 to Zcn.
  • Symbols Zn-P1 to Zn-Pn represent respective taken images (imaging results) obtained in frame periods Tf in the n-th image generation period T1.
  • the image processor 7 uses the imaging results to perform image processing.
  • the image processor 7 calculates the centroid position of a fluorescent image included in each of the taken images Z1-P1 to Z1-Pn as position information.
  • the image processor 7 calculates the centroid position of the fluorescence image, and generates three-dimensional distribution information (denoted with symbol Pa1) on centroid positions by using at least a part of a plurality of centroid positions corresponding to a plurality of fluorescence images included in the taken images Z1-P1 to Z1-Pn.
  • the image processor 7 generates three-dimensional distribution information (denoted with symbol Pa2) on centroid positions by using at least a part of a plurality of centroid positions corresponding to a plurality of fluorescence images included in a plurality of taken images Z2-P1 to Z2-Pn. Subsequently, the image processor 7 similarly generates three-dimensional distribution information (denoted with symbol Pan) on centroid positions by using a plurality of taken images Zn-P1 to Zn-Pn.
  • the image processor 7 may generate three-dimensional distribution information Pa by using at least a part of three-dimensional distribution information (Pa1 to Pan).
  • the image processor 7 may generate a three-dimensional image in which the centroid position is represented as a bright point and display the generated three-dimensional image on the display device 44 based on each piece of three-dimensional distribution information (Pa1 to Pan).
  • the image processor 7 may generate a three-dimensional image in which the centroid position is represented as a bright point and display the generated three-dimensional image on the display device 44 based on the three-dimensional distribution information Pa.
  • Three-dimensional distribution information or a three-dimensional image obtained at a predetermined imaging position is referred to as "layer" (for example, Pa1).
  • the centroid position of fluorescence images may be corrected among the taken images Z1-P1 to Z1-Pn.
  • position correction drift correction
  • one three-dimensional image may be generated based on the generated three-dimensional distribution information Pa1.
  • Fig. 5 includes conceptual diagrams illustrating an example of a three-dimensional image of a specimen imaged at a plurality of imaging positions in the first embodiment.
  • Fig. 5(A) illustrates a perspective view
  • Fig. 5(B) illustrates a sectional view.
  • Fig. 5 illustrates an example of three-dimensional images obtained by imaging at three imaging positions, and in the following, the images are referred to as "first layer Lz1", “second layer Lz2", and "third layer Lz3".
  • the number of imaging positions is not limited, and may be two or four or more. It is assumed in the following description that the layers have the same thickness, but the thickness of at least one of the layers may be different from those of the other layers.
  • the thickness (dimension in Z direction) of each layer is, for example, about several hundreds of nm (for example, 600 nm).
  • the first layer Lz1 is a three-dimensional image (layer) obtained by imaging at a first imaging position closest to the objective lens 21, and is a layer in contact with a cover glass.
  • the second layer Lz2 is a layer adjacent to (right above) the first layer Lz1, and is a three-dimensional image obtained by imaging at a second imaging position different from the first imaging position in the thickness direction of the specimen W (direction of the optical axis 21a of the objective lens 21).
  • the third layer Lz3 is a layer adjacent to (right above) the second layer Lz2, and is a three-dimensional image obtained by imaging at a third imaging position different from the first imaging position and the second imaging position in the thickness direction of the specimen (direction of the optical axis 21a of the objective lens).
  • the first imaging position, the second imaging position, and the third imaging position are set such that each of the first layer Lz1, the second layer Lz2, and the third layer Lz3 overlaps with a corresponding adjacent layer in the Z direction (thickness direction, the same direction as the optical axis 21a of the objective lens 21).
  • symbol Lm1 represents an overlapping region where the first layer Lz1 and the second layer Lz2 overlap with each other
  • symbol Lm2 represents an overlapping region where the second layer Lz2 and the third layer Lz3 overlap with each other.
  • the thicknesses (dimension in the Z direction) of the overlapping region Lm1 and the overlapping region Lm2 are set to any desired values, and for example, set to about several tens of percent (10% or more and 90% or less) of the thickness of each layer. It is assumed that each layer (for example, the first layer Lz1) does not overlap with a layer that is two layers away therefrom in the Z direction (for example, the third layer Lz3), but at least one of a plurality of layers (for example, the first layer) may overlap with a layer that is two or more layers away therefrom (for example, the third layer Lz3).
  • the image processor 7 uses the layers, which are imaging results, to calculate the respective positions of fluorescent substances included in each of the layers as described with reference to Fig. 3 and Fig. 4 .
  • the timing of imaging is different among the layers.
  • the relation of relative positions between the visual field and the fluorescent substance may shift in the XYZ directions among the layers due to the position shift between the objective lens 21 and the specimen W. It is therefore preferred that the image processor 7 correct the position shift (drift) in the XYZ directions among the layers as one processing in the image processing.
  • the image processor 7 corrects the second layer Lz2 with respect to the first layer Lz1 by using an overlapping region between imaging results (first layer Lz1) of the imager 6 at a first imaging position and imaging results (second layer Lz2) of the imager 6 at a second imaging position different from the first imaging position in the same direction as the optical axis 21a of the objective lens 21. Specifically, the image processor 7 uses the overlapping region between the first layer Lz1 and the second layer Lz2 to correct the positions of fluorescent substances in the second layer Lz2 in the XYZ directions, and correct information obtained from the imaging results of the imager 6 (for example, the positions of the fluorescent substances in the second layer Lz2).
  • Fig. 6 is a conceptual diagram illustrating correction processing by the image processor according to the first embodiment.
  • Symbol DQ1 in Fig. 6 represents a distribution in the XY plane of fluorescent substances (bright points Q1) in an overlapping region Lm1 between the first layer Lz1 and the second layer Lz2.
  • the image processor 7 extracts three-dimensional positions of fluorescent substances included in the distribution DQ1 by using three-dimensional information (three-dimensional point group data) on a distribution of fluorescent substances included in the first layer Lz1.
  • the image processor 7 determines a centroid position C1 of a plurality of bright points Q1 in the distribution DQ1.
  • the positions of the bright points Q1 are each expressed by three-dimensional coordinates, and when the weights of the bright points are equal to each other, the centroid position C1 is expressed by an arithmetic average of coordinates of the bright points Q1.
  • the centroid position C1 is expressed by a weighted average obtained by adding the weights of bright points to the coordinates of the bright points Q1.
  • Symbol DQ2 in Fig. 6 represents a distribution in the XY plane of fluorescent substances (bright points Q2) in the overlapping region Lm1 between the second layer Lz2 and the first layer Lz1.
  • the image processor 7 extracts three-dimensional positions of the fluorescent substances included in the distribution DQ2 from three-dimensional information (three-dimensional point group data) on a distribution of fluorescent substances included in the second layer Lz2.
  • the image processor 7 determines a centroid position C2 of a plurality of bright points Q2 in the distribution DQ2.
  • arithmetic average or weighted average may be used similarly to the centroid position C1.
  • the image processor 7 corrects a position shift between the positions of fluorescent substances included in the first layer Lz1 and the positions of fluorescent substances included in the second layer Lz2.
  • a vector V1 indicating the difference between the centroid position C2 and the centroid position C1 corresponds to a position shift amount (drift amount) of the second layer Lz2 with respect to the first layer Lz1.
  • the image processor 7 corrects the positions of the fluorescent substances included in the second layer Lz2 such that the difference between the positions of fluorescent substances included in the overlapping region Lm1 in the imaging results of the first layer Lz1 and the positions of fluorescent substances included in the overlapping region Lm1 in the imaging results of the second layer Lz2 is reduced.
  • the image processor 7 calculates a vector obtained by inverting the direction (symbol) from the vector V1 as a correction amount.
  • Symbol DQ3 in Fig. 6 represents a distribution of fluorescent substances (bright points Q3) in the second layer Lz2 obtained from the imaging results of the second layer Lz2.
  • the distribution DQ3 corresponds to a distribution of fluorescent substances in the overlapping region Lm1 in the second layer Lz2 and in regions other than the overlapping region Lm1.
  • bright points Q3 before correction are indicated by dotted lines
  • corrected bright points Q3 are indicated by solid lines.
  • the image processor 7 corrects each position of bright points Q3 included in the distribution DQ3 by a vector V2 corresponding to a correction amount.
  • the vector V2 is a vector obtained by inverting the orientation of the vector V1 corresponding to the position shift amount.
  • the second layer Lz2 is positioned with respect to the first layer Lz1.
  • the image processor 7 similarly positions the third layer Lz3 illustrated in Fig. 5 with respect to the corrected second layer.
  • the corrected second layer Lz2 is positioned with the first layer Lz1, and hence the corrected third layer Lz3 matches with the first layer Lz1 in terms of position relation.
  • the image processor 7 performs the processing to position an adjacent layer with a layer (for example, the first layer Lz1) serving as a position reference and the processing to position an adjacent layer to the positioned layer in this order, thereby positioning a plurality of layers with the reference layer.
  • the centroid position C1 and the centroid position C2 are calculated to calculate a position shift amount between the first layer Lz1 and the second layer Lz2, but the position shift amount between the first layer Lz1 and the second layer Lz2 may be calculated by analyzing autocorrelation between the distribution DQ1 and the distribution DQ2. Specifically, the autocorrelation between the distribution DQ1 and the distribution DQ2 may be analyzed to correct a position shift between the first layer Lz1 and the second layer Lz2.
  • the control device 8 is communicably connected to a storage device (memory) 43 and a display device (display) 44.
  • the display device 44 is a liquid crystal display.
  • the display device 44 displays various kinds of images, such as images representing various settings of the microscope 1 images taken by the imager 6, and images generated from the taken images.
  • the controller 42 controls the display device 44 to display various kinds of images on the display device 44.
  • the controller 42 supplies data on images (for example, super-resolution images such as STORM images and PALM images) generated by the image processor 7 to the display device 44, and displays the images on the display device 44.
  • the microscope 1 is able to display super-resolution images of a specimen W to be observed as a live video.
  • the storage device 43 is a magnetic disk or an optical disc, and stores therein various kinds of data, such as data on various settings of the microscope 1, data on images taken by the imager 6, and data on images generated by the image processor 7.
  • the controller 42 is able to supply data on super-resolution images stored in the storage device 43 to the display device 44, and display the super-resolution images on the display device 44.
  • the controller 42 controls the storage device 43 to store various kinds of data in the storage device 43.
  • Fig. 7 is a flowchart illustrating an example of the observation method according to the embodiment.
  • the illumination optical system 4 irradiates a specimen W with illumination light (activation light, excitation light).
  • the first observation optical system 5 forms an image of light (for example, fluorescence) from the specimen W.
  • the imager 6 takes the image formed by the first observation optical system 5.
  • image data representing the imaging result by the imager 6 is supplied to the image processor 7.
  • the controller 42 determines in Step S4 whether to change the imaging position.
  • the controller 42 determines to change the imaging position (Yes in S4).
  • the imager 6 controls at least one of the objective lens 21 and the stage 2 to change the relative positions between the specimen W on the stage 2 and the objective lens 21 in the optical axis direction of the objective lens 21.
  • the controller 42 returns to Step S1, and repeatedly performs processing from Step S1 to Step S4.
  • Step S4 when the imaging of the specimen W at imaging positions Z1 to Zn designated in advance (determined in advance) has been completed, the controller 42 determines not to change the imaging position (No in Step S4).
  • Step S6 the image processor 7 corrects the positions of fluorescent substances in at least a part of the first layer Lz1 to the n-th layer Lzn acquired at the imaging positions Z1 to Zn, respectively.
  • the correction method for example, the method described above with reference to Fig. 6 is used.
  • Step S7 the image processor 7 generates an image by using the correction results. For example, the image processor 7 generates a three-dimensional image by using at least a part of bright points included in the first layer Lz1 and at least a part of corrected bright points included in the second layer Lz2 to the n-th layer Lzn.
  • the region used for the image processor 7 to calculate the correction amount may be the entire overlapping region Lm1 or a partial region (for example, a part on XY plane or a part in Z direction).
  • the image processor 7 may calculate the centroid position of bright points by using all or a part of bright points included in the region used to calculate the correction amount.
  • the image processor 7 may calculate the centroid position of bright points by selectively using bright points corresponding to a fluorescence image whose brightness is equal to or more than a threshold among fluorescence images.
  • the centroid position may be calculated by using bright points corresponding to a fluorescence image obtained by irradiating the specimen with light having a given wavelength.
  • the centroid position may be calculated by determining centroid positions by using bright points corresponding to fluorescence images obtained for the wavelengths and averaging the centroid positions for the wavelengths.
  • the following describes a second embodiment.
  • the same configurations as those in the above-mentioned embodiment are denoted with the same reference numerals, and the descriptions thereof are omitted or simplified.
  • an example of position correction of a fluorescent substance in the case where a third layer Lz3 overlaps with a first layer Lz1 and a second layer Lz2 is described.
  • Fig. 8 includes conceptual diagrams illustrating an example of layers set for a specimen in the second embodiment.
  • Fig. 8(A) illustrates a perspective view
  • Fig. 8(B) illustrates a sectional view.
  • Fig. 8 illustrates three layers (first layer Lz1, second layer Lz2, and third layer Lz3), but the number of layers is not limited and may be four or more.
  • the first layer Lz1 and the second layer Lz2 overlap with each other in an overlapping region Lm1 in the same direction as the optical axis 21a (see Fig. 1 ) of the objective lens 21.
  • the thickness of the overlapping region Lm1 is about 2/3 of the thickness of the second layer Lz2.
  • the second layer Lz2 and the third layer Lz3 overlap with each other in an overlapping region Lm2 in the same direction as the optical axis 21a (see Fig. 1 ) of the objective lens 21.
  • the thickness of the overlapping region Lm2 is about 2/3 of the thickness of the third layer Lz3.
  • the third layer Lz3 overlaps with the first layer Lz1 in the overlapping region Lm3 in the overlapping region Lm2.
  • the overlapping region Lm3 between the third layer Lz3 and the first layer Lz1 is a part corresponding to about 1/3 of the thickness of the third layer Lz3.
  • Fig. 9 includes conceptual diagrams illustrating correction processing by the image processor according to the second embodiment.
  • Fig. 9(A) corresponds to processing to calculate a position shift amount of the third layer Lz3 with respect to the first layer Lz1
  • Fig. 9(B) corresponds to processing to calculate a position shift amount of the third layer Lz3 with respect to the second layer Lz2.
  • Symbol DQ4 in Fig. 9(A) represents a distribution of fluorescent substances (bright points Q4) in the overlapping region Lm3 in the first layer Lz1.
  • the image processor 7 extracts three-dimensional positions of the fluorescent substances included in the distribution DQ4 from three-dimensional information (three-dimensional point group data) on the distribution of fluorescent substances included in the first layer Lz1.
  • the image processor 7 determines a centroid position C4 of a plurality of bright points Q4 in the distribution DQ4.
  • arithmetic average or weighted average may be used as described in the first embodiment.
  • Symbol DQ5 in Fig. 9(A) represents a distribution of fluorescent substances (bright points Q5) included in the overlapping region Lm3 in the third layer Lz3.
  • the image processor 7 extracts three-dimensional positions of the fluorescent substances included in the distribution DQ5 from three-dimensional information (three-dimensional point group data) on the distribution of the fluorescent substances included in the third layer Lz3.
  • the image processor 7 determines a centroid position C5 of a plurality of bright points Q5 in the distribution DQ5.
  • arithmetic average or weighted average may be used as described in the first embodiment.
  • a vector V3 representing the difference between the centroid position C5 and the centroid position C4 corresponds to a position shift amount of the third layer Lz3 with respect to the first layer Lz1.
  • Symbol DQ6 in Fig. 9(B) represents a distribution of fluorescent substances (bright points Q6) in the overlapping region Lm2 in the second layer Lz2 positioned with respect to the first layer Lz1 as described in the first embodiment.
  • the image processor 7 extracts the three-dimensional positions of the fluorescent substances included in the distribution DQ6 from three-dimensional information (corrected three-dimensional point group data) on a distribution of fluorescent substances included in the second layer Lz2.
  • the image processor 7 determines a centroid position C6 of a plurality of bright points Q6 in the distribution DQ6.
  • arithmetic average or weighted average may be used as described in the first embodiment.
  • Symbol DQ7 in Fig. 9(B) represents a distribution of fluorescent substances (bright points Q7) included in the overlapping region Lm2 in the third layer Lz3.
  • the image processor 7 extracts the positions of the fluorescent substances included in the distribution DQ7 from three-dimensional information (three-dimensional point group data) on a distribution of fluorescent substances included in the third layer Lz3.
  • the image processor 7 determines a centroid position C7 of a plurality of bright points Q7 in the distribution DQ7.
  • arithmetic average or weighted average may be used as described in the first embodiment.
  • a vector V4 representing the difference between the centroid position C7 and the centroid position C6 corresponds to a corrected position shift amount of the third layer Lz3 with respect to the second layer Lz2.
  • Fig. 10 includes conceptual diagrams illustrating an example of a method of calculating the correction amount by the image processor.
  • the image processor 7 calculates a vector V5, which is obtained by averaging the vector V3 (symbol DQ5 in Fig. 9(A) ) indicating the position shift amount of the third layer Lz3 with respect to the first layer Lz1 and the vector V4 (symbol DQ7 in Fig. 9(B) ) indicating the position shift amount of the third layer Lz3 with respect to the corrected second layer Lz2, as a position shift amount of the third layer Lz3.
  • Fig. 10(A) the image processor 7 calculates a vector V5, which is obtained by averaging the vector V3 (symbol DQ5 in Fig. 9(A) ) indicating the position shift amount of the third layer Lz3 with respect to the first layer Lz1 and the vector V4 (symbol DQ7 in Fig. 9(B) ) indicating the position shift amount of the third layer
  • V3/2 and V4/2 indicate a half vector of the vector V3 and a half vector of the vector V4, respectively.
  • the image processor 7 calculates a vector V6 obtained by inverting the orientation of the vector V5.
  • the image processor 7 uses the vector V6 as a correction amount for the third layer Lz3 to correct the positions of fluorescent substances (bright points) included in the third layer Lz3.
  • the image processor 7 weights information obtained from imaging results of the imager 6 performed on the overlapping region Lm3 between the first layer Lz1 and the third layer Lz3 (for example, the vector V3 in symbol DQ5 in Fig. 9(A) ) and information obtained from imaging results of the imager 6 performed on the overlapping region Lm2 between the second layer Lz2 and the third layer Lz3 (for example, the vector V4 in symbol DQ7 in Fig. 9(B) ), thereby correcting the positions of the fluorescent substances included in the third layer Lz3.
  • the image processor 7 weights information obtained from imaging results of the imager 6 performed on the overlapping region Lm3 between the first layer Lz1 and the third layer Lz3 (for example, the vector V3 in symbol DQ5 in Fig. 9(A) ) and information obtained from imaging results of the imager 6 performed on the overlapping region Lm2 between the second layer Lz2 and the third layer Lz3 (for example, the vector V4 in symbol DQ7 in Fig. 9(
  • the image processor 7 calculates a vector V7 obtained by weighting and combining the vector V3 and the vector V4 as a vector indicating a position shift amount of the third layer Lz3.
  • aV3 and bV4 represent a vector that is a times as large as the vector V3 and a vector that is b times as large as the vector V4.
  • the weighting coefficient a is determined in accordance with time from imaging for the first layer Lz1 to imaging for the third layer Lz3, and is set such that the weighting coefficient a becomes smaller as the time becomes longer.
  • the weighting coefficient b is determined in accordance with time from imaging for the second layer Lz2 to imaging for the third layer Lz3, and is set such that the weighting coefficient b becomes smaller as the time becomes longer.
  • the weighting coefficients a and b may be set in consideration of other parameters than the time between the imaging.
  • the weighting coefficients a and b may be set in consideration of the distance between layers or the number of bright points used to calculate the centroid position of the bright points.
  • the image processor 7 calculates a vector V8 obtained by inverting the orientation of the vector V7.
  • the image processor 7 uses the vector V8 as a correction amount for the third layer Lz3, and corrects the positions of fluorescent substances (bright points) included in the third layer Lz3.
  • the image processor 7 calculates a correction amount of the position of a fluorescent substance included in the second layer Lz2 by selectively using a region designated from among overlapping regions between the first layer Lz1 and the second layer Lz2. As preprocessing for correction processing, the image processor 7 generates information (hereinafter referred to as "reference information") on bright points in each layer obtained from imaging results of the imager 6.
  • Fig. 11 is an image in which the number of bright points included in each of a plurality of regions AR in a predetermined layer (for example, the first layer Lz1) is visualized (with color, contrast, or numerical values).
  • the controller 41 generates the image illustrated in Fig. 11 , and displays the image on the display device 44 (see Fig. 1 ). While looking at the image illustrated in Fig. 11 displayed on the display device 44, for example, the user is able to designate a region in which the number of bright points is relatively large as a region used to calculate a correction amount.
  • the use of the region in which the number of bright points is relatively large to perform correction processing enables improved accuracy of correction.
  • the region AR used to calculate the correction amount may be designated for each layer.
  • the image processor 7 performs correction processing for each of a plurality of regions set for a plurality of layers.
  • Fig. 12 includes conceptual diagrams illustrating an example of correction processing by the image processor 7 according to the fourth embodiment.
  • the image processor 7 calculates, for a plurality of regions AR1 to AR4, centroid positions C11 to C14 of bright points Q11 in the overlapping region Lm1 between the first layer Lz1 and the second layer Lz2 in the first layer Lz1.
  • the image processor 7 calculates, for regions AR1 to AR4, centroid positions C15 to C18 of bright points Q12 in the overlapping region Lm1 between the first layer Lz1 and the second layer Lz2 in the second layer Lz2.
  • the image processor 7 calculates, for the regions AR1 to AR4, vectors V11 to V14 corresponding to the differences between the centroid position of the bright points Q11 and the centroid position of the bright points Q12.
  • the image processor 7 uses vectors obtained by inverting the orientations of the vectors V11 to V14 as correction amounts to correct the positions of bright points Q12 in the regions AR1 to AR4.
  • the image processor 7 is able to generate an image in which the vectors V11 to V14 corresponding to position shift amounts for the regions AR1 to AR4 are visualized.
  • Fig. 12(C) is an example of an image G in which the vectors V11 to V14 are visualized.
  • the controller 42 in Fig. 1 displays the image G on the display device 44. While looking at the image G, the user is able to designate a region for correction from among the regions AR1 to AR4. For example, in Fig. 12(C) , the vector V12 corresponding to a position shift amount in the region AR2 is relatively large, and the user may acquire images again.
  • Fig. 12(C) the vector V12 corresponding to a position shift amount in the region AR2 is relatively large, and the user may acquire images again.
  • Fig. 13 includes conceptual diagrams illustrating another example of the correction processing by the image processor 7 according to the fourth embodiment.
  • the image processor 7 calculates vectors V31 to V33 corresponding to position shift amounts for a plurality of regions AR11 to AR13 in the second layer Lz2.
  • the second layer Lz2 is sectioned into 36 regions in six rows and six columns, and the three regions AR11 to AR13 and the vectors V31 to V33 among the regions are representatively illustrated.
  • the image processor 7 corrects the positions of bright points in the second layer Lz2 based on the relation between a distance from a reference position (for example, visual center Vc) to a reference position in each region (for example, center in each region) and a position shift amount in the region.
  • a reference position for example, visual center Vc
  • a reference position in each region for example, center in each region
  • a position shift amount in the region for example, a position shift amount in the region.
  • Fig. 13(B) is a conceptual diagram illustrating an example of the relation between the distance from the reference position (for example, visual center) and the position shift amount.
  • Symbols PL11, PL12, and PL13 are plots corresponding to respective predetermined positions in the regions AR11 to AR13.
  • the image processor 7 uses the symbols PL11, PL12, and PL13 to calculate the relation between the distance from the visual center and the position shift amount by a formula such as an interpolant or an approximate formula, and uses the formula to calculate a correction amount at each position in the regions AR11 to AR13 in the second layer Lz2.
  • the image processor 7 corrects the positions of the bright points included in the regions AR11 to AR13 based on the calculated correction amount at each position.
  • the position shift amount has a non-linear relation with the distance from the visual center, and the image processor 7 performs non-linear correction.
  • linear correction may be performed.
  • the image processor 7 calculates information (hereinafter referred to as "difference information") indicating a difference between the corrected position of a fluorescent substance included in an overlapping region Lm1 in a second layer Lz2 and the position of the fluorescent substance in the overlapping region Lm1 in a first layer Lz1.
  • Fig. 14 includes conceptual diagrams illustrating an example of images displayed on the display device in the fifth embodiment.
  • Figs. 14(A) and 14(B) illustrate the positions of bright points included in an overlapping region Lm1 between a first layer Lz1 and a second layer Lz2.
  • symbols Qa are bright points included in the first layer Lz1
  • symbols Qb are corrected bright points included in the second layer Lz2.
  • the image processor 7 represents the bright points Qa and the bright points Qb in a region of interest (ROI) BR designated by a user with different colors (for example, bright points Qa are in red and bright points Qb are in blue) or different brightness in a distinguishable manner.
  • ROI region of interest
  • the image processor 7 may similarly represent bright points before correction included in the first layer Lz1 and bright points before correction included in the second layer Lz2 with different colors or different brightness.
  • the image processor 7 is able to generate an image indicating a distribution of bright points Qa and bright points Qb as seen from the +Z side (from above), and as illustrated in Fig. 14(B) , able to generate an image indicating a distribution of the bright points Qa and the bright points Qb as seen from the -Y side (from side).
  • the user is able to grasp the correction results for each region of interest BR by looking at the image indicating the distribution of the bright points Qa and the bright points Qb, and able to determine whether to take an image again.
  • the user is able to grasp whether the correction has succeeded for each region of interest BR, and able to determine whether to change correction conditions and whether to perform correction again.
  • the image processor 7 is able to calculate a position shift amount between a distribution of bright points Qa and a distribution of bright points Qb in a region of interest BR.
  • the image processor 7 is able to calculate the centroid position of a plurality of the bright points Qa and the centroid position of a plurality of the bright points Qb in the region of interest BR, and calculate the difference therebetween as a position shift amount.
  • a smaller difference of the centroid positions means a smaller shift amount between positions of fluorescent substances included in a first layer Lz1 and corrected positions of fluorescent substances included in a second layer Lz2.
  • a correlation coefficient between a distribution of the bright points Qa and a distribution of the bright points Qb in a region of interest BR may be calculated.
  • a larger correlation coefficient means a smaller shift amount between positions of fluorescent substances included in a first layer Lz1 and corrected positions of fluorescent substances included in a second layer Lz2.
  • a threshold may be provided to the position shift amount.
  • bright points Qa and bright points Qb are displayed in different colors (for example, the bright points Qa are in red and the bright points Qb are in blue), and in a region of interest BR where the position shift amount is less than the threshold, the bright points Qa and the bright points Qb may be displayed in the same color (for example, green).
  • hatched bright points are displayed in the same color, and bright points indicated by chain lines are displayed in different colors. In this manner, the user is able to visually grasp the position shift amount for each of a plurality of regions of interest BR.
  • the distance between a bright point Qa and a bright point Qb having a correspondence relation may be calculated. For example, when the distance between a bright point Qa and a bright point Qb having a correspondence relation is less than a threshold, the image processor 7 may display the bright point Qa and the bright point Qb in the same color (for example, green), and when the distance between the bright point Qa and the bright point Qb having a correspondence relation is equal to or more than the threshold, the image processor 7 may display the bright point Qa and the bright point Qb in different colors (for example, the bright point Qa is in red and the bright point Qb is in blue). In this manner, the user is able to visually grasp a position shift amount between a bright point Qa and a bright point Qb in a region of interest BR.
  • a threshold the image processor 7 may display the bright point Qa and the bright point Qb in the same color (for example, green), and when the distance between the bright point Qa and the bright point Qb having a correspondence relation is equal to or more
  • Fig. 15 is an example in which position shift amounts of bright points included in an overlapping region between the first layer Lz1 and the second layer Lz2 are quantitatively displayed.
  • the image processor 7 calculates, as a position shift amount, an index (for example, difference of centroid positions, correlation coefficient) indicating a position shift amount between a bright point Qa and a bright point Qb for each of a plurality of regions AR.
  • the image processor 7 displays an index value (S(1, 1), S(1, 2), ...) on the display device 44 for each region. This display enables the user to quantitatively grasp the position shift amounts.
  • Fig. 16 is a diagram illustrating a microscope 1 according to the sixth embodiment.
  • the microscope 1 is obtained by applying a microscope described in WO 2008/069220 .
  • the microscope 1 includes, in order from a light source 50 to a specimen W on a stage 2, a lens 51, a field stop 53 (illumination stop), a semitransparent mirror 54, and an objective lens 21.
  • a light source 50 to a specimen W on a stage 2
  • the microscope 1 includes, in order from a light source 50 to a specimen W on a stage 2, a lens 51, a field stop 53 (illumination stop), a semitransparent mirror 54, and an objective lens 21.
  • the microscope 1 includes, in the order in which light from the specimen W travels to the imager 6, the objective lens 21, the semitransparent mirror 54, a mirror 55, a lens 56, an optical path brancher 57, an optical path length adjuster 58, and an astigmatic optical system 59.
  • the optical path brancher 57 includes a semitransparent mirror 60, a semitransparent mirror 61, and a mirror 62.
  • the optical path brancher 57 branches light that has entered the semitransparent mirror 60 into transmitted light and reflected light.
  • the optical path brancher 57 branches light that has been reflected by the semitransparent mirror 60 and entered the semitransparent mirror 61 into transmitted light and reflected light.
  • the optical path length adjuster 58 includes a parallel plate 65 that light that has been transmitted through the semitransparent mirror 60 enters, a parallel plate 66 that light that has been reflected by the semitransparent mirror 61 enters, and a parallel plate 67 that light that has been transmitted through the semitransparent mirror 61 and reflected by the mirror 62 enters.
  • the thicknesses and refractive indices of the parallel plates 65 to 67 are adjusted such that the imaging sensors Z1a, Z2a, and Z3a and the position Z1, the position Z2, and the position Z3 in the Z direction are optically conjugate with each other and that the distance between the position Z1 and the position Z3 becomes a desired distance.
  • the imager 6 includes a plurality of imaging sensors Z1a, Z2a, and Z3a that are provided on one-to-one correspondence with the respective branched optical paths.
  • Each of the imaging sensors Z1a, Z2a, and Z3a takes an image of light that has passed through a corresponding optical path.
  • the imaging sensors Z1a, Z2a, and Z3a are optically conjugate with the positions Z1, Z2, and Z3 in the Z direction, respectively.
  • the imager 6 is able to collectively take images of light from positions different in the thickness direction in the specimen W by using the imaging sensors.
  • the imaging sensor Z1a acquires an image focused at the position Z1
  • the imaging sensor Z2a acquires an image focused at the position Z2
  • the imaging sensor Z3a acquires an image focused at the position Z3.
  • the microscope 1 is able to collectively acquire images of light from positions in the specimen W different in the optical axis direction of the objective lens 21 in the state in which the specimen W and the objective lens 21 are positioned with each other in the optical axis direction.
  • the image processor 7 described in the above-mentioned embodiments is applicable also to this microscope 1.
  • Fig. 17 is a diagram conceptually illustrating the operation of the microscope 1 according to the sixth embodiment.
  • the controller 42 sets the focus position of the objective lens 21 to a first position, and controls the imager 6 to execute first imaging processing SH1.
  • the imaging sensor Z1a acquires a taken image P1-Z1 focused at the position Z1
  • the imaging sensor Z2a acquires a taken image P1-Z2 focused at the position Z2
  • the imaging sensor Z3a acquires a taken image P1-Z3 focused at the position Z3.
  • the controller 42 sets the focus position of the objective lens 21 to a second position, and controls the imager 6 to execute second imaging processing SH2.
  • the imaging sensor Z1a acquires a taken image P2-Z3 focused at the position Z3, the imaging sensor Z2a acquires a taken image P2-Z4 focused at the position Z4, and the imaging sensor Z3a acquires a taken image P2-Z5 focused at the position Z5.
  • the controller 42 similarly changes the focus position of the objective lens 21, and controls the imager 6 to execute imaging processing. It is assumed that the number of times of the imaging processing is three, but the number of times of the imaging processing is freely set.
  • the imaging sensor Z1a acquires a taken image P3-Z5 focused at the position Z5
  • the imaging sensor Z2a acquires a taken image P3-Z6 focused at the position Z6, and the imaging sensor Z3a acquires a taken image P3-Z7 focused at a position Z7.
  • the image processor 7 uses the taken images (P1-Z3, P2-Z3) whose positions in the same direction as the optical axis 21a of the objective lens 21 are the same in the first imaging processing SH1 and the second imaging processing SH2 to correct a position shift of the taken images (P2-Z3, P2-Z4, and P2-P5) obtained by the second imaging processing SH2. Through this correction, taken images (P2-Z3c, P2-Z4c, P2-P5c) are obtained.
  • the imaging processor 7 uses, among the corrected taken images (P2-Z3c, P2-Z4c, P2-P5c), the taken images (P2-Z5c, P3-Z5) whose positions in the same direction as the optical axis 21a of the objective lens 21 are the same in the second imaging processing SH2 and the third imaging processing SH3 to correct a position shift of the taken images (P3-Z5, P3-Z6, P3-P7) obtained by the third imaging processing SH3. Through this correction, taken images (P3-Z5c, P2-Z6c, P2-P7c) are obtained.
  • the image processor 7 generates a three-dimensional image PZ by using the taken images (P1-Z1, P1-Z2, P1-P3) obtained by the first imaging processing, the corrected taken images (P2-Z4c, P2-P5c) obtained by the second imaging processing, and the corrected taken images (P2-Z6c, P2-P7c) obtained by the third imaging processing.
  • microscopes capable of collectively acquiring images of light from positions of a specimen W different in the optical axis direction of an objective lens 21 in a state in which the specimen W and the objective lens 21 are positioned in the optical axis direction include microscopes described in Literature 1 ( Paul A. Dalgarno, Heather I. C. Dalgarno, A confusionlie Putoud, Robert Lambert, Lynn Paterson, David C. Logan, David P. Towers, Richard J. Warburton, and Alan H. Greenaway, "Multiplane imaging and three dimensional nanoscale particle tracking in biological microscopy", Optics Express Vol. 18, Issue 2, pp.
  • the control device 8 includes a computer system, for example.
  • the control device 8 reads a computer program stored in the storage device 43, and executes various kinds of processing in accordance with the computer program.
  • the computer program may be an image processing program that causes a computer to execute image processing by using an imaging result obtained by irradiating a specimen including a fluorescent substance with illumination light, forming, by an observation optical system including an objective lens, an image of light from the specimen, moving at least one of a stage to hold the specimen and the objective lens in the same direction as an optical axis of the objective lens, and taking the image formed by the observation optical system.
  • the image processing may include using position information on the fluorescent substance in at least a part of an overlapping region between a first region and a second region from position information on the fluorescent substance in the first region obtained from an imaging result when a focus position of the objective lens is located at a first position and position information on the fluorescent substance in the second region obtained from an imaging result when the focus position of the objective lens is located at a second position, to correct the position information on the fluorescent substance in the second region.
  • the computer program may be an image processing program that causes a computer to execute image processing by using an imaging result obtained by illuminating a specimen with activation light to activate a part of a fluorescent substance included in the specimen and excitation light to excite at least a part of the activated fluorescent substance, forming, by an observation optical system including an objective lens and an astigmatic optical system that generates astigmatism for at least a part of light from the specimen, an image of light from the specimen, moving at least one of a stage to hold the specimen and the objective lens in the same direction as an optical axis of the objective lens, and taking the image formed by the observation optical system.
  • an observation optical system including an objective lens and an astigmatic optical system that generates astigmatism for at least a part of light from the specimen, an image of light from the specimen, moving at least one of a stage to hold the specimen and the objective lens in the same direction as an optical axis of the objective lens, and taking the image formed by the observation optical system.
  • the image processing may include: calculating position information on the fluorescent substance in a first region by using information on the astigmatism included in an imaging result obtained when a focus position of the objective lens is located at a first position; calculating position information on the fluorescent substance in a second region by using information on the astigmatism included in an imaging result obtained when the focus position of the objective lens is located at a second position; and correcting the position information on the fluorescent substance in the second region by using position information on the fluorescent substance in at least a part of an overlapping region between the first region and the second region from the position information on the fluorescent substance in the first region and the position information on the fluorescent substance in the second region.
  • the computer program may be an image processing program that causes a computer to execute image processing by using an imaging result obtained by irradiating an object placed on a stage with illumination light, forming, by an observation optical system including an objective lens, an image of light from the specimen, moving at least one of the stage and the objective lens in the same direction as an optical axis of the objective lens, and taking the image formed by the observation optical system.
  • the image processing may include using information on the object in at least a part of an overlapping region between a first region and a second region from information on the object in the first region obtained from an imaging result when a focus position of the objective lens is located at a first position and information on the object in the second region obtained from an imaging result when the focus position of the objective lens is located at a second position, to correct the information on the object in the second region.
  • This computer program may be provided by being recorded in a computer-readable storage medium.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Optics & Photonics (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biochemistry (AREA)
  • Immunology (AREA)
  • Pathology (AREA)
  • Signal Processing (AREA)
  • Computer Graphics (AREA)
  • Microscoopes, Condenser (AREA)
  • Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)
  • Studio Devices (AREA)

Claims (9)

  1. Mikroskop (1), aufweisend:
    ein beleuchtungsoptisches System (4), das eine Probe (W), das eine fluoreszente Substanz aufweist, mit Beleuchtungslicht (L) bestrahlt;
    ein beobachtungsoptisches System (5), das eine Objektivlinse (21) aufweist und ein Bild von Licht von der Probe ausbildet;
    einen Imager (6), der das Bild aufnimmt, das durch das beobachtungsoptische System ausgebildet ist;
    einen Bildprozessor (7), der Bildverarbeitung mittels eines Ergebnisses der Bildaufnahme durchführt; und
    eine Steuerung (8), die zumindest eins von einem Tisch (2) zum Halten der Probe und der Objektivlinse in einer gleichen Richtung wie eine optische Achse (5a) der Objektivlinse bewegt,
    wobei der Bildprozessor nutzt
    Positionsinformation über die fluoreszente Substanz in zumindest einem Teil eines überlappenden Bereichs zwischen einem ersten Bereich und einem zweiten Bereich,
    Positionsinformation über die fluoreszente Substanz in dem ersten Bereich, die von einem Abbildungsergebnis erhalten wird, wenn sich eine Fokusposition der Objektivlinse an einer ersten Position befindet, und
    Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich, die von einem Abbildungsergebnis erhalten wird, wenn sich eine Fokusposition der Objektivlinse an einer zweiten Position befindet,
    um die Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich zu korrigieren,
    wobei der Bildprozessor die Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich durch Anwenden eines Positionsänderungsbetrags auf die Positionsinformation der fluoreszenten Substanz in dem zweiten Bereich korrigiert,
    welcher Positionsänderungsbetrag in Übereinstimmung mit einer Entfernung in einer Ebene, die die optische Achse kreuzt, von einer Bezugsposition in dem zweiten Bereich variiert,
    um dadurch eine Positionsänderung der fluoreszenten Substanz in dem überlappenden Bereich zwischen einem Zeitpunkt, wenn sich die Fokusposition der Objektivlinse an der ersten Position befindet, und einem Zeitpunkt, wenn sich die Fokusposition der Objektivlinse an der zweiten Position befindet, zu korrigieren.
  2. Mikroskop nach Anspruch 1, wobei das beobachtungsoptische System ein astigmatisches optisches System aufweist, das zumindest für einen Teil des Lichts von der Probe Astigmatismus erzeugt.
  3. Mikroskop nach Anspruch 1 oder 2, wobei der Bildprozessor den ersten Positionsänderungsbetrag in zumindest einem Teil des überlappenden Bereichs zwischen dem ersten Bereich und dem zweiten Bereich mittels der Positionsinformation über die fluoreszente Substanz in dem ersten Bereich und der Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich berechnet.
  4. Mikroskop nach einem der Ansprüche 1 bis 3, ferner aufweisend eine Anzeige (44), wobei
    der Bildprozessor ein dreidimensionales Bild mittels der Positionsinformation über die fluoreszente Substanz in dem ersten Bereich und der korrigierten Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich erzeugt; und
    der Bildprozessor das erzeugte dreidimensionale Bild auf der Anzeige anzeigt.
  5. Mikroskop nach einem der Ansprüche 1 bis 3, wobei der Bildprozessor ein dreidimensionales Bild mittels der Positionsinformation über die fluoreszente Substanz in dem ersten Bereich in dem überlappenden Bereich und der korrigierten Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich in dem überlappenden Bereich erzeug und das erzeugte dreidimensionale Bild auf der Anzeige anzeigt.
  6. Mikroskop nach Anspruch 5, wobei der Bildprozessor das dreidimensionale Bild erzeugt, in dem die Positionsinformation über die fluoreszente Substanz in dem ersten Bereich und die korrigierte Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich durch Punkte mit unterschiedlichen Farben dargestellt sind.
  7. Mikroskop nach Anspruch 5, wobei
    der Bildprozessor einen zweiten Positionsänderungsbetrag für jeden der Mehrzahl von Bereichen in dem überlappenden Bereich mittels der Positionsinformation über die fluoreszente Substanz in dem ersten Bereich und der korrigierten Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich berechnet und
    wenn der zweite Positionsänderungsbetrag kleiner ist als ein Grenzwert, der Bildprozessor das dreidimensionale Bild erzeugt, in dem die Positionsinformation über die fluoreszente Substanz in dem ersten Bereich und die korrigierte Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich durch Punkte mit einer gleichen Farbe in dem Bereich dargestellt sind, in dem der zweite Positionsänderungsbetrag kleiner ist als der Grenzwert, und wenn der zweite Positionsänderungsbetrag gleich dem oder größer als der Grenzwert ist, der Bildprozessor das dreidimensionale Bild erzeugt, in dem die Positionsinformation über die fluoreszente Substanz in dem ersten Bereich und die korrigierte Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich durch Punkte mit unterschiedlichen Farben in dem Bereich dargestellt sind, in dem der zweite Positionsänderungsbetrag gleich dem oder größer als der Grenzwert ist.
  8. Beobachtungsverfahren, umfassend:
    Bestrahlen einer Probe (W), die eine fluoreszente Substanz aufweist, mit Beleuchtungslicht (L);
    Ausbilden, mittels eines beobachtungsoptischen Systems (5), das eine Objektivlinse (21) aufweist, eines Bilds von Licht von der Probe;
    Aufnehmen des durch das beobachtungsoptische System ausgebildeten Bilds;
    Durchführen von Bildverarbeitung mittels eines Ergebnisses der Bildaufnahme; und
    Bewegen zumindest eines von einem Tisch (2) zum Halten der Probe und der Objektivlinse in einer gleichen Richtung wie eine optische Achse (5a) der Objektivlinse,
    wobei die Bildverarbeitung das Nutzen von
    Positionsinformation über die fluoreszente Substanz in zumindest einem Teil eines überlappenden Bereichs zwischen einem ersten Bereich und einem zweiten Bereich,
    Positionsinformation über die fluoreszente Substanz in dem ersten Bereich, die von einem Abbildungsergebnis erhalten wird, wenn sich eine Fokusposition der Objektivlinse an einer ersten Position befindet, und
    Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich, die von einem Abbildungsergebnis erhalten wird, wenn sich eine Fokusposition der Objektivlinse an einer zweiten Position befindet, umfasst,
    um die Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich zu korrigieren,
    wobei die Bildverarbeitung Korrigieren der Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich durch Anwenden eines Positionsänderungsbetrags auf die Positionsinformation der fluoreszenten Substanz in dem zweiten Bereich umfasst,
    welcher Positionsänderungsbetrag in Übereinstimmung mit einer Entfernung in einer Ebene, die die optische Achse kreuzt, von einer Bezugsposition in dem zweiten Bereich variiert,
    um dadurch eine Positionsänderung der fluoreszenten Substanz in dem überlappenden Bereich zwischen einem Zeitpunkt, wenn sich die Fokusposition der Objektivlinse an der ersten Position befindet, und einem Zeitpunkt, wenn sich die Fokusposition der Objektivlinse an der zweiten Position befindet, zu korrigieren.
  9. Bildverarbeitungsprogramm, das auslöst, das ein Computer Bildverarbeitung mittels eines Abbildungsergebnisses durchführt, das durch Bestrahlen einer Probe, die eine fluoreszente Substanz aufweist, mit Beleuchtungslicht, Ausbilden, durch ein beobachtungsoptisches System, das eine Objektivlinse aufweist, eines Bilds von Licht von der Probe, Bewegen zumindest eines von einem Tisch zum Halten der Probe und der Objektivlinse in einer gleichen Richtung wie eine optische Achse der Objektivlinse, und Aufnehmen des durch das beobachtungsoptische System ausgebildeten Bilds,
    wobei die Bildverarbeitung das Nutzen von Positionsinformation über die fluoreszente Substanz in zumindest einem Teil eines überlappenden Bereichs zwischen einem ersten Bereich und einem zweiten Bereich,
    Positionsinformation über die fluoreszente Substanz in dem ersten Bereich, die von einem Abbildungsergebnis erhalten wird, wenn sich eine Fokusposition der Objektivlinse an einer ersten Position befindet, und
    Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich, die von einem Abbildungsergebnis erhalten wird, wenn sich eine Fokusposition der Objektivlinse an einer zweiten Position befindet, umfasst,
    um die Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich zu korrigieren,
    wobei die Bildverarbeitung Korrigieren der
    Positionsinformation über die fluoreszente Substanz in dem zweiten Bereich durch Anwenden eines Positionsänderungsbetrags auf die Positionsinformation der fluoreszenten Substanz in dem zweiten Bereich umfasst,
    welcher Positionsänderungsbetrag in Übereinstimmung mit einer Entfernung in einer Ebene, die die optische Achse kreuzt, von einer Bezugsposition in dem zweiten Bereich variiert,
    um dadurch eine Positionsänderung der fluoreszenten Substanz in dem überlappenden Bereich zwischen einem Zeitpunkt, wenn sich die Fokusposition der Objektivlinse an der ersten Position befindet, und einem Zeitpunkt, wenn sich die Fokusposition der Objektivlinse an der zweiten Position befindet, zu korrigieren.
EP15909327.7A 2015-11-27 2015-11-27 Mikroskop, beobachtungsverfahren und bildverarbeitungsprogramm Active EP3382439B1 (de)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2015/083510 WO2017090210A1 (ja) 2015-11-27 2015-11-27 顕微鏡、観察方法、及び画像処理プログラム

Publications (3)

Publication Number Publication Date
EP3382439A1 EP3382439A1 (de) 2018-10-03
EP3382439A4 EP3382439A4 (de) 2019-12-11
EP3382439B1 true EP3382439B1 (de) 2025-12-10

Family

ID=58764221

Family Applications (1)

Application Number Title Priority Date Filing Date
EP15909327.7A Active EP3382439B1 (de) 2015-11-27 2015-11-27 Mikroskop, beobachtungsverfahren und bildverarbeitungsprogramm

Country Status (4)

Country Link
US (1) US10725278B2 (de)
EP (1) EP3382439B1 (de)
JP (1) JP6635125B2 (de)
WO (1) WO2017090210A1 (de)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017090211A1 (ja) 2015-11-27 2017-06-01 株式会社ニコン 顕微鏡装置
RU2670779C9 (ru) * 2018-02-28 2018-11-23 Общество с ограниченной ответственностью научно-исследовательское предприятие "ПАНОРАМА" Способ получения панорамного изображения гистологических, цитологических и иммуноцитологических препаратов (варианты)
WO2019229913A1 (ja) * 2018-05-30 2019-12-05 株式会社ニコン 情報処理装置、情報処理方法、情報処理プログラム、及び顕微鏡
EP3975120B1 (de) 2020-09-24 2023-01-11 Stryker European Operations Limited Verfahren zur führung der erfassung eines oder mehreren registrierungspunkten am körper eines patienten

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2013015665A (ja) * 2011-07-04 2013-01-24 Nikon Corp 顕微鏡装置及び画像形成方法

Family Cites Families (51)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5748491A (en) * 1995-12-20 1998-05-05 The Perkin-Elmer Corporation Deconvolution method for the analysis of data resulting from analytical separation processes
US20060060781A1 (en) * 1997-08-11 2006-03-23 Masahiro Watanabe Charged-particle beam apparatus and method for automatically correcting astigmatism and for height detection
US6259080B1 (en) * 1998-03-18 2001-07-10 Olympus Optical Co. Ltd. Autofocus device for microscope
US6670602B1 (en) * 1998-06-03 2003-12-30 Nikon Corporation Scanning device and scanning method
JP2000098244A (ja) * 1998-09-24 2000-04-07 Olympus Optical Co Ltd 蛍光顕微鏡
JP2001093455A (ja) * 1999-09-21 2001-04-06 Nikon Corp 電子ビーム装置
JP4608043B2 (ja) * 1999-09-24 2011-01-05 オリンパス株式会社 顕微鏡用焦点検出装置
US6577453B2 (en) * 2000-04-26 2003-06-10 Fuji Photo Film Co., Ltd. Image reading apparatus
US7668697B2 (en) * 2006-02-06 2010-02-23 Andrei Volkov Method for analyzing dynamic detectable events at the single molecule level
US6750423B2 (en) * 2001-10-25 2004-06-15 Semiconductor Energy Laboratory Co., Ltd. Laser irradiation method, laser irradiation apparatus, and method of manufacturing a semiconductor device
GB2383487B (en) * 2001-12-18 2006-09-27 Fairfield Imaging Ltd Method and apparatus for acquiring digital microscope images
GB2385481B (en) * 2002-02-13 2004-01-07 Fairfield Imaging Ltd Microscopy imaging system and method
US20030184856A1 (en) * 2002-04-02 2003-10-02 Nikon Corporation Focus point detection device and microscope using the same
JP2005134867A (ja) * 2003-10-08 2005-05-26 Nikon Corp 画像表示装置
DE102004034988A1 (de) * 2004-07-16 2006-02-02 Carl Zeiss Jena Gmbh Lichtrastermikroskop und Verwendung
JP4599941B2 (ja) * 2004-08-20 2010-12-15 株式会社ニコン 自動焦点検出装置およびこれを備える顕微鏡システム
EP2453237B1 (de) * 2005-05-23 2016-12-21 Harald F. Hess Optische Mikroskopie mit phototransformierbaren optischen Markern
WO2007037439A1 (ja) * 2005-09-29 2007-04-05 Olympus Corporation 焦点位置決定方法、焦点位置決定装置、微弱光検出装置及び微弱光検出方法
GB2449568B (en) * 2006-01-12 2011-06-29 Univ Gunma Nat Univ Corp Device for determining aim position of charged particle beam, method of using the device, and treatment device employing device for determining aim position
US7776613B2 (en) * 2006-08-07 2010-08-17 President And Fellows Of Harvard College Sub-diffraction image resolution and other imaging techniques
US7838302B2 (en) 2006-08-07 2010-11-23 President And Fellows Of Harvard College Sub-diffraction limit image resolution and other imaging techniques
WO2008069220A1 (ja) * 2006-11-30 2008-06-12 Nikon Corporation 結像装置及び顕微鏡
US7916304B2 (en) * 2006-12-21 2011-03-29 Howard Hughes Medical Institute Systems and methods for 3-dimensional interferometric microscopy
US7687769B2 (en) * 2007-01-19 2010-03-30 Schlumberger Technology Corporation Methods and apparatus for multi dimension fluorescence spectrum measurement and correlations downhole
JP5463671B2 (ja) * 2007-01-19 2014-04-09 株式会社ニコン 焦点検出装置、顕微鏡
JP5448353B2 (ja) * 2007-05-02 2014-03-19 キヤノン株式会社 光干渉断層計を用いた画像形成方法、及び光干渉断層装置
EP2201352B2 (de) * 2007-09-28 2018-08-29 Illumina, Inc. Fluoreszenzanregungs- und -nachweissystem und entsprechendes verfahren
US8174692B2 (en) * 2008-05-21 2012-05-08 Max-Planck-Gesellschaft Zur Foerderung Der Wissenschaften E.V. High spatial resolution imaging of a structure of interest in a specimen
US20090289200A1 (en) * 2008-05-22 2009-11-26 Fujifilm Corporation Fluorescent image obtainment method and apparatus, fluorescence endoscope, and excitation-light unit
JP5250342B2 (ja) * 2008-08-26 2013-07-31 富士フイルム株式会社 画像処理装置およびプログラム
DE102008059328A1 (de) * 2008-11-27 2010-06-02 Carl Zeiss Microimaging Gmbh Auflösungsgesteigerte Mikroskopie
GB201007055D0 (en) * 2010-04-28 2010-06-09 Vib Vzw Method and apparatus for the imaging of a labelled sample
DE102010036709A1 (de) * 2010-07-28 2012-02-02 Leica Microsystems Cms Gmbh Einrichtung und Verfahren zur mikroskopischen Bildaufnahme einer Probenstruktur
JP2013539074A (ja) * 2010-09-24 2013-10-17 カール・ツァイス・マイクロスコピー・ゲゼルシャフト・ミット・ベシュレンクテル・ハフツング ３ｄ局在顕微鏡法並びに４ｄ局在顕微鏡法及び追跡方法並びに追跡システム
JP5751986B2 (ja) * 2010-12-08 2015-07-22 キヤノン株式会社 画像生成装置
DE102011055294B4 (de) * 2011-11-11 2013-11-07 Leica Microsystems Cms Gmbh Mikroskopische Einrichtung und Verfahren zur dreidimensionalen Lokalisierung von punktförmigen Objekten in einer Probe
DE102012201003B4 (de) * 2012-01-24 2024-07-25 Carl Zeiss Microscopy Gmbh Mikroskop und Verfahren für die hochauflösende 3-D Fluoreszenzmikroskopie
US9395303B2 (en) * 2012-03-12 2016-07-19 Mitsubishi Rayon Co., Ltd. Fluorescence detection device and fluorescence detection method
WO2013183345A1 (ja) * 2012-06-06 2013-12-12 ソニー株式会社 微小粒子測定装置におけるデータ補正方法及び微小粒子測定装置
JP2014098835A (ja) * 2012-11-15 2014-05-29 Canon Inc 顕微鏡用照明光学系およびこれを用いた顕微鏡
DE102012224306A1 (de) * 2012-12-21 2014-06-26 Carl Zeiss Microscopy Gmbh Verfahren zur hochauflösenden 3D-Lokalisierungsmikroskopie
KR20140091177A (ko) * 2013-01-10 2014-07-21 삼성전자주식회사 병변 진단 장치 및 방법
US9435993B2 (en) * 2013-03-24 2016-09-06 Bruker Nano, Inc. Three dimensional microscopy imaging
DE102013208415B4 (de) * 2013-05-07 2023-12-28 Carl Zeiss Microscopy Gmbh Mikroskop und Verfahren für die 3D-hochauflösende Lokalisierungsmikroskopie
DE102013208927A1 (de) * 2013-05-14 2014-11-20 Carl Zeiss Microscopy Gmbh Verfahren zur 3D-hochauflösenden Lokalisierungsmikroskopie
DE102013208926A1 (de) * 2013-05-14 2014-11-20 Carl Zeiss Microscopy Gmbh Verfahren zur 3D-hochauflösenden Lokalisierungsmikroskopie
DE102013106895B4 (de) * 2013-07-01 2015-09-17 Leica Microsystems Cms Gmbh Lichtmikroskopisches Verfahren zur Lokalisierung von Punktobjekten
DE102013019348A1 (de) * 2013-08-15 2015-02-19 Carl Zeiss Microscopy Gmbh Hochauflösende Scanning-Mikroskopie
DE102013015932A1 (de) * 2013-09-19 2015-03-19 Carl Zeiss Microscopy Gmbh Hochauflösende Scanning-Mikroskopie
DE102013015933A1 (de) * 2013-09-19 2015-03-19 Carl Zeiss Microscopy Gmbh Hochauflösende Scanning-Mikroskopie
DE102013015931B4 (de) * 2013-09-19 2024-05-08 Carl Zeiss Microscopy Gmbh Mikroskop und Verfahren zur hochauflösenden Scanning-Mikroskope

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2013015665A (ja) * 2011-07-04 2013-01-24 Nikon Corp 顕微鏡装置及び画像形成方法

Also Published As

Publication number Publication date
JPWO2017090210A1 (ja) 2018-09-13
US10725278B2 (en) 2020-07-28
US20180299659A1 (en) 2018-10-18
JP6635125B2 (ja) 2020-01-22
EP3382439A4 (de) 2019-12-11
EP3382439A1 (de) 2018-10-03
WO2017090210A1 (ja) 2017-06-01

Similar Documents

Publication Publication Date Title
Krieger et al. Imaging fluorescence (cross-) correlation spectroscopy in live cells and organisms
US8704196B2 (en) Combination microscopy
JP6310671B2 (ja) レーザー顕微解剖領域の画定方法及び関連するレーザー顕微解剖システム
US10725278B2 (en) Microscope, observation method, and storage medium
EP3040756B1 (de) Ansteuerungsverfahren für eine objektivlinse und fluoreszenzmikroskopsystem
JP2018517178A (ja) ライトシート顕微鏡法のための構成及び方法
JP2009065848A (ja) 遺伝子発現解析方法および遺伝子発現解析システム
Shaw Imaging the live plant cell
CN104122662A (zh) 一种超高密度超分辨光学闪烁显微成像系统及方法
US20220057329A1 (en) Method and apparatus for detecting fluorescence signals in a three-dimensional region of a sample
US20120140057A1 (en) Microscope for Measuring Total Reflection Fluorescence
US20180017773A1 (en) Microscope device, observation method, and storage medium
JP6822483B2 (ja) 球面収差補正算出装置、顕微鏡、球面収差補正算出プログラム及び球面収差補正算出方法
EP3249441B1 (de) Bildaufnahmevorrichtung und bildaufnahmeverfahren
US10823675B2 (en) Microscope, observation method, and a storage medium
EP3276335B1 (de) Mikroskopvorrichtung, betrachtungsverfahren und steuerungsprogramm
KR101703543B1 (ko) 디지털 홀로그래피 방법을 결합한 다중모드 sted 현미경시스템
JP6617774B2 (ja) 顕微鏡装置
EP3249440B1 (de) Bildaufnahmevorrichtung und bildaufnahmeverfahren
JP2019049756A (ja) 顕微鏡装置、観察方法、及び制御プログラム
US8294728B2 (en) Process for generating display images from acquired recorded images, and means for carrying out the process
JP2019045737A (ja) 観察装置
Kim et al. Parallel frequency-multiplexed aberration measurement for widefield fluorescence microscopy
Elson et al. Single-shot fluorescence lifetime imaging using a novel segmented gated optical intensifier

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20180614

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
RIC1 Information provided on ipc code assigned before grant

Ipc: G01N 21/64 20060101ALI20190628BHEP

Ipc: G02B 21/00 20060101AFI20190628BHEP

Ipc: H04N 5/232 20060101ALI20190628BHEP

A4 Supplementary search report drawn up and despatched

Effective date: 20191111

RIC1 Information provided on ipc code assigned before grant

Ipc: G02B 21/36 20060101ALI20191105BHEP

Ipc: G02B 21/00 20060101AFI20191105BHEP

Ipc: G01N 21/64 20060101ALI20191105BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20210209

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230517

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: G02B0021000000

Ipc: H04N0023630000

Ref country code: DE

Ref legal event code: R079

Ref document number: 602015092779

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G02B0021000000

Ipc: H04N0023630000

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 23/63 20230101AFI20250613BHEP

Ipc: G02B 21/36 20060101ALI20250613BHEP

Ipc: G01N 21/64 20060101ALI20250613BHEP

Ipc: G02B 21/00 20060101ALI20250613BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20250813

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

RAP3 Party data changed (applicant data changed or rights of an application transferred)

Owner name: NIKON CORPORATION

RIN1 Information on inventor provided before grant (corrected)

Inventor name: SASE, ICHIRO

Inventor name: KANEKO, YASUTOSHI

Inventor name: FUKUI, TATSUO

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: CH

Ref legal event code: F10

Free format text: ST27 STATUS EVENT CODE: U-0-0-F10-F00 (AS PROVIDED BY THE NATIONAL OFFICE)

Effective date: 20251210

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602015092779

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D