WO2018131172A1 - 画像処理装置、顕微鏡システム、画像処理方法、及びプログラム - Google Patents

画像処理装置、顕微鏡システム、画像処理方法、及びプログラム Download PDF

Info

Publication number
WO2018131172A1
WO2018131172A1 PCT/JP2017/001289 JP2017001289W WO2018131172A1 WO 2018131172 A1 WO2018131172 A1 WO 2018131172A1 JP 2017001289 W JP2017001289 W JP 2017001289W WO 2018131172 A1 WO2018131172 A1 WO 2018131172A1
Authority
WO
WIPO (PCT)
Prior art keywords
microscope
image data
image
sample
imaging
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2017/001289
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
一郎 佐瀬
佐々木　豊
岡本　高明
勇輝 照井
功記 小西
三村　正文
マーティン ベルガー
ペトル ガザック
ミロスラフ スヴォボダ
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nikon Corp
Original Assignee
Nikon Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nikon Corp filed Critical Nikon Corp
Priority to PCT/JP2017/001289 priority Critical patent/WO2018131172A1/ja
Priority to JP2018561782A priority patent/JP6911873B2/ja
Priority to US16/478,434 priority patent/US11442262B2/en
Priority to EP17891716.7A priority patent/EP3570087A4/en
Publication of WO2018131172A1 publication Critical patent/WO2018131172A1/ja
Anticipated expiration legal-status Critical
Priority to JP2021113684A priority patent/JP2021184264A/ja
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365—Control or image processing arrangements for digital or video microscopes
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365—Control or image processing arrangements for digital or video microscopes
    • G02B21/367—Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/37—Determination of transform parameters for the alignment of images, i.e. image registration using transform domain methods
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01J—ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS
    • H01J37/00—Discharge tubes with provision for introducing objects or material to be exposed to the discharge, e.g. for the purpose of examination or processing thereof
    • H01J37/02—Details
    • H01J37/22—Optical, image processing or photographic arrangements associated with the tube
    • H01J37/222—Image processing arrangements associated with the tube
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01J—ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS
    • H01J37/00—Discharge tubes with provision for introducing objects or material to be exposed to the discharge, e.g. for the purpose of examination or processing thereof
    • H01J37/26—Electron or ion microscopes; Electron or ion diffraction tubes
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01J—ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS
    • H01J37/00—Discharge tubes with provision for introducing objects or material to be exposed to the discharge, e.g. for the purpose of examination or processing thereof
    • H01J37/26—Electron or ion microscopes; Electron or ion diffraction tubes
    • H01J37/28—Electron or ion microscopes; Electron or ion diffraction tubes with scanning beams
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G01N21/645—Specially adapted constructive features of fluorimeters
    • G01N21/6456—Spatial resolved fluorescence measurements; Imaging
    • G01N21/6458—Fluorescence microscopy
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/0004—Microscopes specially adapted for specific applications
    • G02B21/002—Scanning microscopes
    • G02B21/0024—Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/0004—Microscopes specially adapted for specific applications
    • G02B21/002—Scanning microscopes
    • G02B21/0024—Confocal scanning microscopes (CSOMs) or confocal "macroscopes"; Accessories which are not restricted to use with CSOMs, e.g. sample holders
    • G02B21/0052—Optical details of the image generation
    • G02B21/0076—Optical details of the image generation arrangements using fluorescence or luminescence
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/58—Optics for apodization or superresolution; Optical synthetic aperture systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10056—Microscopic image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20048—Transform domain processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01J—ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS
    • H01J2237/00—Discharge tubes exposing object to beam, e.g. for analysis treatment, etching, imaging
    • H01J2237/22—Treatment of data
    • H01J2237/226—Image reconstruction

Definitions

  • the present invention relates to an image processing apparatus, a microscope system, an image processing method, and a program.
  • Patent Document 1 Japanese Translation of PCT International Publication No. 2012-507756
  • the corresponding image data corresponding to the first microscope image data obtained under the first observation condition is used as the second microscope image data and the third microscope image data obtained under the second observation condition.
  • An image processing apparatus includes an image generation unit that generates data based on the image data and an image output unit that outputs corresponding image data.
  • a microscope system including a first microscope, a second microscope, and the image processing device is provided.
  • the corresponding image data corresponding to the first microscope image data obtained under the first observation condition is used as the second microscope image data and the third microscope image data obtained under the second observation condition.
  • An image processing method is provided that includes generating based on and outputting corresponding image data.
  • the corresponding image data corresponding to the first microscope image data obtained under the first observation condition is used as the second microscope image data and the third microscope image data obtained under the second observation condition.
  • the structure of the main-body part contained in the microscope system which concerns on 1st Embodiment is shown.
  • 1 shows a functional configuration of a microscope system according to a first embodiment.
  • 2 shows a flow of microscope observation and image processing according to the first embodiment.
  • An example of the correspondence of the captured image based on 1st and 2nd microscope image data is shown.
  • the principle of image data generation using Fourier transform will be described.
  • An example of the display screen which displayed Z stack image is shown.
  • the estimation of the shift amount of image data is shown.
  • the principle of generating image data using the weighted average is shown.
  • the structure of the main-body part contained in the microscope system which concerns on 2nd Embodiment is shown.
  • the flow of microscope observation and image processing concerning a 2nd embodiment is shown.
  • the imaging state by the 1st microscope apparatus in the microscope system concerning a 2nd embodiment is shown.
  • the imaging state by the 2nd microscope apparatus in the microscope system concerning a 2nd embodiment is shown.
  • An example of the correspondence of the captured image based on 1st and 2nd microscope image data is shown.
  • generated by the convolution calculation of 2nd microscope image data is shown.
  • the flow of microscope observation and image processing concerning a 3rd embodiment is shown.
  • An example of the correspondence of the captured image based on 1st and 2nd microscope image data is shown.
  • the flow of microscope observation and image processing concerning a 4th embodiment is shown.
  • the imaging state by the 1st microscope apparatus in time lapse imaging is shown.
  • the imaging state by the 2nd microscope apparatus in time-lapse imaging is shown.
  • Timing chart An example of a timing chart in which a sample is imaged by the first and second microscope apparatuses is shown.
  • the principle of generating image data by temporal interpolation using image recognition will be described.
  • a principle of generating image data by time interpolation using luminance value calculation will be described.
  • Another example of a timing chart in which a sample is imaged by the first and second microscope apparatuses is shown.
  • 2 shows an exemplary configuration of a computer according to the present embodiment.
  • FIGS. 1 and 2 show the configuration of the main body 99 and the functional configuration of the microscope system 100 included in the microscope system 100 according to the first embodiment, respectively.
  • the microscope system 100 is a system that images a sample by using different types of microscopy using a plurality of types of microscope apparatuses, and includes a main body 99, a control unit 50, and an image processing unit 60.
  • a Z axis is defined in parallel with an optical axis L of an optical system 20 described later in FIG.
  • the main body 99 includes a first microscope apparatus 30 that observes the sample 9 under the first observation condition, and a second microscope apparatus 40 that observes the sample 9 under the second observation condition.
  • the first and second observation conditions include, for example, microscopy, illumination conditions, imaging conditions, and the like.
  • the first microscopy that is the microscopy of the first microscope device 30 is a confocal microscope
  • the second microscopy that is the microscopy of the second microscope device 40 is SIM (structured illumination microscope). Law).
  • the illumination conditions include the brightness of the illumination light that illuminates the sample 9, the wavelength of the illumination light, whether or not polarized light is used for the illumination light, and the direction of polarized light, the size of the diaphragm, etc. when used with polarized light.
  • the imaging conditions include the numerical aperture of the objective lens 21a, a range in the XY plane for imaging the sample 9 (referred to as an XY scanning range), a position on the Z axis (referred to as a Z position), and the like.
  • the first microscope apparatus 30 and the second microscope apparatus 40 share the stage system 10.
  • the stage system 10 is a system that supports and moves the sample 9, and includes a stage 11, a sensor 12, and a driving device 13.
  • the sample 9 is, for example, a cell into which a fluorescent dye is introduced, and is used while being held on a holding member 8 such as a translucent glass plate.
  • two colors of fluorescent dyes that is, a fluorescent dye used in the first microscope apparatus 30 and a fluorescent dye used in the second microscope apparatus 40 are introduced.
  • Stage 11 is a device that supports the sample 9.
  • the stage 11 has an opening 11 a through which the optical axis L of the optical system 20 passes.
  • the sample 9 is positioned on the opening 11 a by supporting the holding member 8 on the stage 11.
  • the stage 11 is configured to be movable along the optical axis L of the optical system 20.
  • the sensor 12 measures the position or displacement of the stage 11 in the Z-axis direction.
  • a linear encoder can be employed as the sensor 12 for example.
  • the measurement result is transmitted to the control unit 50.
  • the driving device 13 drives the stage 11 in the Z-axis direction.
  • the driving device 13 for example, a plurality of motors or the like can be employed.
  • the driving device 13 is controlled by the control unit 50 to drive the stage 11 to the target position. By this movement, the sample 9 on the stage 11 moves with respect to the optical axis L of the optical system 20.
  • the first microscope device 30 and the second microscope device 40 share the optical system 20.
  • the optical system 20 is a system that irradiates the sample 9 with illumination light and collects light emitted from the sample 9, and includes an objective lens 21 a, a scanner 23, and a filter 24.
  • the objective lens 21a is an optical element that connects an intermediate image of the sample 9 on the stage 11, and is disposed immediately below the stage 11 as an example in the present embodiment.
  • a focal plane 20a parallel to the XY plane including the focal point of the optical system 20 including the objective lens 21a is also shown.
  • the scanner 23 is a mechanism that swings illumination light in a plane orthogonal to the optical axis L of the optical system 20, that is, in the XY directions in the figure, and a pair of galvanometer mirrors can be used as an example.
  • One galvanometer mirror rotates about the X axis
  • the other galvanometer mirror rotates about the Y axis.
  • Illumination light incident on the scanner 23 is reflected by the pair of galvanometer mirrors and thus is swung in the X-axis and Y-axis directions with respect to the optical axis L, and the sample 9 is scanned in the XY directions.
  • the scanner 23 can be advanced or retracted from the optical axis L by a driving device (not shown), and is retracted from the optical axis L when the sample 9 is observed by the SIM.
  • the filter 24 is disposed on the optical axis L of the optical system 20 to reflect light having specific wavelengths ( ⁇ 1 , ⁇ 1 ′) and to transmit light having other wavelengths ( ⁇ 2 , ⁇ 2 ′). It is an optical element that transmits. As the filter 24, for example, a dichroic mirror can be used.
  • the filter 24 reflects the illumination light (wavelength ⁇ 1 ) emitted from the first illumination / imaging unit 31, sends it to the sample 9 via the scanner 23 and the objective lens 21 a, and returns light (wavelength ⁇ 1 ′) from the sample 9.
  • the first microscope apparatus 30 includes a first illumination / imaging unit 31, an optical system 20, and a stage system 10, and images the sample 9 by the first microscope method.
  • confocal microscopy is adopted as the first microscopy as described above.
  • the specimen 9 is scanned in the XY direction on the focal plane 20a by illumination light from a laser light source (not shown) of the first illumination / imaging unit 31, and a cross section of the specimen located on the focal plane 20a is scanned in the first direction.
  • a two-dimensional image is generated by imaging with an imaging element (not shown) of one illumination / imaging unit 31.
  • the sample 9 is displaced relatively in the Z-axis direction with respect to the focal plane 20a, and a different surface in the sample 9 is imaged. More specifically, the sample 9 is imaged by scanning the sample 9 at a plurality of Z positions to generate a Z stack image.
  • the first microscope apparatus 30 can observe the sample 9 at a high speed but with a relatively low resolution.
  • the first illumination / imaging unit 31 illuminates the sample 9 and detects light from the sample 9.
  • the first illumination / imaging unit 31 generates illumination light, for example, illumination light having a wavelength ⁇ 1 (for example, 488 nm) according to illumination conditions relating to luminance, wavelength, polarization, diaphragm, and the like, and passes through a filter such as a dichroic mirror (not shown). Then, the light is emitted toward the optical system 20 (filter 24).
  • the illumination light illuminates the observation position 9a in the sample 9 located on the focal plane 20a via the optical system 20.
  • the first illumination / imaging unit 31 captures an image of the sample 9 by separating and detecting light incident from the sample 9 from light that becomes noise such as illumination light by the filter (not shown). Image data obtained by imaging is transmitted to the image processing unit 60 as microscope image data of the first microscope apparatus 30.
  • the second microscope apparatus 40 includes a second illumination / imaging unit 41, the optical system 20, and the stage system 10, and images the sample 9 by the second microscope method.
  • SIM is employed as the second microscope method as described above.
  • structured illumination that is, patterned illumination
  • the moire to be captured is imaged by an image sensor (not shown) of the second illumination / imaging unit 41.
  • An image having a resolution exceeding the diffraction limit is generated by rearranging the low-frequency component and the high-frequency component in the Fourier-transformed frequency space and then performing inverse Fourier transform.
  • the sample 9 is displaced relatively in the Z-axis direction with respect to the focal plane 20a, and a different surface in the sample 9 is imaged. More specifically, the sample 9 is imaged by scanning the sample 9 at a plurality of Z positions to generate a Z stack image.
  • the second microscope apparatus 40 can observe the sample 9 with high resolution but at a relatively low speed.
  • the second illumination / imaging unit 41 illuminates the sample 9 and detects light from the sample 9.
  • the second illumination / imaging unit 41 generates illumination light, for example, illumination light having a wavelength of ⁇ 2 (for example, 561 nm) according to the illumination conditions, and transmits the illumination light to the optical system 20 (filter 24) via a filter such as a dichroic mirror (not shown). Inject towards.
  • the illumination light is sent to the sample 9 through the optical system 20, and the observation position 9a in the sample 9 located on the focal plane 20a is illuminated. Accordingly, for example, light such as fluorescence having a wavelength ⁇ 2 ′ (for example, 600 nm) emitted from the sample 9 returns to the second illumination / imaging unit 41 via the optical system 20.
  • the second illumination / imaging unit 41 images the sample 9 by separating and detecting light incident from the sample 9 from light that becomes noise such as illumination light by the filter (not shown). Image data obtained by imaging is transmitted to the image processing unit 60 as microscope image data of the second microscope apparatus 40.
  • the control unit 50 has an input unit 51, and in response to an instruction input to the input unit 51, each component of the main body 99, that is, the stage system 10, the optical system 20, the first microscope device 30, and the second microscope device. 40 is controlled.
  • the control unit 50 expresses each functional unit by causing an information processing device including a computer, a microcontroller, etc. to execute a control program stored in a storage device such as a nonvolatile memory or a recording medium such as a CD-ROM. And function as a control device.
  • control unit 50 controls the stage system 10 in response to an instruction to change the Z position of the sample 9 to be imaged, and the stage 11 that supports the sample 9 is a direction parallel to the optical axis L (also referred to as an optical axis direction). ) To move the desired Z position of the sample 9 onto the focal plane 20a. Further, the control unit 50 controls the first and second illumination / imaging units 31 and 41 to image the sample 9.
  • the input unit 51 includes a mouse, a keyboard, a touch panel, and the like.
  • the user instructs the control unit 50 to perform observation with a microscope (also simply referred to as microscope observation), that is, imaging of the sample 9 with the first and second microscope apparatuses 30 and 40 via the input unit 51. Can do.
  • the user can instruct the generation of new image data obtained by processing the microscope image data obtained by setting the observation conditions and microscope observation via the input unit 51.
  • the image processing unit 60 processes the microscope image data obtained by the first and second microscope apparatuses 30 and 40, or generates new image data.
  • the image processing unit 60 includes an image generation unit 61 and an image output unit 62.
  • the image processing unit 60 causes each functional unit to execute an image processing program stored in a storage device such as a nonvolatile memory or a recording medium such as a CD-ROM on an information processing device including a computer and a microcontroller. It expresses and functions as an image processing device.
  • microscope image data is hereinafter referred to as microscope image data in embodiments described later. Since the information to be shown is an image, the same reference number may be used and the image may be illustrated as an image.
  • the microscope image data of the first and second microscope apparatuses 30 and 40 is data corresponding to one microscope image.
  • the image generation unit 61 generates new image data based on the microscope image data obtained by the first and second microscope apparatuses 30 and 40.
  • the image generation unit 61 can obtain the corresponding image data corresponding to the first microscope image data obtained by the first microscope apparatus 30 under the first observation condition by the second microscope apparatus 40 under the second observation condition.
  • the second microscopic image data and the third microscopic image data are generated.
  • the image generation unit 61 stores the first, second, and third microscope image data received from the first and second microscope apparatuses 30 and 40 and the generated corresponding image data. The generation of the corresponding image data by the image generation unit 61 will be described later.
  • Each of the first, second and third microscope image data is data corresponding to one microscope image.
  • the image output unit 62 processes the first, second and third microscope image data obtained by the first and second microscope apparatuses 30 and 40 and / or the corresponding image data generated by the image generation unit 61, Output to the display unit 63. Accordingly, at least a microscope image based on the first microscope image data obtained under the first observation condition by the first microscope apparatus 30 and the second microscope apparatus 40 obtained under the second observation condition corresponding to the microscope image. The corresponding image based on the corresponding image data generated based on the second and third microscope image data is displayed on the display unit 63.
  • the display unit 63 includes display devices such as a CRT, a liquid crystal display, a plasma display, an organic EL display, and a projector.
  • the display unit 63 is a microscope image based on the image data processed by the image output unit 62, that is, the first, second, and third microscope image data obtained by the first and second microscope apparatuses 30 and 40. And at least one of the corresponding images based on the corresponding image data generated by the image generation unit 61 is displayed on the screen.
  • the display unit 63 converts the corresponding image based on the corresponding image data generated by the image generating unit 61 into the first, second, and third microscope image data obtained by the first and second microscope apparatuses 30 and 40. It is displayed separately from the captured image based on it.
  • the image output unit 62 distinguishes from the captured image by, for example, thickening, blinking, or changing the coloring of the outer frame surrounding the image. Thereby, the generated image and the captured image can be distinguished.
  • FIG. 3 shows a flow of microscope observation and image processing according to the first embodiment.
  • a Z stack image of the sample 9 is captured.
  • the first and second observation conditions are set by the user.
  • the first microscope method and the second microscope method are already set out of the first and second observation conditions.
  • use of illumination light having a wavelength of 488 nm is set as an example of the illumination condition of the first observation condition.
  • the objective lens 21a having a numerical aperture of 1.49 is used, the range in the XY plane to be imaged (that is, the XY scanning range), and the Z position to be imaged.
  • the Z position at which the sample 9 is imaged is set by the reference position Z 1 on the Z axis in the imaging of the sample 9, the step amount ⁇ Z 1 in the Z direction, and the number N 1 of images to be captured.
  • use of illumination light having a wavelength of 561 nm is set as an example of illumination conditions of the second observation condition.
  • the objective lens 21a having the same numerical aperture 1.49 as that of the first observation condition is used, the range in the XY plane to be imaged, and the reference position Z 2 on the Z axis in the imaging.
  • the step amount ⁇ Z 2 in the Z direction and the number N 2 of images to be captured are set.
  • step 120 the sample 9 is imaged in parallel by the first and second microscope apparatuses 30 and 40.
  • the control unit 50 controls the stage system 10, the optical system 20, and the first and second microscope apparatuses 30 and 40 according to the first and second observation conditions for the first and second microscope apparatuses 30 and 40, respectively. .
  • the control unit 50 sets illumination conditions for the first and second illumination / imaging units 31 and 41.
  • Control unit 50 drives the stage 11 so that the focal plane 20a of the optical system 20 in the Z-position Z 1 of the sample 9 is positioned to image a sample 9 by the first and second microscope apparatus 30 and 40.
  • the first microscope apparatus 30 emits illumination light having a wavelength ⁇ 1 from the first illumination / imaging unit 31 and illuminates the sample 9 on the stage 11 via the optical system 20.
  • the fluorescent dye contained in the sample 9 emits fluorescence of wavelength ⁇ 1 ′.
  • the fluorescent light is collected through the optical system 20 and captured by the first illumination / imaging unit 31.
  • the control unit 50 controls the scanner 23 and swings the illumination light in the XY direction, so that it is within the XY scanning range on the focal plane 20a.
  • the sample 9 is scanned.
  • the first microscope 30, the Z position Z 1 of the specimen 9 is captured, the image data of confocal microscopy is generated.
  • Image data of the Z position Z 1 is sent to the image processing unit 60, it is stored in the image generating unit 61 as a first microscope image data.
  • the second microscope apparatus 40 emits structured illumination having a wavelength ⁇ 2 from the second illumination / imaging unit 41 and illuminates the sample 9 on the stage 11 through the optical system 20.
  • the observation position 9a of the sample 9 located on the focal point is illuminated, and another fluorescent dye contained in the sample 9 emits fluorescence of wavelength ⁇ 2 ′.
  • the fluorescent light is collected through the optical system 20 and captured by the second illumination / imaging unit 41.
  • the microscope method of the second microscope apparatus 40 is SIM
  • the sample 9 is imaged by changing the direction of the structured illumination pattern, and a SIM image at the Z position Z 1 of the sample 9 is generated.
  • the SIM image data at the Z position Z 1 is stored in the image generation unit 61 as the microscope image data of the second microscope apparatus 40.
  • the scanner 23 is retracted from the optical axis L.
  • Control unit 50 the imaging of the sample 9 in the Z-position Z 1 is completed, by driving the step the stage 11 by [Delta] Z 1, only by the first microscope 30, the cross-section of the sample 9 located on the focal plane 20a Take an image.
  • the sample 9 is imaged by the first and second microscope devices 30 and 40 placed on the surface 20a.
  • the control unit 50 sequentially drives the stage 11 by a step amount ⁇ Z 1 in the Z-axis direction, and the sample is formed on the focal plane 20 a by both the first and second microscope apparatuses 30 and 40 or only by the first microscope apparatus 30. 9 is scanned, and a plurality of Z positions in the sample 9 are imaged.
  • the first microscope apparatus 30 generates a plurality of confocal microscope images from a series of imaging results, and sends them to the image processing unit 60 as microscope image data of the first microscope apparatus 30.
  • the second microscope apparatus 40 generates a plurality of SIM images from a series of imaging results, and sends them to the image processing unit 60 as a plurality of microscope image data of the second microscope apparatus 40.
  • a plurality of microscopic images generated from a series of imaging results are collectively referred to as a Z stack image.
  • FIG. 4 shows an example of correspondence of captured images based on a plurality of microscope image data obtained by the first and second microscope apparatuses 30 and 40, respectively.
  • a part of the Z position in the example of FIG. 4, Z position Z 1 + ⁇ Z 1 , Z 1 + 3 ⁇ Z 1, etc.
  • No image is taken in, and there is no microscope image that can be compared with the microscope image of the first microscope device 30.
  • step 130 the user selects a target space, which is a spatial location for which the corresponding image data is generated by interpolation.
  • the object space can be given by the Z position of the sample 9.
  • the image processing unit 60 processes the microscope image data respectively obtained by the first and second microscope apparatuses 30 and 40, and displays a list of captured images included in each on the screen of the display unit 63. To display. The user grasps from the display on the screen that there is no microscope image of the second microscope apparatus 40 at the Z positions Z 1 + ⁇ Z 1 and Z 3 + 3 ⁇ Z 3 where the microscope image of the first microscope apparatus 30 exists. .
  • the user selects the target space, for example, by inputting a numerical value of the Z position of the sample 9 or clicking or touching a point on the screen indicating the Z position via the input unit 51.
  • the Z position Z 1 + ⁇ Z 1 is selected as the target space.
  • the microscope image data at the Z position Z 1 + ⁇ Z 1 among the plurality of microscope image data of the first microscope device 30 is specified as the first microscope image data.
  • the second microscope image data and the third microscope image data used for generation of the corresponding image data are specified from among the plurality of microscope data of the second microscope device 40.
  • the microscope image data closest to the Z position Z 1 + ⁇ Z 1 in the ⁇ Z direction is specified as the second microscope image data and the third microscope image data.
  • the microscope image of the second microscope apparatus 40 corresponding to the first microscope image data that is the microscope image data of the first surface (Z position Z 1 + ⁇ Z 1 ) in the microscope image data of the first microscope apparatus 30.
  • the first microscope image data and Corresponding images at the same Z position are generated.
  • step 140 the image processing unit 60 confirms the sampling condition for the target space selected in step 130, and evaluates whether the corresponding image data for the target space can be generated by interpolation.
  • the sampling condition includes at least the following condition regarding the interval between the Z positions where at least two different captured images based on the microscope image data are captured, that is, the step amount ⁇ Z in the optical axis direction.
  • the cut-off frequency k z, max is the maximum range in which the optical transfer function (OTF) of the optical system 20 extends in a conjugate frequency direction with respect to the optical axis direction.
  • OTF optical transfer function
  • the image processing unit 60 details the sampling condition [Expression (1)], that is, the values of the step amount ⁇ Z and the cut-off frequency k z, max , whether the sampling condition is satisfied, and when the condition is not satisfied On the screen of the display unit 63, there is a possibility that aliasing may occur and the degree of the possibility of occurrence. As a result, the user can confirm whether or not the sampling condition is satisfied. For example, when the sampling condition is not satisfied, the occurrence of aliasing can be avoided.
  • step 150 the user determines whether to generate image data for the target space by interpolation or to re-image.
  • the user selects either interpolation or re-imaging via the input unit 51 based on the sampling condition confirmed in step 140. For example, the user can select interpolation when the sampling condition is satisfied, and can select re-imaging when the sampling condition is not satisfied. The user can also select interpolation even when the sampling condition is not satisfied.
  • the image processing unit 60 receives an instruction and proceeds to step 160.
  • the control unit 50 receives an instruction and proceeds to step 165.
  • step 160 the image generation unit 61 generates corresponding image data for the target space by interpolation.
  • FIG. 5 shows the principle of generation of corresponding image data using Fourier transform.
  • the image generation unit 61 determines the shift amount ⁇ for the target space selected in step 130.
  • the shift amount ⁇ is a difference between the Z position Z S in the target space and the Z position where any one of the N captured images based on the microscope image data f is captured.
  • the captured image f 1 captured at the Z position Z 1 closest to the Z position Z S may be selected.
  • the image generation unit 61 generates image data for the target space using the determined shift amount ⁇ .
  • the image generation unit 61 calculates a discrete Fourier transform F related to the Z coordinate of the microscope image data f.
  • N may be the total number of captured images included in the microscope image data f, or may be the number of two or more images that satisfy the sampling condition [Expression (1)] among all captured images.
  • the image generation unit 61 calculates an inverse discrete Fourier transform f using a phase factor (2 ⁇ / N) K (Z ⁇ ) that incorporates the shift amount ⁇ with respect to the Z coordinate. As shown in the center of FIG. 5, new microscope image data f (X, Y, Z ⁇ ) in which the Z position is shifted by ⁇ is generated by the calculation of Expression (3). Finally, the image generation unit 61 extracts image data f 1 ′ for the Z position Z S from the generated microscope image data. Thereby, corresponding image data for the target space is obtained.
  • the corresponding image data is an example of image data corresponding to the first microscope image data in that the Z position is the same as the first microscope image data at the Z position Z 1 + ⁇ Z 1 of the sample 9.
  • the second microscope image data which is image data captured at a Z position different from the Z position Z 1 + ⁇ Z 1, and an image captured at another Z position.
  • Corresponding image data is generated based on the third microscope image data that is image data.
  • the first microscope image data obtained at the Z position Z 1 + ⁇ Z 1 of the sample 9 is referred to as the first microscope image data obtained on the first surface
  • the corresponding image data is the first image of the sample 9.
  • step 165 the control unit 50 generates image data for the target space by re-imaging.
  • the control unit 50 sets an observation condition for re-imaging.
  • Control unit 50 captures a sample 9 at Z position Z s by the second microscope 40. Details are the same as in step 120 above.
  • the obtained corresponding image data is transmitted to the image processing unit 60.
  • step 170 the corresponding image data generated by the image generation unit 61 in step 160 or the image data generated by re-imaging in step 165 by the image output unit 62 is converted into the microscope image data generated in step 120.
  • step 180 the display unit 63 displays the Z stack image edited in step 170 on the screen.
  • FIG. 6 shows an example of the display screen 63a of the display unit 63 that displays the Z stack images obtained by the first and second microscope apparatuses 30 and 40, respectively.
  • the display unit 63 includes the first and second microscope apparatuses and the microscope method on the upper stage of the display screen 63a, one image 63c, 63d among the Z stack images obtained by the respective microscope apparatuses in the middle stage, and the Z stack on the lower stage.
  • Information about the image is displayed.
  • the display unit 63 displays the correspondence image 63d generated by interpolation by the image generation unit 61 and integrated with the microscope image data, visually distinguished from the captured image, for example, the image 63c.
  • the outer frame surrounding the corresponding image 63d is displayed thick, and the character “interpolation” indicating that the corresponding image 63d is interpolated is displayed.
  • the cross section of the sample imaged under one observation condition cannot be imaged under the other observation condition.
  • the reason is, for example, that the time for fading is different between the fluorescent dye for the first viewing condition and the fluorescent dye for the second viewing condition, and the number of images that can be captured before fading is different.
  • the wavelengths for exciting the multicolor fluorescent dyes are different under the first and second observation conditions, the amount of damage given to the sample is different even during the same imaging time, so the total amount of damage given to the sample is considered. Thus, the number of images that can be captured is different.
  • the microscope system 100 for one observation condition as described above, even if there is no microscope image data corresponding to the microscope image data at a certain Z position of the sample in the other observation condition, A corresponding image corresponding to the other microscope image can be generated and interpolated from the microscope image data. Therefore, it becomes possible to easily compare microscope images obtained under different observation conditions.
  • Image processing for generating image data corresponding to the image data is performed. Since a high-resolution microscope method generally requires long-time observation, the first microscope device 30 performs imaging at a certain Z position of the sample 9 for reasons such as fading of the fluorescent dye, but the second microscope. In some cases, imaging by the device 40 is not performed. Even in such a case, it is possible to easily compare the microscope images obtained from the first and second microscope devices 30 and 40 by generating and supplementing the corresponding images from the microscope image data obtained by the second microscope device 40. It becomes possible.
  • image processing for generating image data corresponding to the microscope image data obtained by the first microscope device 30 is performed based on the microscope image data obtained by the second microscope device 40.
  • corresponding image data corresponding to the microscope image data obtained by the second microscope device 40 may be generated based on the microscope image data obtained by the first microscope device 30.
  • imaging with the second microscope apparatus 40 with high resolution is performed, but imaging with the first microscope apparatus 30 with low resolution may not be performed. In such a case, it is possible to easily compare the microscope images obtained by the first and second microscope devices 30 and 40 by generating and supplementing the corresponding images from the microscope image data obtained by the first microscope device 30. It becomes.
  • a plurality of objective lenses may be provided in the microscope system 100, and the objective lenses may be changed according to the first observation condition and the second observation condition.
  • the stage 11 and a part of the optical system 20 are shared by the first and second microscope apparatuses 30 and 40.
  • Each of the microscope apparatuses 30 and 40 may have a separate stage and optical system.
  • a marker is provided close to the sample on the holding member 8 that holds the sample, and the position of the sample is determined with reference to the marker when observing the sample with each microscope apparatus, so that the same range can be obtained. The sample can be imaged.
  • the sampling condition [Expression (1)] is checked in order to evaluate whether the corresponding image data can be generated by interpolation in Step 140. However, instead of this, it may be confirmed whether or not both of the following two interpolation conditions are satisfied.
  • the first condition is that the Z position of the sample 9 corresponding to the target space is located between the Z positions at which at least two different captured images are captured in the image data based on the microscope image data.
  • the second condition is that the Z position of the sample 9 corresponding to the target space is a predetermined range from the Z position where each of at least two different captured images is captured, for example, the focal point of the objective lens used at the time of imaging. It is included within the depth range.
  • the image of the sample 9 on the target space captured in the captured image is obtained by performing interpolation using image data corresponding to at least two different captured images by the method of the first embodiment.
  • the reflected corresponding image data can be reproduced. If interpolation is performed using image data that does not satisfy the interpolation condition, the image of the sample 9 on the target space to be extracted is not captured in the captured image, and thus differs greatly from the original sample image and includes noise. Corresponding image data may be generated.
  • step 150 the user determines whether the corresponding image data is generated by interpolation or re-imaged. It may be determined by the generation unit 61. For example, the image processing unit 60 may select interpolation when the sampling condition [Expression (1)] is satisfied, and may select re-imaging when the sampling condition is not satisfied.
  • the shift amount ⁇ is known when the corresponding image data is generated in step 160.
  • the first, second, and third microscopes are known. Since the reference positions Z 1 and Z 2 in the optical axis direction are unknown in the image data g and f, the shift amount ⁇ may be unknown. In such a case, in step 160, first, the shift amount ⁇ is calculated. However, it is assumed that the Z intervals ⁇ Z 1 and ⁇ Z 2 are known in each data. The Z intervals ⁇ Z 1 and ⁇ Z 2 may not be equal to each other.
  • FIG. 7 shows an estimation of the shift amount ⁇ by taking the first microscope image data g, the second and third microscope image data f as an example.
  • the image generation unit 61 converts the Z interval ⁇ Z 2 of each image based on the second and third microscope image data f into ⁇ Z 1 to generate new microscope image data f ′.
  • Image data for .about.N ⁇ 1) are extracted, and new microscope image data f ′ having a Z interval ⁇ Z 1 is constructed from the extracted N images.
  • the image generation unit 61 shifts the image data f ′ by the shift amount ⁇ with respect to the Z coordinate by Fourier transform [Expression (2) and Expression (3)], and the correlation coefficient ⁇ ( ⁇ ) regarding the shift amount ⁇ Is calculated.
  • E (f) is the average of f.
  • the image generation unit 61 estimates a target shift amount from ⁇ that maximizes the correlation coefficient ⁇ ( ⁇ ).
  • the image generation unit 61 converts the second and third microscope image data f into a Fourier transform [Equation (2)] and a phase factor that incorporates the shift amount ⁇ as described above.
  • Corresponding image data can be generated by calculating the transformation [Expression (3)] and extracting the image data from the inverse Fourier transform.
  • the method for estimating the shift amount it is possible to generate corresponding image data even when the shift amount ⁇ is unknown. Therefore, even when the shift amount ⁇ is unknown, a corresponding image corresponding to the other microscope image can be generated and interpolated from one microscope image data, and the microscope images obtained under different observation conditions can be easily compared. It becomes possible to do.
  • other microscope image data of the second microscope device 40 may be used together.
  • the corresponding image data is generated by the discrete Fourier transform in Step 160, but instead, the corresponding image is obtained by the weighted average or the fast multipole method. Data may be generated.
  • FIG. 8 shows the generation principle of the corresponding image data using the weighted average.
  • the second and third microscope image data are the image data I A and I B at the first and second positions Z A and Z B including the position Z X of the image data I X therebetween.
  • d A Z A ⁇ Z X
  • d B Z X ⁇ Z B.
  • the fast multipole method based on the second and second microscope image data, it is assumed that charges equal to the respective luminance values exist at positions on the two-dimensional surface corresponding to the respective pixels of the captured image, and the image is generated by each charge.
  • Corresponding image data is generated by estimating the superposition of the Coulomb potential generated at each position on the two-dimensional surface corresponding to each pixel as a luminance value.
  • the fast multipole method is applicable not only when the sampling condition is satisfied but also when the sampling condition is not satisfied.
  • the first condition in the interpolation condition that is, the target space is at least two different used for the interpolation. It is not always necessary to satisfy that the captured images are located between the Z positions of the captured samples 9. That is, the corresponding image data may be generated by extrapolating at least two different captured images to a position outside the Z position of the sample 9 on which each captured image is captured.
  • step 150 the user can select re-imaging when the sample 9 can be reused, for example, when it is a fixed specimen. If the user selects re-imaging in step 150 even though the sample 9 cannot be reused, step 165 may proceed to the next step 170 without re-imaging. Further, in step 150, it may be displayed on the display unit 63 that the sample 9 cannot be reused to suggest the user to select interpolation.
  • the corresponding image data at the same Z position as the Z position of the first microscope image data specified at step 130 is generated.
  • the Z position of the corresponding image data may not be the same as the Z position of the first microscope image data.
  • the Z position of the corresponding image data may be different from the Z position of the first microscope image data within the range. That is, the corresponding image data is not in the case where the Z position is the same as that of the first microscope image data, but in the range where there is no problem in comparing the image based on the first microscope image data and the image based on the contrast image data. Even when the positions are different, it can be said that the corresponding image data corresponds to the first microscope image data.
  • the first microscope apparatus 30 employs confocal microscopy as the first observation condition (first microscope method), and the second microscope apparatus 40 uses the second observation condition (second microscope method).
  • first microscope method the fluorescent dye introduced into the sample 9 is activated at a low density and irradiated with excitation light, so that only the activated fluorescent dye (only some of the fluorescent dyes) emits light.
  • second microscope method the second observation condition
  • the fluorescent dye introduced into the sample 9 is activated at a low density and irradiated with excitation light, so that only the activated fluorescent dye (only some of the fluorescent dyes) emits light.
  • the fluorescent image the images of the fluorescent dyes that emit light at a low density are individually separated, so that the positions of the individual fluorescent dyes can be specified.
  • the localization method includes STORM (probabilistic optical reconstruction method), PALM (photo activated localization method), and the like.
  • STORM probabilistic optical reconstruction method
  • PALM photo activated localization method
  • a three-dimensional STORM is adopted as an example, and a fluorescence image generated by the STORM is also called a STORM image.
  • the main body 99 includes an optical system 20 ′ instead of the optical system 20 according to the first embodiment.
  • Other configurations included in the main body 99, that is, the stage system 10, the first microscope device 30, and the second microscope device 40 are configured in the same manner as in the first embodiment.
  • FIG. 9 shows the configuration of the main body 99 included in the microscope system 100 according to the second embodiment, particularly the configuration of the optical system 20 ′.
  • the optical system 20 ′ includes a plurality of objective lenses 21 a and 21 b, a cylindrical lens 25, an imaging optical system 26, and a filter 24.
  • the filter 24 is configured similarly to the first embodiment.
  • the plurality of objective lenses 21a and 21b are optical elements that connect an intermediate image of the sample 9 on the stage 11, and are arranged directly below the stage 11 as an example in the present embodiment.
  • the plurality of objective lenses 21a and 21b have relatively deep and shallow focal depths, respectively, and the focal depths of the first and second microscope apparatuses 30 and 40 are changed by switching between them.
  • FIG. 9 shows a state in which the objective lens 21b is arranged on the optical axis L, and shows the focal plane 20b of the optical system 20 ′ including the objective lens 21b.
  • the cylindrical lens 25 is a semi-cylindrical lens element that collects light in only one direction within a plane orthogonal to the optical axis L, and is used when observing a sample by STORM.
  • the light detected via the cylindrical lens 25 changes in image size according to the distance from the focal plane 20b and changes in image shape depending on which side it is positioned with respect to the focal plane 20b. Therefore, the three-dimensional coordinates of the position of the emitted fluorescent dye can be specified from the size and shape of the detected image.
  • the imaging optical system 26 is one or a plurality of optical elements that condense light passing through the cylindrical lens 25 toward the second illumination / imaging unit 41, and is used when observing a sample with STORM.
  • the plurality of objective lenses 21a and 21b are supported by a revolver (not shown), and the plurality of objective lenses 21a and 21b are arranged on the optical axis L of the optical system 20 by rotating the revolver.
  • the cylindrical lens 25 and the imaging optical system 26 can move back and forth on the optical axis Z independently of the rotation of the revolver.
  • FIG. 10 shows a flow of microscopic observation and image processing according to the second embodiment.
  • a Z stack image or a three-dimensional fluorescence image
  • description of items that are the same as or correspond to the flow according to the first embodiment will be omitted as appropriate.
  • the first and second observation conditions are set by the user.
  • the first microscope method and the second microscope method are already set out of the first and second observation conditions.
  • the imaging condition of the first observation condition the objective lens 21a is used, the range in the XY plane to be imaged, the reference position Z 1 on the Z axis in imaging, the step amount ⁇ Z 1 in the Z direction, and the image to be captured the number N 1 is set.
  • the objective lens 21b is used, the range in the XY plane to be imaged, the reference position Z 2 on the Z axis in imaging, the Z direction The step amount ⁇ Z 2 and the number N 2 of images to be captured are set.
  • the user inputs observation conditions via the input unit 51, and the input conditions are transmitted to the control unit 50.
  • step 220 the sample 9 is imaged independently by the first and second microscope apparatuses 30 and 40, respectively.
  • FIG. 11 shows an imaging state by the first microscope device 30.
  • the control unit 50 controls the optical system 20 ′ to retract the objective lens 21b, the cylindrical lens 25, and the imaging optical system 26 from the optical axis L, and arrange the objective lens 21a on the optical axis L.
  • Control unit 50, Z position Z 1 of the sample 9 drives the stage system 10 to be positioned in the focal plane 20a of the optical system 20 'including the objective lens 21a, for imaging a sectional image of the sample 9.
  • the details are the same as in the first embodiment.
  • the imaging result is sent to the image processing unit 60 and stored in the image generation unit 61 as microscope image data of the first microscope apparatus 30.
  • the sample 9 is imaged by being positioned on the surface 20a.
  • a series of imaging results obtained by the first microscope device 30 is sent to the image processing unit 60, and each is stored in the image generation unit 61 as microscope image data of the first microscope device 30.
  • FIG. 12 shows an imaging state by the second microscope apparatus 40.
  • the control unit 50 controls the optical system 20 ′ to retract the objective lens 21a and the scanner 23 from the optical axis L, and arranges the objective lens 21b, the cylindrical lens 25, and the imaging optical system 26 on the optical axis L. To do.
  • Control unit 50 a stage system 10 to Z position Z 2 of the sample 9 come drives the focal plane 20b of the optical system 20 ', to image the sample 9.
  • the second microscope apparatus 40 emits illumination light from the second illumination / imaging unit 41, and illuminates the sample 9 on the stage 11 via the optical system 20 ′.
  • the second microscope apparatus 40 activates the fluorescent dye contained in the observation position 9a of the sample 9 located on the focal point at a low density and irradiates the excitation light (wavelength ⁇ 2 ). Fluorescence (wavelength ⁇ 2 ′) of only the activated fluorescent dye is emitted and detected by the second illumination / imaging unit 41 via the objective lens 21b, the cylindrical lens 25, and the imaging optical system 26, and the fluorescence image To get.
  • the second microscope apparatus 40 identifies the position of each fluorescent dye.
  • the activation, excitation, acquisition of the fluorescence image, and specification of the position are repeated a plurality of times, and a specific brightness value is assigned to the specified positions of the plurality of fluorescent dyes to generate a STORM image.
  • the sample 9 is imaged, and a STORM image is constructed at each Z position.
  • the STORM image at each Z position is stored in the image generation unit 61 as microscope image data of the second microscope apparatus 40.
  • FIG. 13 shows an example of correspondence of captured images based on the microscope image data obtained by the first and second microscope apparatuses 30 and 40, respectively.
  • the microscope image data of the second microscope apparatus 30 is image data in which a specific luminance value is assigned to a three-dimensional position
  • the microscope image data is represented by a rectangular parallelepiped in the drawing to indicate that it is three-dimensional image data.
  • the number of the rectangular parallelepiped corresponds to the number of steps (N 2 ) at the time of imaging.
  • step 230 as in step 130 in the first embodiment, the user selects a target space, which is a spatial location for which image data is to be generated by interpolation.
  • the object space can be given by the Z position of the sample 9.
  • the image processing unit 60 processes the microscope image data respectively obtained by the first and second microscope apparatuses 30 and 40 and displays a list of captured images included in each on the screen of the display unit 63.
  • the user selects an image at the Z position corresponding to the captured image captured by the first microscope apparatus 30 from the microscope image data obtained by the second microscope apparatus 40 via the input unit 51 (in the figure, parallel lines of dotted lines). To generate an image corresponding to the depth of focus of the objective lens 21a.
  • the microscope image data captured at the plurality of Z positions are respectively the first microscope image data. Identified as
  • step 240 as in step 140 in the first embodiment, the image processing unit 60 confirms sampling conditions for the target space selected in step 230, and generates image data for the target space by interpolation. It is evaluated whether or not it can be done.
  • the details of the sampling conditions are the same as in the first embodiment.
  • step 250 as in step 150 in the first embodiment, the user determines whether to generate image data for the target space by interpolation or to re-image. If interpolation is selected, the image processing unit 60 receives the instruction and proceeds to step 260. If re-imaging is selected, the control unit 50 receives the instruction and proceeds to step 265.
  • step 260 the image generation unit 61 generates corresponding image data corresponding to each first microscope image data specified in step 230.
  • the corresponding image data is generated from the second and third microscope image data so as to correspond to the focal depth of the optical system 20 ′ including the objective lens 21 a used in the first microscope apparatus 30.
  • the image generation unit 61 From the second and third microscope image data, the image generation unit 61 superimposes a point assigned with a specific luminance value within the focal depth range of the objective lens 21a in the Z-axis direction or projects it onto the XY plane. Thus, the corresponding image data is generated. In such a case, the corresponding image data may be generated using different colors depending on the Z position.
  • the second and third microscope image data used for generating the corresponding image data is such that the Z position of the first microscope data is sandwiched between the plurality of microscope image data of the second microscope apparatus 40. In other words, the first and second surfaces from which the first microscope data is obtained are between the second surface from which the second microscope image data is obtained and the third surface from which the third microscope image data is obtained. Corresponding image data is generated based on the three microscope image data.
  • step 265 the control unit 50 generates microscope image data for the target space by re-imaging.
  • the control unit 50 sets observation conditions for re-imaging, particularly the objective lens 21a to be used.
  • the control unit 50 images the sample 9 by the second microscope device 40 in the same manner as in step 120, except that the objective lens 21a is used. Details thereof are omitted.
  • the imaging result is transmitted to the image processing unit 60 and stored in the image generation unit 61.
  • step 270 similarly to step 170 in the first embodiment, the image output unit 62 performs corresponding image data generated by the image generation unit 61 in step 260 or a microscope image obtained by re-imaging in step 265. The data is integrated with the microscope image data obtained in step 220 previously.
  • step 280 the image integrated in step 270 is displayed on the screen by the display unit 63, as in step 180 in the first embodiment.
  • the details are as described above.
  • FIG. 14 shows an example of the generated corresponding image.
  • the image 141 is a corresponding image generated by superimposing, in the Z-axis direction, a point to which a specific luminance value is assigned within the range of the focal depth of the objective lens 21a in the second and third microscope image data. It is.
  • Corresponding image data is a point in the Z-axis direction at which a specific brightness value is assigned within the focal depth range of the objective lens 21a with the same Z position as the Z position of the first microscope image data as the center on the Z-axis. Since they are generated by overlapping, the first microscope image data corresponds to the Z position.
  • the corresponding image data is such that a point assigned with a specific luminance value in the range of the focal depth of the objective lens 21a used for obtaining the first microscope image data is overlapped in the Z-axis direction.
  • image data corresponding to the first microscope image data since the second and third microscope image data is microscope image data obtained by a localization method in which the position of a fluorescent dye is specified and a luminance value is assigned like STORM image data, the image is not blurred. An arbitrary Z position and an arbitrary range along the Z axis centered on the Z position can be cut out. Therefore, it is possible to easily generate a corresponding image corresponding to the depth of focus of the first microscope image data using confocal microscopy, and to facilitate comparison.
  • corresponding image data is generated from the second and third microscope image data by superimposing in the Z-axis direction a point assigned with a specific luminance value within the range of the focal depth of the objective lens 21a.
  • the corresponding image data may be generated by performing a convolution operation using the point spread function PSF of the first microscope apparatus 30 as described below.
  • the corresponding image data I can be calculated as follows using the second and third microscope image data O and the point spread function PSF.
  • the second and third microscope image data O is obtained as a distribution of points to which specific luminance values are assigned in a preset three-dimensional space (X, Y, Z). Yes.
  • the second and third microscope image data O is a set of luminance values related to discrete X, Y, and Z coordinates, but is represented here as a function related to the coordinates X, Y, and Z for convenience.
  • Equation (5) can be rewritten as follows.
  • the PSF of the first microscope apparatus 30 is used as the point spread function PSF, and the second and third microscope image data O are convolved with respect to the depth range corresponding to the focal depth of the objective lens 21a.
  • the PSF of the first microscope apparatus 30 include a PSF of the optical system 20, a PSF of an illumination optical system that expresses an illumination pattern, a PSF of a pinhole, and a product thereof.
  • the right side of Equation (6) can be calculated using discrete Fourier transform.
  • the image generation unit 61 performs discrete Fourier transform on the second and third microscope image data O and the point spread function PSF, respectively, and calculates the right side by performing inverse discrete Fourier transform on the product of each Fourier transform.
  • the image 14 shows an example of an image obtained by using the point spread function PSF that expresses the spread of the bright spot image by the objective lens 21a in the equation (6).
  • the blur component included in the first microscope image data is captured by performing a convolution operation on the second microscope image data using Equation (6) using the point spread function PSF, and the S / The N ratio (signal to noise ratio) is close to the first microscope image data. Since the S / N ratios of the image quality of the image data are close to each other, the image data can be easily compared with each other.
  • the second and third microscope image data are used to generate the target image data, but the present invention is not limited to this.
  • other microscope image data may be used in addition to the second and third microscope image data.
  • different objective lenses 21a and 21b are used in the imaging of the sample by the first and second microscope apparatuses 30 and 40 in step 220, respectively.
  • a common objective lens may be used.
  • the objective lens 21b may be used in the first microscope apparatus 30 and the objective lens 21a may be used in the second microscope apparatus 40.
  • the first microscope apparatus 30 used the second and third microscope image data obtained by the second microscope apparatus 40 in step 260.
  • a corresponding image corresponding to the depth of focus of the objective lens 21a is generated.
  • the depth of focus of the objective lens 21b used by the second microscope device 40 is obtained from the first microscope image data obtained by the first microscope device 30. Corresponding images corresponding to may be generated.
  • the target space for generating the corresponding image data is selected by the user.
  • the object space can be given by the Z position of the sample 9.
  • the image processing unit 60 processes the microscope image data respectively obtained by the first and second microscope apparatuses 30 and 40 and displays a list of captured images included in each on the screen of the display unit 63. Based on the display on the screen, the user selects any Z position of the microscope image data of the second microscope device 40 as the target space from the microscope image data obtained by the first microscope device 30 via the input unit 51. Then, it is selected to generate a corresponding image corresponding to the focal depth of the objective lens 21b used by the second microscope apparatus 40. Accordingly, the first microscope image data is specified from the microscope image data of the first microscope device 30, and the second and third microscope image data are specified from the microscope image data of the second microscope device 40.
  • step 260 for example, the first microscope image data is deconvolved by the image generation unit 61 so that the corresponding image data corresponds to the focal depth of the objective lens 21b used by the second microscope apparatus 40. Is generated. The degree of blur is reduced in the corresponding image by the deconvolution operation.
  • the deconvolution operation By generating a corresponding image corresponding to the focal depth of the objective lens 21b used by the second microscope apparatus 40 from the first microscope image data, it is possible to easily compare the image data with each other.
  • the image output unit 62 superimposes specific luminance value points included in the respective focal depth ranges of the Z position in the target space from the second and third microscope image data in the Z-axis direction.
  • a pair image for the corresponding image data may be used, and the pair image may be displayed on the screen of the display unit 63 in step 280.
  • the stage 11 and a part of the optical system 20 are shared by the first and second microscope apparatuses 30 and 40.
  • Each of the microscope apparatuses 30 and 40 may have a separate stage and optical system.
  • a marker is provided close to the sample on the holding member 8 that holds the sample, and the position of the sample is determined with reference to the marker when observing the sample with each microscope apparatus, so that the same range can be obtained.
  • the sample can be imaged.
  • step 250 the user can select re-imaging when the sample 9 can be reused, for example, when it is a fixed specimen. If the user selects re-imaging in step 250 even though the sample 9 cannot be reused, the process may proceed to the next step 270 without re-imaging in step 265. Further, in step 250, it may be indicated on the display unit 63 that the sample 9 cannot be reused to suggest the user to select interpolation.
  • the first and second microscope apparatuses 30 and 40 were used to image a sample using a common or different objective lens, respectively.
  • the obtained microscope image data captures the sample image within the range of the focal depth of the objective lens used, the sample image at an arbitrary depth within the range of the focal depth from the microscope image data. Can be extracted.
  • a microscope method that does not use an objective lens for example, an electron microscope method in which a sample is irradiated with an electron beam to observe the surface, the sample is sliced and the structure of the surface is imaged. The image is not captured in the microscope image data and cannot be extracted from the microscope image data.
  • image data including an image of a portion that has not been sliced is generated from the microscope image data obtained for the sliced sample portion.
  • the microscope system 100 according to the first embodiment is used, and as an example, STORM is adopted as the first observation condition (first microscope method) for the first microscope apparatus 30, and the second microscope apparatus is used. Electron microscopy is employed as the second observation condition (second microscopy) for 40.
  • the second microscope apparatus 40 includes, for example, an electron gun, a focusing lens and an objective lens, a scanning coil, and a detector (all not shown).
  • the sample to be observed is sliced to an appropriate thickness and supported on the stage 11 as a sample.
  • the second microscope apparatus 40 accelerates the electron beam with an acceleration voltage of, for example, several hundred volts to several tens of kV with an electron gun, focuses the electron beam into one spot with a focusing lens and an objective lens, and focuses the spot on the sample with a scanning coil.
  • the sample is moved and scanned with an electron beam, and signal electrons generated from the sample are detected by a detector.
  • the detection result is transmitted to the image processing unit 60 (image generation unit 61) as second and third microscope image data, and image data having the amount of signal electrons as the brightness of each point is generated. Since the amount of signal electrons generated varies depending on the surface structure of the sample, the image data represents the surface structure of the sample.
  • the first microscope apparatus 30 is configured independently of the second microscope apparatus 40 in the microscope system 100.
  • the configuration of the first microscope apparatus 30 is the same as that of the second embodiment.
  • FIG. 15 shows a flow of microscope observation and image processing according to the third embodiment.
  • description of the matters that are the same as or correspond to the flows according to the first and second embodiments will be appropriately omitted.
  • the first and second observation conditions are set by the user, as in steps 110 and 210 in the first and second embodiments.
  • the first microscope method and the second microscope method are already set out of the first and second observation conditions.
  • a reference position Z 1 on the Z axis in imaging, a step amount ⁇ Z 1 in the Z direction, and the number N 1 of images to be captured are set.
  • a range in the XY plane for imaging the sample 9 and the number of images to be captured (equal to the number of samples) N 2 are set.
  • the information on the sample includes the thickness of each sample, the Z position of the sample 9 of each sample, and the like.
  • step 320 the sample is imaged independently by the first and second microscope apparatuses 30 and 40, respectively.
  • the sample is imaged by the first microscope apparatus 30.
  • the details are the same as in the second embodiment.
  • the sample is imaged by the second microscope device 40.
  • the user slices the sample used for imaging by the first microscope apparatus 30 in the depth direction according to the imaging conditions, and supports one of them as a sample on the stage 11.
  • the control unit 50 controls the stage system 10 and drives the stage 11 in the Z-axis direction to position the surface of the sample on the electron beam spot. In this state, the control unit 50 scans the sample with an electron beam using the second microscope apparatus 40. Thereby, the surface structure of the first sample is imaged.
  • the control unit 50 stops the electron gun and retracts the stage 11 from the spot.
  • the user supports the second sample on the stage 11.
  • the controller 50 scans the second sample with an electron beam in the same manner as described above.
  • the control unit 50 sequentially scans all the samples with an electron beam.
  • a series of imaging results obtained by the second microscope device 40 is sent to the image processing unit 60 and stored in the image generating unit 61 as a plurality of microscope image data of the second microscope device 40.
  • FIG. 16 shows an example of correspondence between captured images based on the microscope image data obtained by the first and second microscope apparatuses 30 and 40, respectively.
  • the microscope image data (electron microscope image) of the second microscope device 40 represents the surface structure of the sample sliced at different Z positions Za and Zc in the sample.
  • the microscope image data (STORM image) of the first microscope apparatus 30 is as described in the second embodiment.
  • step 330 as in steps 130 and 230 in the first and second embodiments, the user selects a target space, which is a spatial location for which the corresponding image data is generated by interpolation.
  • the object space can be given by the Z position of the sample 9.
  • the image processing unit 60 processes the microscope image data obtained by the first and second microscope apparatuses 30 and 40, respectively, and obtains a list of captured images based on each, images 191, 192 and images 196 for the Z positions Za, Zc. , 198 are displayed on the screen of the display unit 63.
  • the user grasps from the display on the screen that the first microscope apparatus 30 has taken an image but the second microscope apparatus 40 has not taken an image at the Z position Zb.
  • the user selects to generate corresponding image data at the Z position Zb of the sample 9 from the microscope image data obtained by the second microscope device 40 via the input unit 51.
  • the microscope image data at the Z position Zb among the plurality of microscope image data of the first microscope apparatus 30 is specified as the first microscope image data.
  • the second microscope image data imaged at the Z position Za and the third microscope image data imaged at the Z position Zc sandwiching the Z position Zb therebetween are specified.
  • a corresponding image at the same Z position Zb as the first microscope image data is generated from the second microscope image data and the third microscope image data as follows.
  • step 340 the sampling condition is confirmed by the image processing unit 60 as in steps 140 and 240 in the first and second embodiments.
  • step 350 as in steps 150 and 250 in the first and second embodiments, the user determines whether to generate image data for the target space by interpolation. If interpolation is selected, the image processing unit 60 receives the instruction and proceeds to step 360. When it is selected not to interpolate, the control unit 50 receives the instruction and ends the flow.
  • the image generation unit 61 generates corresponding image data for the target space by interpolation.
  • the corresponding image data 195 corresponding to the depth Zb in the sample is the second and third microscope image data obtained by electron microscopy, and the respective patterns of the captured images 191 and 192 at the positions Za and Zc. Is generated based on
  • step 360 the segmentation method is executed to classify the patterns in the captured images 191 and 192.
  • the image generation unit 61 extracts the captured images 191 and 192 at the depths Za and Zc in the sample from the first microscope image data, and displays them on the screen of the display unit 63. Segmentation is performed on each captured image 191 and 192 by the image generation unit 61 or the user. That is, in the sample captured images 191 and 192 obtained by electron microscopy, structures in the image are determined, and areas including the determined structures are colored using different colors. Thereby, as shown in FIG. 16, two images 193 and 194 classified as structures are obtained.
  • image data 195 is generated from the two images 193 and 194 classified in structure.
  • the image generation unit 61 compares the structures determined between the two images 193 and 194 and interpolates the shape of the area of the corresponding structure according to the depths Za, Zb, and Zc, thereby corresponding image data.
  • the area of the structure in 195 is generated. Interpolation is performed, for example, by arranging the same number of interpolation points on the outer edge of the area of the corresponding structure between the two images 193 and 194, and corresponding two of the corresponding interpolation points, for example, closest to each other.
  • the positions of the points in the XY plane are weighted using the depths Za, Zb, and Zc to obtain the positions of the interpolation points in the corresponding image 195, and the obtained plurality of interpolation points are continuously connected. Can be done.
  • the image generation unit 61 can determine whether or not the structure corresponds between the two images 193 and 194 based on the similarity of the shape of the area of the structure, the degree of overlap, and the like.
  • the continuous deformation in the depth direction can be understood. Since the images 191 and 192 obtained by electron microscopy are usually expressed in binary, for example, monochrome, it is difficult to distinguish the structure on the image, and the structural change in the depth direction of the sample is understood. Is difficult. Therefore, by identifying the structure by the segmentation method, the structure on the images 191 and 192 is identified, and the corresponding image data 195 is more accurately obtained by using the continuous change in the depth direction of the same structure. Can be generated.
  • step 370 the image generation unit 61 is selected from the first microscope image data (STORM image data) in the Z positions Za and Zc of the sample 9 captured by the first microscope apparatus 30 in step 320 and in step 330.
  • the three-dimensional position information of a specific luminance value at each of the Z positions Zb of the sample 9 is extracted.
  • step 375 the image generation unit 61 reconstructs the STORM images 196 to 198 for each of the Z positions Za, Zb, and Zc using the three-dimensional position information of the specific luminance value extracted in step 370.
  • the STORM images 196 to 198 are reconstructed by, for example, overlapping points having specific luminance values in the Z direction by the thickness of the sample.
  • step 380 similarly to steps 170 and 270 in the first and second embodiments, the image output unit 62 converts the corresponding image data 195 generated by the image generation unit 61 in step 365 to the second and third microscopes.
  • the image data is integrated into a series of image data and edited, and the STORM images 196 to 198 reconstructed in step 375 are integrated into the series of image data.
  • step 390 the image edited or integrated in step 380 is displayed on the screen by the display unit 63, similarly to steps 180 and 280 in the first and second embodiments.
  • the details are as described above.
  • the Z position Zb of the sample 9 is selected as the target space in Step 330, and the Z position Zb is sandwiched between Steps 360 and 365.
  • Corresponding image data was generated based on the patterns of the electron microscope images obtained at the Z positions Za and Zc, respectively. Instead, at least two different electron microscope images are used to extrapolate to a position (Zb ⁇ Za or Zb> Zc) outside the depth (Za, Zc) in the sample from which each image was obtained. Corresponding image data corresponding to the target space may be generated.
  • the second operation is performed in steps 360 and 365.
  • the image data of the electron microscope image is generated from the third microscope image data and, conversely, prior to the generation of the image data of the electron microscope image from the second and third microscope image data, from the first microscope image data
  • a STORM image may be reconstructed.
  • the STORM image is reconstructed by superimposing points having specific luminance values of the first microscope image data in the Z direction.
  • a STORM image may be reconstructed by performing a convolution operation using the point spread function PSF.
  • the image captured by the electron microscope method using the second microscope apparatus 40 is structurally classified by the segmentation method, and image data is generated by interpolation.
  • image data may be generated from the weighted average of two images (that is, luminance values) as described above with reference to FIG.
  • the sample 9 is sliced, and each slice is used as a sample and imaged by the second microscope device 40.
  • the sample is sliced prior to imaging by the first and second microscope apparatuses 30 and 40, and each slice is used as a sample by both the first and second microscope apparatuses 30 and 40. It is good also as imaging.
  • the order of imaging of the sample by the first and second microscope apparatuses 30 and 40 may be arbitrary.
  • time-series images also referred to as time-lapse images
  • each time-series image is included in the other time-series image.
  • the microscope system 100 is used.
  • confocal microscopy is adopted as a microscopy for the first microscope device 30, and as a microscopy for the second microscope device 40.
  • Adopt SIM structured illumination microscopy.
  • the same fluorescent dye is introduced into the sample 9 in the first and second microscopy, and correspondingly, the same wavelength is adopted under the illumination conditions.
  • FIG. 17 shows a flow of microscope observation and image processing according to the fourth embodiment.
  • a time-series image of the sample 9 is captured.
  • description of items that are the same as or correspond to those in the flows according to the first to third embodiments will be omitted as appropriate.
  • the first and second observation conditions are set by the user.
  • the first microscope method and the second microscope method are already set out of the first and second observation conditions.
  • an imaging condition of the first observation condition the use of the objective lens 21a, the range in the XY plane for imaging, the imaging start time T 1, the imaging interval [Delta] T 1, the number of images N 1 for imaging is set.
  • the imaging condition of the second observation condition use of the objective lens 21b, a range in the XY plane for imaging, an imaging start time T 2 , an imaging interval ⁇ T 2 , and the number of images to be captured N 2 are set.
  • the sample 9 is alternately imaged by the first and second microscope apparatuses 30 and 40.
  • the user inputs these observation conditions via the input unit 51, and the input conditions are transmitted to the control unit 50.
  • step 420 the sample 9 is imaged alternately by the first and second microscope apparatuses 30 and 40.
  • the control unit 50 drives the stage 11 in the Z-axis direction to position the Z position of the sample 9 on the focal plane 20a.
  • Control unit 50 at time T 1, for imaging a sample 9 by the first microscope 30. Prior to imaging, the control unit 50 controls the optical system 20 to switch to the objective lens 21a.
  • the first microscope apparatus 30 emits illumination light having a wavelength ⁇ 1 from the first illumination / imaging unit 31 and illuminates the sample 9 on the stage 11 via the optical system 20 (objective lens 21a). To do. As a result, the observation position 9a of the sample 9 located on the focal point is illuminated, and the fluorescent dye contained therein emits fluorescence of wavelength ⁇ 1 ′. The fluorescent light is collected through the optical system 20 (objective lens 21a) and captured by the first illumination / imaging unit 31. In parallel with the imaging of fluorescence, the control unit 50 scans the sample 9 within the XY scanning range on the focal plane 20a by controlling the scanner 23 and shaking the illumination light in the XY direction. Thereby, the sample 9 is scanned, and a cross-sectional image of the sample 9 at the Z scanning position is captured.
  • the second microscope device 40 is stopped.
  • control unit 50 After completion of the imaging by the first microscope 30, the control unit 50, at time T 2, to image the sample 9 by the second microscope 40. Prior to this, the control unit 50 controls the optical system 20 to switch to the objective lens 21b.
  • the second microscope apparatus 40 emits illumination light having a wavelength ⁇ 2 from the second illumination / imaging unit 41 and illuminates the sample 9 on the stage 11 via the optical system 20 (objective lens 21b). To do. Thereby, the observation position 9b of the sample 9 located on the focal point is illuminated, and the fluorescent dye contained therein emits fluorescence having the wavelength ⁇ 2 ′. The fluorescence is condensed via the optical system 20 (objective lens 21b) and imaged by the second illumination / imaging unit 41. Note that the scanner 23 is retracted from the optical axis L during the imaging of the second microscope apparatus 40.
  • the first microscope apparatus 30 is stopped.
  • the control unit 50 controls the optical system 20 to switch to the objective lens 21a, images the sample 9 by the first microscope device 30 at time T 1 + ⁇ T 1 , and the optical system. 20 is switched to the objective lens 21b, and the sample 9 is imaged by the second microscope device 40 at time T 2 + ⁇ T 2 .
  • the control unit 50 alternately repeats imaging of the sample 9 by the first and second microscope apparatuses 30 and 40. Thereby, each sample 9 is continuously imaged with respect to time.
  • a series of imaging results obtained by the first microscope device 30 is sent to the image processing unit 60 and stored in the image generating unit 61 as a plurality of microscope image data of the first microscope device 30.
  • a series of imaging results obtained by the second microscope device 40 is sent to the image processing unit 60 and stored in the image generating unit 61 as a plurality of microscope image data of the second microscope device.
  • the microscope image data of the first and second microscope apparatuses 30 and 40 includes image data obtained under the first and second observation conditions at different times.
  • the target time for generating corresponding image data by interpolation is selected by the user.
  • the image processing unit 60 processes the microscope image data respectively obtained by the first and second microscope apparatuses 30 and 40 and displays a list of captured images included in each on the screen of the display unit 63. From the display, the user captures an image corresponding to an image captured by the second microscope device 40 at time T 2 (from time T 1 + ⁇ T 1 ) in the microscope image data of the first microscope device 30. It is grasped that the image corresponding to the image captured at time T 1 + ⁇ T 1 by the first microscope device 30 is not captured in the microscope image data of the second microscope device 40.
  • the user selects the target time via the input unit 51 by, for example, clicking or touching a point on the screen indicating the time.
  • the times T 2 and T 21 are selected for the microscope image data of the first microscope device 30 and the times T 1 + ⁇ T 1 are selected for the microscope image data of the second microscope device 40 as the target time.
  • step 440 the image generation unit 61 generates corresponding image data for the target time by interpolation.
  • FIG. 21 shows the principle of generating corresponding image data by time interpolation.
  • the image generation unit 61 applies image recognition to each of the two images 231 and 232, extracts the objects 231a and 232a from the respective images, and positions the objects ra and rb in the images of the objects 231a and 232a.
  • the luminance values Ia and Ib of the objects 231a and 232a are calculated.
  • image recognition for example, template matching (autocorrelation method), optical flow method, segmentation method and the like can be employed.
  • the template is overlapped while shifting the position in the image to find the phase, and the target is searched from the image by determining that there is a figure that matches the template at the position where the calculated phase is the largest.
  • the template By superimposing the template on the image at the current time in the vicinity of the position where the template matches in the image before the current time, it is possible to efficiently search for the target from within the image.
  • template matching By applying template matching to each of the image before the current time and the image at the current time, it is possible to detect the object and its movement and deformation from within the image.
  • a feature vector that is invariant with the passage of time for example, a luminance value of a pixel or area that projects the object, is used to calculate a movement vector that represents the movement of the object between the two images.
  • the segmentation method is as described in the third embodiment, and the movement and deformation of an object are detected over time by classifying patterns in two images and comparing the classified structures with each other. Can do.
  • the image generation unit 61 calculates the target position rc and the luminance value Ic in the image 235 based on the patterns obtained in the images 231 and 232 at the two times Ta and Tb by image recognition.
  • , cb
  • the image generation unit 61 generates a corresponding image 235 by displaying a figure having the same shape as the template with a luminance value Ic at a position rc on a background such as a white background. By applying image recognition, the corresponding image 235 can be generated even when the target moves.
  • the image generation unit 61 converts the image data of two images 231 and 232 captured at time T 1 and T 1 + ⁇ T 1 out of the plurality of microscope image data of the first microscope apparatus 30. Based on this, image data of corresponding images 235 and 236 at the times T 2 and T 21 selected in step 430 are generated. The time T 1 selected in step 430 based on the image data of the two images 233 and 234 captured at time T 2 and T 2 + ⁇ T 2 among the plurality of microscope image data of the second microscope apparatus 40. Image data of the corresponding image 237 at + ⁇ T 1 is generated.
  • the correspondence image 237 will be further described as an example.
  • the microscope image data at the time T 1 + ⁇ T 1 is selected and specified as the first microscope image data.
  • a second microscope obtained at different second time (T2) from the first time the first microscope image data is obtained (T 1 + ⁇ T 1)
  • Image data and third microscope image data obtained at a third time (T2 + ⁇ T2) different from both the first time and the second time are specified.
  • the correspondence image 237 is generated based on the second and third third microscope image data.
  • step 450 the image output unit 62 integrates the image data generated by the image generation unit 61 in step 440 with the microscope image data previously obtained in step 420 and edits it into a time-series image.
  • step 460 the display unit 63 displays the time series image edited in step 450 on the screen.
  • the time required for imaging differs depending on the resolution for each microscope apparatus.
  • time-series images are constructed by sequentially capturing the same sample with the second microscope devices 30 and 40, one time-series image does not include an image at the same time as the image included in the other time-series image, In some cases, images at the same time cannot be compared.
  • the first and second microscope apparatuses 30 and 40 use illumination light having the same wavelength or adjacent wavelength spectra, the same sample cannot be imaged at the same time. Similarly, images at the same time cannot be compared. There is.
  • the image data corresponding to the time when the image of the other time-series image is captured from the microscope image data of one time-series image is generated and supplemented, so that the images at the same time can be compared. It becomes possible to do.
  • the image recognition is applied in step 440 to generate the image data.
  • two images are not applied without applying the image recognition.
  • the image data may be generated only by calculating the luminance value.
  • FIG. 22 shows a principle of generating processed image data by time interpolation using luminance value calculation.
  • An image 236 at 21 ) is generated.
  • , cb
  • the target 236b corresponding to a part of the target 231a in the image 231 is the luminance value Ia
  • the target 236c corresponding to a part of the target 232a in the image 232 is the luminance value Ib
  • An object 236d corresponding to the overlap with the object 232a in 232 is displayed with the luminance value Ic.
  • the image recognition in step 440 or the calculation of the brightness value described above is used at different times Ta and Tb (where Ta ⁇ Tb).
  • the two captured images are interpolated to generate an image at the target time Tc between the times Ta and Tb (that is, Ta ⁇ Tc ⁇ Tb), but the target time Tc is at least two different for use in the interpolation. It is not necessary that the time is between the time when each image was obtained. In other words, using at least two different images, an image corresponding to the target time is extrapolated at a time earlier (or Tc ⁇ Ta or Tc> Tb) than the time (Ta, Tb) at which each image was obtained. Data may be generated.
  • different objective lenses 21a and 21b are used under the first and second observation conditions, and in step 420, the first and second microscope apparatuses 30 are used. , 40 alternately images the sample 9, but the same objective lens is used in each of the first and second observation conditions, and the sample 9 is imaged in parallel by the first and second microscope devices 30, 40. May be.
  • the details are as described in the first embodiment.
  • two different fluorescent dyes are introduced into the sample 9 in the first and second microscopy methods, and correspondingly different wavelengths are employed in the illumination conditions.
  • an imaging start time T 1 T 2
  • the time required for imaging in the first and second microscope apparatuses 30 and 40 is determined according to the respective microscopy, for example.
  • the imaging intervals ⁇ T 1 and ⁇ T 2 of the first and second microscope apparatuses 30 and 40 are obtained by, for example, the time required for imaging in each microscope method and the first and second illumination / imaging units 31 and 41. It is determined according to the processing speed of the image processing unit 60 by the image processing unit 60. Therefore, the sample may be imaged at different times by the first and second microscope apparatuses 30 and 40, respectively.
  • FIG. 23 shows another example of a timing chart in which a sample is imaged in parallel by each of the first and second microscope apparatuses 30 and 40.
  • Each of the first and second microscope image data includes a series of image data obtained under the first and second observation conditions at a plurality of times.
  • corresponding image data at the same time as the imaging time of the first microscope image data specified in step 430 is generated.
  • the time of the corresponding image data may not be the same as the time of the first microscope image data.
  • the time of the corresponding image data may be different from the time of the first microscope image data. That is, the corresponding image data does not have the same time as the first microscope image data, but the time of each other is within a range that does not interfere with the comparison between the image based on the first microscope image data and the image based on the contrast image data. Even if they are different, it can be said that the corresponding image data corresponds to the first microscope image data.
  • the first and second microscope apparatuses 30 and 40 employ confocal microscopy and SIM (structured illumination microscopy) as the microscopy, respectively. May also employ the same microscopy, eg, confocal microscopy.
  • the sample is introduced by a multicolor fluorescence method, i.e., a plurality of (e.g., two) fluorescent dyes are introduced into the sample, and the first and second microscope apparatuses 30, 40 are irradiated with a plurality of illumination lights.
  • the sample may be imaged by illuminating and detecting the fluorescence emitted by each fluorescent dye in the sample. This makes it possible to simultaneously observe different portions of the sample into which different fluorescent substances are introduced.
  • the sample is alternately imaged by the first and second microscope apparatuses 30 and 40.
  • each of the first and second microscope apparatuses 30 and 40 is separately provided.
  • the stage and the optical system may be dedicated, and the same sample may be individually imaged.
  • a marker is provided close to the sample on the holding member 8 that holds the sample, and the position of the sample is determined with reference to the marker when observing the sample with each microscope apparatus, so that the same range can be obtained.
  • the sample can be imaged.
  • the user selects the target space or target time for generating image data by interpolation in steps 130, 230, 330, and 430.
  • the image generation unit 61 may automatically select the target space or the target time.
  • the first and second imaging conditions include the imaging conditions included in the first and second observation conditions, the reference position in the optical axis direction, the step amount in the optical axis direction, the number of images to be captured, and the like.
  • a target image can be selected by specifying a missing captured image in the second microscope image data.
  • the target time can be selected from the imaging conditions included in the first and second observation conditions, the imaging start time, the imaging interval, the number of images to be captured, and the like.
  • the sampling condition is confirmed before generating the image data for the target space. Instead, the image data is generated. After that, sampling conditions may be confirmed.
  • step 170, 270, 380, 450 the corresponding image data generated by the image generation unit 61 by interpolation is performed by the image output unit 62 in steps 170, 270, 380, and 450.
  • Steps 180 and 280 are integrated with the microscope image data obtained by imaging with the second microscope apparatus 40 (or the first microscope apparatus 30) and edited into a series of image data such as a Z stack image and a time series image.
  • 390, and 460 are displayed on the screen of the display unit 63.
  • the corresponding image data generated by the interpolation may not be integrated with the microscope image data obtained by the imaging, and the image based on the corresponding image data may be used.
  • an image based on the first microscope image data corresponding thereto may be displayed on the screen of the display unit 63.
  • the display unit 63 may display the corresponding image without associating the corresponding image with the microscope image obtained by the other microscope method.
  • a Z stack image is generated by imaging a plurality of Z positions of the sample 9 in the confocal microscopy, SIM, and electron microscopy. Only one Z position of the sample 9 may be imaged.
  • STORM a plurality of Z positions of the sample 9 are imaged to generate the STROM image. However, only one Z position of the sample 9 may be imaged to generate a STORM image.
  • the image processing unit 60 is not limited to the microscope image data transmitted from the first and second microscope apparatuses 30 and 40, and is separate from the microscope system 100. Image data obtained by the microscope apparatus may be processed. Further, the image output unit 62 processes the microscope image data obtained by the microscope system 100, and displays the microscope image captured by the separate microscope apparatus and the microscope image captured by the microscope system 100 on the screen of the display unit 63. They may be displayed side by side or superimposed.
  • the imaging elements of the first and second illumination / imaging units 31 and 41 may be charge coupled devices (CCD), CMOS, etc.
  • a light receiving element such as a photomultiplier tube (PMT) may be used instead of the imaging element.
  • PMT photomultiplier tube
  • an appropriate element may be adopted according to the wavelength of light to be received.
  • an arbitrary microscope method such as PET (Positron Tomography), MRI (Magnetic Resonance Imaging) can be applied to the first and second microscope apparatuses 30 and 40.
  • CT computed tomography
  • stereofluorescence microscopy epifluorescence microscopy
  • confocal microscopy confocal microscopy
  • SIM structured illumination microscopy
  • STORM probabilistic optical reconstruction microscopy
  • PALM photoactivated localization method
  • STED stirled emission control method
  • electron microscope atomic force microscope, or the like
  • the microscope system 100 includes two microscope apparatuses that employ two microscope methods, but is not limited to this, and three or more microscope apparatuses that employ three or more microscope methods, respectively. May be provided. Moreover, you may provide the at least 2 microscope apparatus which employ
  • the same fluorescent dye may be used in the first microscope apparatus 30 and the second microscope apparatus 40, or three or more colors of fluorescent dyes may be used.
  • One or both of the first microscope apparatus 30 and the second microscope apparatus 40 may image the sample 9 by autofluorescence or epi-illumination without using a fluorescent dye. Even if the fluorescent dye is not used in one or both of the first microscope apparatus 30 and the second microscope apparatus 40, the wavelength of illumination light is different between the first microscope apparatus 30 and the second microscope apparatus 40. The degree to which the sample 9 is damaged may be different. Therefore, imaging by one of the first microscope apparatus 30 and the second microscope apparatus 40 is performed at a certain Z position of the sample 9, but imaging by the other of the first microscope apparatus 30 and the second microscope apparatus 40 may not be performed. .
  • the sample 9 is moved with respect to the focal points of the objective lenses 21a and 21b by driving the stage 11 supporting the sample 9 in the Z-axis direction.
  • the revolver that supports the objective lenses 21a and 21b is driven in the Z-axis direction, or an optical element having refractive power is disposed on the optical axis L of the optical system 20, and the optical axis L
  • a configuration in which the sample 9 is moved with respect to the focal points of the objective lenses 21a and 21b by moving in a direction parallel to the lens may be employed.
  • the first and second microscope apparatuses 30 and 40 are made to emit or stop the illumination light from the first and second illumination / imaging units 31 and 41. Although it was decided to operate or stop, by arranging or retracting a filter such as a dichroic mirror on the optical axis L of the optical system 20 and sending or not sending the illumination light to the optical system 20, the first and second The microscope apparatuses 30 and 40 may be operated or stopped.
  • a filter such as a dichroic mirror
  • the microscope system 100 employs an inverted microscope system in which the objective lenses 21a and 21b are arranged below the stage 11 that supports the sample 9, and the sample is observed from below.
  • an erecting microscope system in which the objective lenses 21a and 21b are arranged above the stage 11 and the sample is observed from above may be adopted.
  • either an inverted type or an upright type may be adopted for each microscope apparatus.
  • the microscope image data of the first microscope apparatus 30 can also be the first microscope image data, but a separate name is used for convenience of explanation.
  • the microscope image data of the second microscope apparatus 40 can be the second microscope image data and the third microscope image data, but separate names are used for convenience of explanation.
  • a block is either (1) a stage in a process in which the operation is performed or (2) an apparatus responsible for performing the operation. May represent a section of Certain stages and sections are implemented by dedicated circuitry, programmable circuitry supplied with computer readable instructions stored on a computer readable medium, and / or processor supplied with computer readable instructions stored on a computer readable medium. It's okay.
  • Dedicated circuitry may include digital and / or analog hardware circuitry and may include integrated circuits (ICs) and / or discrete circuits.
  • Programmable circuits include memory elements such as logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA), etc. Reconfigurable hardware circuitry, including and the like.
  • Computer readable media may include any tangible device capable of storing instructions to be executed by a suitable device, such that a computer readable medium having instructions stored thereon is specified in a flowchart or block diagram. A product including instructions that can be executed to create a means for performing the operation. Examples of computer readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like.
  • Computer readable media include floppy disks, diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), Electrically erasable programmable read only memory (EEPROM), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (RTM) disc, memory stick, integrated A circuit card or the like may be included.
  • RAM random access memory
  • ROM read only memory
  • EPROM or flash memory erasable programmable read only memory
  • EEPROM Electrically erasable programmable read only memory
  • SRAM static random access memory
  • CD-ROM compact disc read only memory
  • DVD digital versatile disc
  • RTM Blu-ray
  • Computer readable instructions can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or object oriented programming such as Smalltalk, JAVA, C ++, etc. Including any source code or object code written in any combination of one or more programming languages, including languages and conventional procedural programming languages such as "C" programming language or similar programming languages Good.
  • Computer readable instructions may be directed to a general purpose computer, special purpose computer, or other programmable data processing device processor or programmable circuit locally or in a wide area network (WAN) such as a local area network (LAN), the Internet, etc.
  • the computer-readable instructions may be executed to create a means for performing the operations provided via and specified in the flowchart or block diagram.
  • processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and the like.
  • FIG. 24 shows an example of a computer 2200 in which aspects of the present invention may be embodied in whole or in part.
  • the program installed in the computer 2200 can cause the computer 2200 to function as an operation associated with the apparatus according to the embodiment of the present invention or one or more sections of the apparatus, or to perform the operation or the one or more sections.
  • the section can be executed and / or the computer 2200 can execute a process according to an embodiment of the present invention or a stage of the process.
  • Such a program may be executed by CPU 2212 to cause computer 2200 to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.
  • the computer 2200 includes a CPU 2212, a RAM 2214, a graphic controller 2216, and a display device 2218, which are connected to each other by a host controller 2210.
  • the computer 2200 also includes input / output units such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the host controller 2210 via the input / output controller 2220.
  • the computer also includes legacy input / output units, such as ROM 2230 and keyboard 2242, which are connected to input / output controller 2220 via input / output chip 2240.
  • the CPU 2212 operates according to the programs stored in the ROM 2230 and the RAM 2214, thereby controlling each unit.
  • the graphic controller 2216 obtains the image data generated by the CPU 2212 in a frame buffer or the like provided in the RAM 2214 or itself so that the image data is displayed on the display device 2218.
  • the communication interface 2222 communicates with other electronic devices via a network.
  • the hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200.
  • the DVD-ROM drive 2226 reads a program or data from the DVD-ROM 2201 and provides the program or data to the hard disk drive 2224 via the RAM 2214.
  • the IC card drive reads programs and data from the IC card and / or writes programs and data to the IC card.
  • the ROM 2230 stores therein a boot program executed by the computer 2200 at the time of activation and / or a program depending on the hardware of the computer 2200.
  • the input / output chip 2240 may also connect various input / output units to the input / output controller 2220 via parallel ports, serial ports, keyboard ports, mouse ports, and the like.
  • the program is provided by a computer readable medium such as a DVD-ROM 2201 or an IC card.
  • the program is read from a computer-readable medium, installed in the hard disk drive 2224, the RAM 2214, or the ROM 2230, which are also examples of the computer-readable medium, and executed by the CPU 2212.
  • Information processing described in these programs is read by the computer 2200 to bring about cooperation between the programs and the various types of hardware resources.
  • An apparatus or method may be configured by implementing information manipulation or processing in accordance with the use of computer 2200.
  • the CPU 2212 executes a communication program loaded in the RAM 2214 and performs communication processing on the communication interface 2222 based on processing described in the communication program. You may order.
  • the communication interface 2222 reads transmission data stored in a transmission buffer processing area provided in a recording medium such as the RAM 2214, the hard disk drive 2224, the DVD-ROM 2201, or an IC card under the control of the CPU 2212, and the read transmission. Data is transmitted to the network, or received data received from the network is written in a reception buffer processing area provided on the recording medium.
  • the CPU 2212 allows the RAM 2214 to read all or a necessary part of a file or database stored in an external recording medium such as a hard disk drive 2224, a DVD-ROM drive 2226 (DVD-ROM 2201), an IC card, etc. Various types of processing may be performed on the data on the RAM 2214. Next, the CPU 2212 writes back the processed data to the external recording medium.
  • an external recording medium such as a hard disk drive 2224, a DVD-ROM drive 2226 (DVD-ROM 2201), an IC card, etc.
  • Various types of processing may be performed on the data on the RAM 2214.
  • the CPU 2212 writes back the processed data to the external recording medium.
  • the CPU 2212 describes various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information retrieval, which are described in various places in the present disclosure and specified by the instruction sequence of the program with respect to the data read from the RAM 2214. Various types of processing may be performed, including / replacement etc., and the result is written back to the RAM 2214. Further, the CPU 2212 may search for information in files, databases, etc. in the recording medium.
  • the CPU 2212 specifies the attribute value of the first attribute.
  • the entry that matches the condition is searched from the plurality of entries, the attribute value of the second attribute stored in the entry is read, and thereby the first attribute that satisfies the predetermined condition is associated.
  • the attribute value of the obtained second attribute may be acquired.
  • the program or software module described above may be stored in a computer-readable medium on the computer 2200 or in the vicinity of the computer 2200.
  • a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium, thereby providing a program to the computer 2200 via the network. To do.
  • DESCRIPTION OF SYMBOLS 8 ... Holding member, 9 ... Sample, 9a, 9b ... Observation position, 10 ... Stage system, 11a ... Opening, 11 ... Stage, 12 ... Sensor, 13 ... Drive device, 20 ... Optical system, 20a, 20b ... Focal plane, 21a, 21b ... objective lens, 23 ... scanner, 24 ... filter, 25 ... cylindrical lens, 26 ... imaging optical system, 30 ... first microscope device, 31 ... first illumination / imaging unit, 40 ... second microscope device, 41 ... second illumination / imaging unit, 50 ... control unit, 51 ... input unit, 60 ... image processing unit, 61 ... image generation unit, 62 ... image output unit, 63a ...

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Optics & Photonics (AREA)
  • Health & Medical Sciences (AREA)
  • Theoretical Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biochemistry (AREA)
  • General Health & Medical Sciences (AREA)
  • Immunology (AREA)
  • Pathology (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Microscoopes, Condenser (AREA)
  • Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)
  • Image Processing (AREA)
  • Investigating Or Analysing Materials By Optical Means (AREA)
  • Image Analysis (AREA)
PCT/JP2017/001289 2017-01-16 2017-01-16 画像処理装置、顕微鏡システム、画像処理方法、及びプログラム Ceased WO2018131172A1 (ja)

Priority Applications (5)

Application Number Priority Date Filing Date Title
PCT/JP2017/001289 WO2018131172A1 (ja) 2017-01-16 2017-01-16 画像処理装置、顕微鏡システム、画像処理方法、及びプログラム
JP2018561782A JP6911873B2 (ja) 2017-01-16 2017-01-16 画像処理装置、顕微鏡システム、画像処理方法、及びプログラム
US16/478,434 US11442262B2 (en) 2017-01-16 2017-01-16 Image processing device, microscope system, image processing method, and program
EP17891716.7A EP3570087A4 (en) 2017-01-16 2017-01-16 IMAGE PROCESSING DEVICE, MICROSCOPE SYSTEM, IMAGE PROCESSING METHOD, AND PROGRAM
JP2021113684A JP2021184264A (ja) 2017-01-16 2021-07-08 画像処理装置、顕微鏡システム、画像処理方法、及びプログラム

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2017/001289 WO2018131172A1 (ja) 2017-01-16 2017-01-16 画像処理装置、顕微鏡システム、画像処理方法、及びプログラム

Publications (1)

Publication Number Publication Date
WO2018131172A1 true WO2018131172A1 (ja) 2018-07-19

Family

ID=62840128

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2017/001289 Ceased WO2018131172A1 (ja) 2017-01-16 2017-01-16 画像処理装置、顕微鏡システム、画像処理方法、及びプログラム

Country Status (4)

Country Link
US (1) US11442262B2 (https=)
EP (1) EP3570087A4 (https=)
JP (2) JP6911873B2 (https=)
WO (1) WO2018131172A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023034038A (ja) * 2021-08-30 2023-03-13 本田技研工業株式会社 解析装置、解析方法、およびプログラム

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022155096A1 (en) * 2021-01-12 2022-07-21 University Of Washington Apparatuses, systems and methods for generating synethettc image sets
DE112021008418T5 (de) * 2021-11-02 2024-08-22 Leica Microsystems Cms Gmbh Verfahren zur Bereitstellung von Positionsinformationen zum Auffinden einer Zielposition in einer mikroskopischen Probe, Verfahren zur Untersuchung und/oder Verarbeitung einer solchen Zielposition und Mittel zur Durchführung dieser Verfahren
JP7811608B2 (ja) * 2023-04-03 2026-02-05 エヴィデント テクノロジー センター ヨーロッパ ゲーエムベーハー 合成顕微鏡画像を記録するための方法、顕微鏡システム、およびコンピュータプログラム製品

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012507756A (ja) 2008-11-03 2012-03-29 カール・ツァイス・マイクロイメージング・ゲゼルシャフト・ミット・ベシュレンクテル・ハフツング 組み合わせ顕微鏡検査法
JP2015057682A (ja) * 2013-08-12 2015-03-26 キヤノン株式会社 画像生成装置および画像生成方法
JP2016062004A (ja) * 2014-09-19 2016-04-25 レーザーテック株式会社 検査装置、及び波面収差補正方法

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3350071B2 (ja) 1991-09-21 2002-11-25 株式会社東芝 磁気共鳴イメージング装置
US7738688B2 (en) * 2000-05-03 2010-06-15 Aperio Technologies, Inc. System and method for viewing virtual slides
US6683316B2 (en) * 2001-08-01 2004-01-27 Aspex, Llc Apparatus for correlating an optical image and a SEM image and method of use thereof
WO2005119575A2 (en) 2004-05-27 2005-12-15 Aperio Technologies, Inc Systems and methods for creating and viewing three dimensional virtual slides
AU2006203027B2 (en) * 2006-07-14 2009-11-19 Canon Kabushiki Kaisha Improved two-dimensional measurement system
JP2013152426A (ja) 2011-12-27 2013-08-08 Canon Inc 画像処理装置、画像処理システム、画像処理方法、およびプログラム
EP2929507A1 (en) 2012-12-07 2015-10-14 Canon Kabushiki Kaisha Image generating apparatus and image generating method
DE102014004249A1 (de) 2014-03-24 2015-09-24 Carl Zeiss Microscopy Gmbh Konfokales Mikroskop mit Aperturkorrelation
US9786057B2 (en) 2014-09-19 2017-10-10 Lasertec Coporation Inspection apparatus, coordinate detection apparatus, coordinate detection method, and wavefront aberration correction method

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012507756A (ja) 2008-11-03 2012-03-29 カール・ツァイス・マイクロイメージング・ゲゼルシャフト・ミット・ベシュレンクテル・ハフツング 組み合わせ顕微鏡検査法
JP2015057682A (ja) * 2013-08-12 2015-03-26 キヤノン株式会社 画像生成装置および画像生成方法
JP2016062004A (ja) * 2014-09-19 2016-04-25 レーザーテック株式会社 検査装置、及び波面収差補正方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3570087A4

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023034038A (ja) * 2021-08-30 2023-03-13 本田技研工業株式会社 解析装置、解析方法、およびプログラム
JP7330240B2 (ja) 2021-08-30 2023-08-21 本田技研工業株式会社 解析装置、解析方法、およびプログラム
US11977009B2 (en) 2021-08-30 2024-05-07 Honda Motor Co., Ltd. Analysis device, analysis method, and storage medium

Also Published As

Publication number Publication date
JP2021184264A (ja) 2021-12-02
JPWO2018131172A1 (ja) 2019-12-12
JP6911873B2 (ja) 2021-07-28
EP3570087A1 (en) 2019-11-20
US20200218054A1 (en) 2020-07-09
US11442262B2 (en) 2022-09-13
EP3570087A4 (en) 2020-08-26

Similar Documents

Publication Publication Date Title
CN110214290B (zh) 显微光谱测量方法和系统
JP2021184264A (ja) 画像処理装置、顕微鏡システム、画像処理方法、及びプログラム
JP6594294B2 (ja) 顕微鏡画像の画像品質評価
JP5996334B2 (ja) 顕微鏡システム、標本画像生成方法及びプログラム
US10001634B2 (en) Method for preparing for and carrying out the acquisition of image stacks of a sample from various orientation angles
CN102362168B (zh) 观察设备
JP5185151B2 (ja) 顕微鏡観察システム
JP5826561B2 (ja) 顕微鏡システム、標本画像生成方法及びプログラム
JP5747690B2 (ja) 顕微鏡装置及び画像形成方法
WO2009146016A1 (en) 3d biplane microscopy
US20140022373A1 (en) Correlative drift correction
JP6790013B2 (ja) 細胞サンプルのサイトメトリー解析方法
JP5940288B2 (ja) 画像処理装置、顕微鏡システム、画像処理方法、及び画像処理プログラム
WO2014006964A1 (ja) 情報処理装置、情報処理方法、プログラム及び顕微鏡システム
JPWO2018131173A1 (ja) 画像処理装置、顕微鏡システム、画像処理方法、及びプログラム
Subramanian Advanced Techniques in UV Microscopy: Integrating Deep Learning for Autofocusing, Whole Slide Imaging Optimization, and Spicule Detection in Bone Marrow Aspirations
Backová Bioinformatical analysis of the complex multidimensional microscopy datasets
Ward et al. AN OVERVIEW OF IMAGE SCANNING MICROSCOPY
Haase OMX-a novel high speed and high resolution microscope and its application to nuclear and chromosomal structure analysis
Koho Bioimage informatics in STED super-resolution microscopy
Gerlich et al. 4D imaging to assay complex dynamics in live specimens
Dyson Ultrastructure Imaging: Imaging and Probing the Structure and Molecular Make-Up of Cells
Amrein Ultrastructure Imaging: Imaging and Probing the Structure and Molecular Make-Up of Cells and Tissues

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17891716

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2018561782

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2017891716

Country of ref document: EP