WO2024176615A1 - 分離画像取得方法、分離画像取得装置、及び分離画像取得プログラム - Google Patents

分離画像取得方法、分離画像取得装置、及び分離画像取得プログラム Download PDF

Info

Publication number
WO2024176615A1
WO2024176615A1 PCT/JP2023/046652 JP2023046652W WO2024176615A1 WO 2024176615 A1 WO2024176615 A1 WO 2024176615A1 JP 2023046652 W JP2023046652 W JP 2023046652W WO 2024176615 A1 WO2024176615 A1 WO 2024176615A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
fluorescence
separation
fluorescent
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2023/046652
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
貴文 樋口
賢一郎 池村
航 加茂
なつみ 加藤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hamamatsu Photonics KK
Original Assignee
Hamamatsu Photonics KK
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hamamatsu Photonics KK filed Critical Hamamatsu Photonics KK
Priority to CN202380094657.6A priority Critical patent/CN120752513A/zh
Priority to EP23924259.7A priority patent/EP4653848A1/en
Priority to JP2024537453A priority patent/JP7566216B1/ja
Publication of WO2024176615A1 publication Critical patent/WO2024176615A1/ja
Priority to JP2024172416A priority patent/JP2024174139A/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G01N21/645—Specially adapted constructive features of fluorimeters
    • G01N21/6456—Spatial resolved fluorescence measurements; Imaging
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G01N21/6428—Measuring fluorescence of fluorescent products of reactions or of fluorochrome labelled reactive substances, e.g. measuring quenching effects, using measuring "optrodes"
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G01N2021/6417—Spectrofluorimetric devices
    • G01N2021/6419—Excitation at two or more wavelengths
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G01N2021/6417—Spectrofluorimetric devices
    • G01N2021/6421—Measuring at two or more wavelengths
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G01N21/6428—Measuring fluorescence of fluorescent products of reactions or of fluorochrome labelled reactive substances, e.g. measuring quenching effects, using measuring "optrodes"
    • G01N2021/6439—Measuring fluorescence of fluorescent products of reactions or of fluorochrome labelled reactive substances, e.g. measuring quenching effects, using measuring "optrodes" with indicators, stains, dyes, tags, labels, marks
    • G01N2021/6441—Measuring fluorescence of fluorescent products of reactions or of fluorochrome labelled reactive substances, e.g. measuring quenching effects, using measuring "optrodes" with indicators, stains, dyes, tags, labels, marks with two or more labels
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G01N21/645—Specially adapted constructive features of fluorimeters
    • G01N2021/6463—Optics
    • G01N2021/6471—Special filters, filter wheel

Definitions

  • One aspect of the embodiment relates to a separation image acquisition method, a separation image acquisition device, and a separation image acquisition program.
  • Non-Patent Document 1 discloses the application of a nonnegative matrix factorization (NMF) method to blind unmix fluorescent images in which fluorescence in multiple wavelength ranges is observed to obtain separate images for each substance in the sample.
  • NMF nonnegative matrix factorization
  • Non-Patent Document 2 discloses a method of unmixing fluorescent images by clustering the fluorescent images, extracting the maximum value of the fluorescent intensity for each clustered pixel group, and generating a separate image based on the maximum value.
  • one aspect of the embodiment has been made in consideration of such problems, and aims to provide a separated image acquisition method, a separated image acquisition device, and a separated image acquisition program that can improve throughput when acquiring separated images by unmixing.
  • the separation image acquisition method includes a selection step of selecting desired image separation information from a storage unit that stores multiple pieces of image separation information for obtaining a fluorescence separation image obtained by separating a fluorescence image based on multiple reference fluorescence images in multiple optical states having different wavelength characteristics; an irradiation step of irradiating a sample with each of multiple wavelengths of excitation light; an acquisition step of acquiring a target fluorescence image in at least one optical state of the multiple optical states for each of multiple fluorescences generated from the sample by each of the multiple wavelengths of excitation light via a fluorescence filter unit having multiple reflection wavelength ranges and multiple transmission wavelength ranges; and a generation step of generating a fluorescence separation image for each of the multiple fluorescences based on the target fluorescence image acquired in the acquisition step and the image separation information selected in the selection step.
  • a separation image acquisition device includes a storage unit that stores image separation information for obtaining a fluorescence separation image obtained by separating a fluorescence image based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics, an irradiation device that irradiates a sample with each of excitation light having a plurality of wavelength distributions, an image acquisition device that acquires a target fluorescence image in at least one of the plurality of optical states for each of a plurality of fluorescences generated from the sample by each of the excitation light having a plurality of wavelengths via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges, and an image processing device that processes the fluorescence image, and the image processing device selects desired image separation information from the plurality of image separation information stored in the storage unit, and generates a fluorescence separation image for each of the plurality of fluorescences based on the target
  • a separation image acquisition program is a separation image acquisition program for generating a fluorescence separation image based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics, which are acquired through a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges for each of a plurality of fluorescences generated from a sample by each of the excitation light of a plurality of wavelengths by irradiating the sample with each of the excitation light of a plurality of wavelength distributions, and causes a computer to execute a storage process for storing a plurality of pieces of image separation information for obtaining a fluorescence separation image obtained by separating the fluorescence, which is acquired based on the plurality of reference fluorescence images, a selection process for selecting desired image separation information from the plurality of stored image separation information, and a generation process for generating a fluorescence separation image based on a target fluorescence image acquired in
  • desired image separation information is selected from a plurality of image separation information acquired based on a plurality of reference fluorescent images, and a target fluorescent image is acquired in which a fluorescent image of the sample is captured in the same optical state as one of the plurality of reference fluorescent images using excitation light with a different wavelength distribution through a fluorescent filter unit, and a fluorescent separation image is generated using the acquired target fluorescent image and the selected image separation information.
  • FIG. 1 is a schematic configuration diagram of a fluorescent dye image acquisition system 1 according to an embodiment.
  • FIG. 2 is a perspective view showing a configuration of an image acquisition device 3 in FIG. 1 .
  • 2 is a block diagram showing an example of a hardware configuration of the image processing device 5 in FIG. 1 .
  • FIG. 2 is a block diagram showing a functional configuration of the image processing device 5 of FIG. 1 .
  • FIG. 5 is a diagram showing an image of a pixel group clustered by a first clustering function of the matrix acquisition unit 203 in FIG. 4 .
  • 1 is a graph showing wavelength characteristics of the absorptance of excitation light for multiple fluorescent dyes contained in sample S. 5 is a graph showing the distribution of centroid fluorescence wavelengths identified by the matrix acquisition unit 203 of FIG.
  • FIG. 5 is a diagram showing an image of a pixel group clustered by the second clustering function of the matrix acquisition unit 203 in FIG. 4 .
  • 5 is a diagram showing an image of matrix data Y' and fluorescent dye matrix data X' regenerated by the matrix acquisition unit 203 in FIG. 4.
  • 4 is a flowchart showing the procedure of a separated image acquisition method according to the embodiment.
  • 1 is a diagram showing an example of a fluorescent dye image generated based on a target fluorescent dye image by the fluorescent dye image acquisition system 1 according to the present embodiment.
  • FIG. FIG. 13 is a diagram showing an example of a fluorescent dye image generated based on a target fluorescent image without mixing matrix correction by the fluorescent dye image acquisition system 1 according to the present embodiment.
  • FIG. 13 is a block diagram showing a functional configuration of an image processing device 5A according to a modified example.
  • the fluorescent dye image acquisition system 1 is a schematic diagram of a fluorescent dye image acquisition system 1, which is a separation image acquisition device according to an embodiment.
  • the fluorescent dye image acquisition system 1 is a device for generating fluorescent dye images (fluorescence separation images) for identifying the distribution of fluorescent dyes in a sample such as a biological tissue to be observed.
  • the images generated by the fluorescent dye image acquisition system 1 are used for purposes such as drug development and treatment method research through the analysis of the images. For this reason, the fluorescent dye image acquisition system 1 is required to generate images that can quantitatively identify the distribution of many substances (fluorescent dyes) contained in the sample with high throughput.
  • the fluorescent dye image acquisition system 1 includes an image acquisition device 3 that irradiates the sample S with excitation light and acquires an image of the fluorescence generated in response to the irradiation, and an image processing device 5 that processes the image acquired by the image acquisition device 3.
  • the image acquisition device 3 and the image processing device 5 may be configured to be capable of transmitting and receiving image data between them using wired or wireless communication, or may be configured to be capable of inputting and outputting image data via a recording medium.
  • FIG. 2 is a perspective view showing the configuration of the image acquisition device 3 in FIG. 1.
  • the optical path of the excitation light is indicated by a dotted line with an arrow
  • the optical path of the fluorescence is indicated by a solid line with an arrow.
  • the image acquisition device 3 is composed of an excitation light source (illumination device) 7, a light source side filter set 9a, a dichroic mirror 11, a camera side filter set (fluorescence filter section) 9b, a wavelength information acquisition optical system (optical filter) 13, and a camera (image acquisition device) 15.
  • the excitation light source 7 is a light source capable of switching between and irradiating excitation light of multiple wavelength bands (wavelength distributions), and is, for example, an LED (Light Emitting Diode) light source, a light source consisting of multiple monochromatic laser light sources, or a light source combining a white light source and a wavelength selection optical element.
  • the light source side filter set 9a is a multi-bandpass filter that is provided on the optical path of the excitation light from the excitation light source 7 and has the property of transmitting light of multiple predetermined wavelength bands.
  • the transmission wavelength band of this light source side filter set 9a is set according to the multiple wavelength bands of the excitation light that can be used.
  • the dichroic mirror 11 is provided between the light source side filter set 9a and the sample S, and is an optical member that reflects the excitation light toward the sample S and transmits the fluorescence emitted from the sample S accordingly.
  • the camera side filter set 9b is a multi-bandpass filter that is provided on the optical path of the fluorescence transmitted by the dichroic mirror 11 and has the property of transmitting light of multiple predetermined wavelength bands.
  • the transmission wavelength band of this camera side filter set 9b is set according to the wavelength band of the fluorescence generated by the fluorescent dye that can be contained in the sample S to be observed.
  • the camera-side filter set 9b has, as its wavelength characteristics, a transmission wavelength band (transmission wavelength range) corresponding to a plurality of fluorescence wavelength bands, and a reflection wavelength band (reflection wavelength range) between these transmission wavelength bands.
  • the wavelength information acquisition optical system 13 is detachably supported on the optical path of the fluorescence transmitted by the camera-side filter set 9b, and is an optical system for acquiring wavelength information of the fluorescence. That is, the wavelength information acquisition optical system 13 is provided so as to be switchable between two states: a state in which it is arranged on the optical path of the fluorescence from the sample S (first optical state) and a state in which it is removed from the optical path of the fluorescence (second optical state). Note that the wavelength information acquisition optical system 13 only needs to realize a plurality of optical states with different wavelength characteristics, and may be, for example, a fluorescence filter set having two or more fluorescence filters with different wavelength characteristics.
  • the wavelength information acquisition optical system 13 transmits the fluorescence generated in the sample S and transmitted through the dichroic mirror 11 and the camera-side filter set 9b toward the camera 15 with a predetermined wavelength characteristic.
  • the wavelength information acquisition optical system 13 allows the fluorescence transmitted through the dichroic mirror 11 and the camera-side filter set 9b to be incident on the camera 15 in its original optical state (without transmitting through the wavelength information acquisition optical system 13).
  • a dichroic mirror also called a gradient filter
  • having wavelength characteristics of transmittance such that the transmittance increases linearly as the wavelength increases is used as the wavelength information acquisition optical system 13.
  • a dichroic mirror having wavelength characteristics of transmittance such that the transmittance decreases linearly as the wavelength decreases may be used as the wavelength information acquisition optical system 13.
  • a wavelength information acquisition optical system 13 using such a dichroic mirror can cause fluorescence to enter the camera 15 with two different wavelength transmission characteristics.
  • the camera 15 is an imaging device that captures a two-dimensional image composed of N pixels (N is an integer equal to or greater than 2, for example, 2048 x 2048), and is a camera that captures the fluorescence that has passed through the wavelength information acquisition optical system 13 to obtain a first fluorescence image when the wavelength information acquisition optical system 13 is switched onto the optical path of the fluorescence (in the first optical state).
  • N is an integer equal to or greater than 2, for example, 2048 x 2048
  • the camera 15 captures the fluorescence that does not pass through the wavelength information acquisition optical system 13 to obtain a second fluorescence image.
  • the camera 15 captures a first fluorescence image and a second fluorescence image for each of the multiple fluorescences generated from the sample S by each of the multiple wavelength bands of excitation light irradiated by the excitation light source 7.
  • the camera 15 outputs the captured first fluorescence image and second fluorescence image to the image processing device 5 using communication or via a recording medium.
  • Figure 3 is a block diagram showing an example of the hardware configuration of the image processing device 5
  • Figure 4 is a block diagram showing the functional configuration of the image processing device 5.
  • the image processing device 5 is physically a computer or the like including a processor, a CPU (Central Processing Unit) 101, a recording medium, a RAM (Random Access Memory) 102 or a ROM (Read Only Memory) 103, a communication module 104, and an input/output module 106, each of which is electrically connected.
  • the image processing device 5 may include a display, keyboard, mouse, touch panel display, etc. as input/output devices, or may include a data recording device such as a hard disk drive or semiconductor memory.
  • the image processing device 5 may also be composed of multiple computers.
  • the image processing device 5 includes, as functional components, an information search unit (selection unit) 201, an image acquisition unit 202, a matrix acquisition unit 203, a matrix correction unit 204, an image generation unit 205, and an image separation information storage unit (storage unit) 206.
  • the information search unit 201 and the image separation information storage unit 206 may be external devices connected to the image processing device 5.
  • Each functional unit of the image processing device 5 shown in FIG. 4 is realized by loading a program (a separated image acquisition program according to the embodiment) onto hardware such as the CPU 101 and the RAM 102, and operating the communication module 104 and the input/output module 106 under the control of the CPU 101, and reading and writing data in the RAM 102.
  • the CPU 101 of the image processing device 5 executes this computer program to operate each functional unit of FIG. 4, and sequentially executes processing corresponding to a separated image acquisition method described later.
  • the CPU 101 may be a standalone piece of hardware, or may be implemented in a programmable logic such as an FPGA, like a software processor.
  • the RAM and ROM may also be standalone pieces of hardware, or may be built into a programmable logic such as an FPGA. All of the various data required to execute this computer program, and all of the various data generated by executing this computer program, are stored in built-in memories such as the ROM 103 and RAM 102, or in storage media such as a hard disk drive.
  • the functions of the functional components of the image processing device 5 are described in detail below.
  • the information search unit 201 searches for (selects) a desired mixing matrix from the data stored in the image separation information storage unit 206 based on the image acquisition conditions when the first or second fluorescent image of the sample S is acquired, information (estimation processing information) related to the estimation processing of the mixing matrix (image separation information) based on the first and second fluorescent images, and storage setting information when the mixing matrix is stored in the image separation information storage unit 206.
  • the information search unit 201 may set the image acquisition conditions, estimation processing information, storage setting information, etc.
  • the search based on information input by the user via the input/output module 106 of the image processing device 5, based on information transmitted from an external device such as the image acquisition device 3, or based on information referenced from the RAM 102 or ROM 103 of the image processing device 5.
  • the above image acquisition conditions include the type of sample S, barcode information attached to sample S, type of fluorescent dye contained in sample S, date and time of staining of sample S, date and time of measurement of sample S, type of excitation light, intensity of each excitation light, type or number of filters (presence or absence of gradient filter, etc.), type of camera, type or magnification of objective lens, exposure time of each excitation light, environmental temperature, data of the acquired fluorescent image itself, etc.
  • the above estimation processing information includes version information of the software used in the estimation processing, algorithm, parameters or options used in the estimation processing, time required for the estimation processing, etc.
  • the above saved setting information includes the saved user name, project name, assay name, mixing matrix name, tag information (freely added keywords, SNS hashtags), etc.
  • the mixing matrix acquired by the matrix acquisition unit 203 described later is stored in association with the image acquisition conditions at the time of acquisition of the multiple fluorescent images (e.g., the first fluorescent image and the second fluorescent image) on which the mixing matrix was based, the estimated processing information related to the estimated processing used at the time of acquisition of the mixing matrix, and the storage setting information at the time of saving the mixing matrix.
  • the information search unit 201 searches (selects) a mixing matrix associated with information that matches or corresponds (similar to) a search key including the image acquisition conditions, estimated processing information, or storage setting information set by a user, an external device, etc. Then, the information search unit 201 transfers the search results of the mixing matrix to the image acquisition unit 202 and the matrix acquisition unit 203.
  • the information search unit 201 searches for a mixing matrix using at least one item of the above-mentioned image acquisition conditions, estimated processing information, and storage setting information as a search key.
  • the information search unit 201 searches for a mixing matrix using a search key that includes at least one item of the image acquisition conditions.
  • the image acquisition unit 202 acquires a first fluorescence image (first reference fluorescence image) and a second fluorescence image (second reference fluorescence image) from the image acquisition device 3 in order to acquire a new mixing matrix.
  • the image acquisition unit 202 acquires pre-specified C (C is an integer equal to or greater than 2) first reference fluorescence images of the sample S from the image acquisition device 3.
  • C first reference fluorescence images are fluorescence images composed of N pixels that are generated by irradiating the sample S with excitation light of C wavelength bands in the first optical state and capturing the fluorescence generated from the sample S in response.
  • the number C of first reference fluorescence images to be acquired (the number C of wavelength bands of excitation light irradiated to the sample S) is specified in advance to be equal to or greater than the maximum number of fluorescent dyes that can be contained in the sample S.
  • the image acquisition unit 202 acquires C second reference fluorescence images in the second optical state.
  • the image acquisition unit 202 does not acquire a fluorescence image in the first optical state, but acquires C fluorescence images of the sample S in the second optical state as C target fluorescence images for acquiring a fluorescent dye image, in the same manner as described above.
  • the matrix acquisition unit 203 acquires a mixing matrix based on the first reference fluorescent image and the second reference fluorescent image of set C acquired by the image acquisition unit 202.
  • the function of acquiring this mixing matrix includes a clustering function and a statistical value calculation function.
  • the clustering function of the matrix acquisition unit 203 is explained below.
  • the matrix acquisition unit 203 estimates the centroid fluorescence wavelength indicating the center of gravity of the fluorescence wavelength distribution by calculating the ratio of the fluorescence intensity (brightness value) of one fluorescence image to the fluorescence intensity of the other fluorescence image for each of the first and second reference fluorescence images of set C. At this time, the matrix acquisition unit 203 calculates the average value of the fluorescence intensity of one fluorescence image to the average value of the fluorescence intensity of the other fluorescence image for the pixel group clustered by the clustering process described below, and calculates the ratio of these average values. The matrix acquisition unit 203 acquires the estimated centroid fluorescence wavelength as wavelength information related to the fluorescence wavelength.
  • the matrix acquisition unit 203 performs clustering on N pixels constituting the C first reference fluorescence images and the C second reference fluorescence images, based on the C first reference fluorescence images and the C second reference fluorescence images acquired by the image acquisition unit 202, and the acquired wavelength information. Prior to the clustering process, the matrix acquisition unit 203 generates matrix data Y in which the fluorescence intensity values of the N pixels constituting each of the C first reference fluorescence images and the C second reference fluorescence images are arranged in parallel in one dimension.
  • the matrix acquisition unit 203 has a function (first clustering function) of clustering N pixels of the second reference fluorescence image into C pixel groups based on distribution information of excitation light of C wavelength bands of fluorescence intensity.
  • the matrix acquisition unit 203 clusters pixels having the same wavelength band of excitation light with the highest fluorescence intensity into the same pixel group.
  • FIG. 5 shows an image of pixel groups clustered by the first clustering function of the matrix acquisition unit 203
  • FIG. 6 shows wavelength characteristics of excitation light absorption rates of multiple fluorescent dyes contained in the sample S. As shown in FIG.
  • the matrix acquisition unit 203 clusters N pixels contained in the six second reference fluorescence images GC 1 to GC 6 into six pixel groups PGr 1 to PGr 6 .
  • different types of fluorescent dyes generally have wavelength characteristics of different absorptances
  • the three types of fluorescent dyes C1 , C2 , and C3 also have wavelength characteristics CW1 , CW2 , and CW3 with different peak wavelengths.
  • the fluorescent dye with the highest absorptance is determined to be one of the three fluorescent dyes C1 , C2 , and C3 .
  • the fluorescent dye C1 has the highest absorptance of excitation light in the wavelength band EW1
  • the fluorescent dye C1 has the highest absorptance of excitation light in the wavelength band EW2
  • the fluorescent dye C2 has the highest absorptance of excitation light in the wavelength band EW3 .
  • the matrix acquisition unit 203 can cluster N pixels into pixel groups in ranges where the same fluorescent dye is distributed by the first clustering function.
  • the six pixel groups PGr 1 to PGr 6 clustered by the first clustering function do not have a one-to-one correspondence with the three types of fluorescent dyes C 1 , C 2 , and C 3 .
  • the matrix acquisition unit 203 has a function (second clustering function) of further clustering the C pixel groups clustered by the first clustering function into L pixel groups (L is an integer between 2 and N-1) based on the wavelength information.
  • the number L of pixel groups to be clustered is set in advance as a parameter stored in the image processing device 5, for example, corresponding to the number of types of fluorescent dyes that may be present in the sample S.
  • the number L of pixel groups may be determined according to the type of excitation light or the number C of wavelength distributions of the excitation light, or may be determined independently of the type of excitation light or the number C of wavelength distributions of the excitation light.
  • the matrix acquisition unit 203 identifies the centroid fluorescence wavelength estimated for the wavelength band of the excitation light corresponding to that pixel group. More specifically, the matrix acquisition unit 203 acquires wavelength information for the pixel groups clustered as having the highest absorptance in a certain wavelength band, and identifies the centroid fluorescence wavelength based on the acquired wavelength information. At this time, wavelength information is acquired using the average value of the fluorescence intensity in the pixel groups of a pair of the first and second reference fluorescence images obtained corresponding to that wavelength band.
  • the matrix acquisition unit 203 clusters the C pixel groups into L pixel groups by determining the distance (closeness of values) between the centroid fluorescence wavelengths identified for each of the C pixel groups. Then, the matrix acquisition unit 203 divides and regenerates matrix data Y, in which the fluorescence intensity values of the pixels of the C first reference fluorescence image and the fluorescence intensity values of the pixels of the C second reference fluorescence image are arranged in parallel in one dimension, into cluster matrices for each of the L pixel groups.
  • Fig. 7 shows the distribution of centroid fluorescence wavelengths identified by the matrix acquisition unit 203
  • Fig. 8 shows an image of pixel groups clustered by the second clustering function by the matrix acquisition unit 203.
  • Fig. 7 and Fig. 8 show an image of pixel groups clustered by the second clustering function by the matrix acquisition unit 203.
  • centroid fluorescence wavelengths FW 1 to FW 6 are identified for each of the six pixel groups PGr 1 to PGr 6 clustered by the first clustering function, and pixel groups PGr 1 and PGr 2 , whose centroid fluorescence wavelengths are close to each other, are clustered into a new pixel group PGr 01 , and similarly, pixel groups PGr 3 and PGr 4 are clustered into pixel group PGr 02 , and pixel groups PGr 5 and PGr 6 are clustered into pixel group PGr 03.
  • the division number L by the second clustering function is set to be equal to or less than the number C of fluorescent images (the number C of wavelength bands of excitation light).
  • the matrix acquisition unit 203 obtains a mixing matrix A for generating K fluorescent dye images showing the respective distributions of K fluorescent dyes (K is an integer between 2 and C) from the C first reference fluorescent images and the C second reference fluorescent images, based on the L cluster matrices obtained for the sample S.
  • NMF non-negative matrix factorization
  • the matrix acquisition unit 203 regenerates matrix data Y' by compressing the matrix data Y generated by the clustering function in units of pixel groups clustered by the clustering function.
  • the matrix acquisition unit 203 calculates statistical values for each pixel group of the clustered cluster matrix for the fluorescence intensity of each row of the matrix data Y, and compresses the pixel group of each row into one pixel having the calculated statistical value.
  • the matrix acquisition unit 203 regenerates matrix data Y', which is matrix data of C x 2 rows and L columns.
  • the matrix acquisition unit 203 may calculate, as the statistical value, an average value based on the integrated value of the fluorescence intensity, may calculate the most frequent value of the fluorescence intensity, or may calculate the median value of the fluorescence intensity.
  • Fig. 9 shows an image of matrix data Y' regenerated by matrix acquisition unit 203 and its corresponding fluorescent dye matrix data X'. One square shown in Fig. 9 represents one element of the matrix data.
  • the fluorescent dye matrix data X and matrix data Y divided into three pixel groups PGr 01 to PGr 03 are compressed into three columns of fluorescent dye matrix data X' and matrix data Y', with the statistical values for each of pixel groups PGr 01 to PGr 03 used as representative values.
  • the matrix acquisition unit 203 derives the mixing matrix A based on the matrix data Y' as follows. That is, the matrix acquisition unit 203 sets an initial value to the mixing matrix A, calculates the following loss function (loss value) Los while sequentially changing the value of the mixing matrix A, and derives the mixing matrix A that reduces the value of the loss function Los. Note that a regularization term such as the L1 norm ⁇
  • j is a parameter indicating the row position of the matrix data (corresponding to the wavelength band of the excitation light for each of the two fluorescent images)
  • the matrix subscript 1j indicates the matrix data of the jth row of the first cluster matrix
  • the matrix subscript 2j indicates the matrix data of the jth row of the second cluster matrix
  • the matrix subscript 3j indicates the matrix data of the jth row of the third cluster matrix
  • the parameters a, b, and c indicate the average values of the statistics of each column of the matrix data Y'.
  • the matrix acquisition unit 203 calculates a loss function for each of the L cluster matrices divided by the clustering function by referring to the statistical values of the C ⁇ 2 matrix data Y', calculates the loss function Los based on the sum of the L loss functions, and obtains the mixing matrix A based on the loss function Los.
  • the matrix acquisition unit 203 corrects the loss function calculated for each of the L cluster matrices by dividing it by the average values a, b, and c of the statistical values of the C ⁇ 2 matrix data Y', and then obtains the loss function Los by calculating the sum of the corrected loss functions.
  • the matrix acquisition unit 203 may calculate the loss function for each of the L cluster matrices by correcting it by dividing the row components of the difference value Y'-AX' for each wavelength band of the excitation light by the C ⁇ 2 statistical values corresponding to each wavelength band of the excitation light.
  • the above formula can also be generalized as follows. That is, the matrix acquisition unit 203 derives the mixing matrix A and the fluorescent dye matrix data X' based on the matrix data Y' as follows. That is, the matrix acquisition unit 203 sets initial values to the mixing matrix A and the fluorescent dye matrix data X', calculates a loss function (loss value) Los using the following formula while sequentially changing the values of the mixing matrix A and the fluorescent dye matrix data X', and derives the mixing matrix A and the fluorescent dye matrix data X' that reduce the value of the loss function Los.
  • a regularization term such as the L1 norm ⁇
  • the calculation may be performed with a constraint that the mixing matrix A and the fluorescent dye matrix data X' are non-negative values.
  • j is a parameter indicating the row position of the matrix data (corresponding to the wavelength band of the excitation light)
  • i is a parameter indicating the column position of the matrix data (corresponding to the i-th cluster).
  • w ij represents the weight of each element of the matrix data, and may be calculated from the value of each element or its standard deviation. It is also possible to set all w ij to the same value and not consider the weight of each element.
  • the matrix acquisition unit 203 calculates a loss function for each of the L cluster matrices divided by the clustering function by referring to the statistical values of the C ⁇ 2 matrix data Y′, calculates a loss function Los based on the sum of the L loss functions Los i , and obtains a mixing matrix A based on the loss function Los.
  • the matrix acquisition unit 203 may calculate the loss function Los i for each of the L cluster matrices by correcting the row components for each wavelength band of the excitation light of the difference value Y′-AX′ by dividing them by the C ⁇ 2 statistical values corresponding to each wavelength band of the excitation light.
  • the matrix acquisition unit 203 stores the mixing matrix (image separation information for obtaining a fluorescent dye image by separating the fluorescence) acquired using the above-mentioned clustering function and statistical value calculation function in the image separation information storage unit 206 by linking it to the image acquisition conditions, estimated processing information, and save setting information.
  • the image acquisition conditions, estimated processing information, and save setting information linked to the mixing matrix are information corresponding to the acquisition of the first or second fluorescent image, information corresponding to the estimation of the mixing matrix, or information corresponding to the saving of the mixing matrix.
  • the matrix correction unit 204 acquires a corrected mixing matrix based on the mixing matrix and the C target fluorescent images acquired by the image acquisition unit 202. That is, the matrix correction unit 204 generates matrix data Y2 in which the values of the fluorescent intensities of the pixels of the C target fluorescent images are arranged in parallel in one dimension. The matrix correction unit 204 also extracts matrix data of C rows and K columns corresponding to the rows of the second fluorescent image as a mixing matrix A0 from the elements of the searched mixing matrix A. Furthermore, the matrix correction unit 204 acquires a corrected mixing matrix (corrected image separation information) A2 using the matrix data Y2 and the mixing matrix A0 .
  • the mixing matrix A2 is matrix data of C rows and K columns including only the rows corresponding to the rows of the second fluorescent image among the elements of the mixing matrix A.
  • the matrix correction unit 204 regenerates matrix data Y2 ' by clustering the matrix data Y2 using a clustering function and compressing it in units of pixel groups. This clustering is performed in the same manner as the process in the matrix acquisition unit 203, for example, based on the distribution of fluorescence intensity for each of C wavelength bands.
  • a term such as L1 norm ⁇ A 2 ⁇ 1 may be added to impose a sparsity constraint on the matrix A 2 .
  • values that are expected to be correct among the elements of the mixing matrix A 0 may be fixed, and the values of the other elements may be corrected.
  • a loss function such as Kullback-Leibler divergence or Itakura-Saito divergence may be used instead of the Euclidean distance.
  • the image generation unit 205 acquires K fluorescent dye images by unmixing the C second reference fluorescent images obtained of the sample S to be observed using the mixing matrix A newly acquired by the matrix acquisition unit 203. Specifically, the image generation unit 205 extracts mixing matrix A2 from mixing matrix A, and calculates fluorescent dye matrix data X by applying the inverse matrix A2-1 of mixing matrix A2 to matrix data Y2 generated based on the C second reference fluorescent images.
  • the image generation unit 205 calculates fluorescent dye matrix data X by unmixing the C target fluorescent images obtained of the sample S to be observed in the same manner as described above, using the mixing matrix A2 corrected by the matrix correction unit 204.
  • the image generating unit 205 regenerates K fluorescent dye images from the fluorescent dye matrix data X, and outputs the regenerated K fluorescent dye images.
  • the output destination at this time may be an output device of the image processing device 5, such as a display or a touch panel display, or may be an external device connected to the image processing device so as to be capable of data communication.
  • Figure 10 is a flowchart showing the procedure for the observation process using the fluorescent dye image acquisition system 1.
  • the information search unit 201 of the image processing device 5 searches for the mixing matrix A from among the data stored in the image separation information storage unit 206 using the set image acquisition conditions, etc. as a search key (step S1). If the mixing matrix is not found as a result of the search (step S2; No), the image acquisition unit 202 of the image processing device 5 acquires C first reference fluorescent images and C second reference fluorescent images that are the results of the fluorescent observation of the sample S (step S3).
  • the matrix acquisition unit 203 of the image processing device 5 acquires a mixing matrix A based on the C first reference fluorescent images and the C second reference fluorescent images (step S4).
  • the acquired mixing matrix A is linked to the image acquisition conditions and the like and stored in the image separation information storage unit 206 by the matrix acquisition unit 203 (step S5; storage step).
  • the image generation unit 205 acquires fluorescent dye matrix data X based on matrix data Y2 in which N pixels of the C second reference fluorescent images are arranged in parallel and matrix A2 extracted from the mixing matrix A (step S6).
  • the image acquisition unit 202 of the image processing device 5 acquires C target fluorescent images that are the result of the fluorescent observation of the sample S (step S7; acquisition step).
  • the matrix correction unit 204 of the image processing device 5 extracts (selects) the mixing matrix A from the searched mixing matrix A (step S8).
  • the matrix correction unit 204 acquires a mixing matrix A 2 corrected using the matrix data Y 2 generated from the C target fluorescent images and the mixing matrix A 0 (step S9).
  • the image generation unit 205 acquires fluorescent dye matrix data X based on the matrix data Y 2 and the mixing matrix A 2 (step S10; generation step).
  • the corrected mixing matrix A 2 may be stored in the image separation information storage unit 206.
  • the image generation unit 205 of the image processing device 5 reproduces K fluorescent dye images from the fluorescent dye matrix data X acquired in step S6 or step S10, and outputs these K fluorescent dye images (step S11; generation step). This completes the observation process for the sample S.
  • a desired mixing matrix is selected from a plurality of mixing matrices for obtaining a fluorescent dye image acquired based on C first reference fluorescent images and C second reference fluorescent images, C target fluorescent images captured in the same optical state as either the first reference fluorescent image or the second reference fluorescent image are acquired, and a fluorescent dye image is generated using the acquired C target fluorescent images and the selected mixing matrix.
  • a mixing matrix that corresponds to the image acquisition conditions is selected from among the mixing matrices generated based on previously acquired fluorescent images, and unmixing is performed using the selected mixing matrix.
  • a mixing matrix suitable for the target fluorescent image is selected, which improves the separation accuracy of the acquired fluorescent dye image.
  • a corrected mixing matrix is obtained based on the target fluorescent image and the searched mixing matrix.
  • the previously obtained mixing matrix is corrected using the target fluorescent image, so unmixing appropriate for the target fluorescent image is performed, and the separation accuracy of the obtained fluorescent dye image can be further improved.
  • a fluorescent dye image is generated using the corrected mixing matrix and the target fluorescent image.
  • unmixing appropriate to the target fluorescent image is performed, and the separation accuracy of the acquired fluorescent dye image can be further improved.
  • a mixing matrix generated in the past may contain errors due to the influence of noise in the past fluorescent image used to estimate the mixing matrix.
  • the mixing matrix is used as is for the target fluorescent image, there is a tendency for errors in the fluorescent dye image generated from the target fluorescent image containing noise of a different nature to become large.
  • a mixing matrix corrected based on the target fluorescent image is used, and errors in the generated fluorescent dye image can be suppressed.
  • FIG. 11 shows an example of a fluorescent dye image generated based on a target fluorescent image by the fluorescent dye image acquisition system 1 according to this embodiment.
  • FIG. 12 shows an example of a fluorescent dye image generated based on a target fluorescent image by the fluorescent dye image acquisition system 1 according to this embodiment without correcting the mixing matrix.
  • the same fluorescent dye images are shown in corresponding positions in FIGS. 11 and 12.
  • this embodiment significantly improves the image quality of some fluorescent dye images (particularly the fluorescent dye image in the upper right corner) by correcting the mixing matrix.
  • the first fluorescence image and the second fluorescence image are acquired by attaching and detaching the wavelength information acquisition optical system 13.
  • the image acquisition device 3 may be configured to have two cameras and acquire fluorescence images in two or more different optical states by detecting the fluorescence that transmits through the wavelength information acquisition optical system 13 and the fluorescence that reflects off the wavelength information acquisition optical system 13 with each camera.
  • the image acquisition device 3 may be provided with a color camera such as an RGB color camera, and acquire a reference fluorescence image and a target fluorescence image based on image data for each wavelength characteristic output from the color camera. Even in such a case, it is configured to acquire fluorescence images in two or more different optical states.
  • the image acquisition device 3 may be provided with a hyperspectral camera, and acquire fluorescence images based on image data for each wavelength characteristic output from the hyperspectral camera. Even in such a case, it is configured to acquire fluorescence images in two or more different optical states.
  • the matrix correction unit 204 of the image processing device 5 also functions to correct the mixing matrix A when the information search unit 201 finds a mixing matrix. In contrast, the matrix correction unit 204 of the image processing device 5 may always function to correct the mixing matrix A, regardless of the search results of the information search unit 201 for a mixing matrix.
  • FIG. 13 shows the functional configuration of an image processing device 5A according to a modified example.
  • Image processing device 5A differs from image processing device 5 in that, instead of matrix correction unit 204 and image generation unit 205, image correction unit 204A and image output unit 205A are provided, and the function of correcting fluorescent dye matrix data X without correcting the mixing matrix is provided.
  • the image correction unit 204A generates fluorescent dye matrix data X by unmixing the C target fluorescent images directly using the mixing matrix A0 extracted from the mixing matrix A searched for by the information search unit 201.
  • the image correction unit 204A then generates fluorescent dye matrix data XC corrected using the matrix data Y2 and the mixing matrix A0 .
  • the acquisition of the corrected fluorescent dye matrix data XC is executed as follows.
  • a formula to which a regularization term is added such as the following formula, may be used.
  • This regularization term is a regularization term of Total Variation Loss, and is a term obtained by calculating the L1 norm of each of the differentials in the x and y directions with the matrix data X as an image.
  • an equation in which a regularization term related to the mixing matrix A is combined with the above equation may be used.
  • the image output section 205A reconstructs K fluorescent dye images from the fluorescent dye matrix data XC corrected by the image correction section 204A, and outputs the reconstructed K fluorescent dye images.
  • the above modified example makes it possible to obtain a fluorescent dye image that is corrected based on a previously acquired mixing matrix and a target fluorescent image, thereby further improving the separation accuracy of the acquired fluorescent dye image.
  • the image correction unit 204A of the image processing device 5A may obtain the corrected fluorescent dye matrix data X by one of the following two methods.
  • the image correction unit 204A generates fluorescent dye matrix data X by unmixing the C target fluorescent images using the mixing matrix A0 extracted from the mixing matrix A searched by the information search unit 201 as is.
  • the image correction unit 204A then inputs the generated fluorescent dye matrix data X to the trained inference model and acquires the output of the trained inference model as corrected fluorescent dye matrix data XC .
  • the image correction unit 204A can construct the trained inference model by using as training data a combination of the fluorescent dye matrix data X derived by unmixing in the matrix acquisition unit 203 and the fluorescent dye matrix data X obtained by unmixing the C target fluorescent images using the mixing matrix A0 as is.
  • the generated fluorescent dye image can be input to an inference model, and a corrected fluorescent dye image can be obtained based on the output of the inference model.
  • the separation accuracy of the acquired fluorescent dye image can be further improved.
  • the image correction unit 204A inputs matrix data Y2 based on C target fluorescent images and mixing matrix A0 extracted from mixing matrix A searched for by the information search unit 201 to the trained inference model, and acquires the output of the trained inference model as corrected fluorescent dye matrix data XC .
  • the image correction unit 204A can construct the trained inference model by using, as training data, a combination of fluorescent dye matrix data X derived by unmixing in the matrix acquisition unit 203, mixing matrix A0 extracted from the searched mixing matrix A, and matrix data Y2 based on C target fluorescent images.
  • the target fluorescent image and the mixing matrix can be input to an inference model, and a corrected fluorescent dye image can be obtained based on the output of the inference model.
  • the separation accuracy of the acquired fluorescent dye image can be further improved.
  • the image separation information is stored in association with the image acquisition conditions when the multiple reference fluorescent images were acquired, and that in the selection step, the image separation information linked to the image acquisition conditions corresponding to the image acquisition conditions when the target fluorescent image was acquired is selected.
  • the storage unit stores the image separation information in association with the image acquisition conditions when the multiple reference fluorescent images were acquired, and that the image processing device selects the image separation information linked to the image acquisition conditions corresponding to the image acquisition conditions when the target fluorescent image was acquired.
  • corrected image separation information is obtained, which is image separation information corrected based on the target fluorescent image and the selected image separation information.
  • the image processing device in generating the fluorescent separation image, obtains corrected image separation information, which is image separation information corrected based on the target fluorescent image and the selected image separation information. In this way, since the image separation information obtained in the past is corrected using the target fluorescent image, unmixing suitable for the target fluorescent image is performed, and the separation accuracy of the obtained separated image can be further improved.
  • the fluorescence separation image is generated using the corrected image separation information and the target fluorescence image.
  • the image processing device generates the fluorescence separation image using the corrected image separation information and the target fluorescence image in generating the fluorescence separation image. In this case, unmixing suitable for the target fluorescence image is performed, and the separation accuracy of the obtained separation image can be further improved.
  • the fluorescence separation image is corrected based on the target fluorescence image and the image separation information.
  • the image processing device corrects the fluorescence separation image based on the target fluorescence image and the image separation information in generating the fluorescence separation image. In this case as well, it is possible to obtain a fluorescence separation image that has been corrected based on the image separation information and the target fluorescence image previously obtained, and it is possible to further improve the separation accuracy of the obtained separation image.
  • the fluorescence separation image generated based on the target fluorescence image and the image separation information is input to a trained inference model, and the output of the trained inference model is obtained as a corrected fluorescence separation image.
  • the image processing device inputs the fluorescence separation image generated based on the target fluorescence image and the image separation information to a trained inference model, and the output of the trained inference model is obtained as a corrected fluorescence separation image.
  • the generated fluorescence separation image can be input to the inference model, and a fluorescence separation image corrected based on the output of the inference model can be obtained. As a result, the separation accuracy of the obtained separation image can be further improved.
  • the target fluorescent image and image separation information are input to a trained inference model, and the output of the trained inference model is obtained as a corrected fluorescent separation image.
  • the image processing device inputs the target fluorescent image and image separation information to a trained inference model, and the output of the trained inference model is obtained as a corrected fluorescent separation image. In this way, the target fluorescent image and image separation information can be input to the inference model, and a corrected fluorescent separation image can be obtained based on the output of the inference model. As a result, the separation accuracy of the obtained separation image can be further improved.
  • the multiple optical states include at least a first optical state in which the fluorescence is transmitted through the optical filter, and a second optical state in which the fluorescence is not transmitted through the optical filter. Furthermore, in the second aspect, it is also preferable that the multiple optical states include at least a first optical state in which the fluorescence is transmitted through the optical filter, and a second optical state in which the fluorescence is not transmitted through the optical filter. In this way, it is possible to obtain fluorescence wavelength information based on fluorescence images in two types of optical states, and it is possible to store image separation information obtained as a result of clustering the pixel groups of the fluorescence images based on the wavelength information. This makes it possible to obtain a highly accurate separation image while reducing the overall calculation time.
  • a fluorescence separation image is generated based on the image separation information and the target fluorescence image acquired in the second optical state.
  • the image processing device in generating the fluorescence separation image, the image processing device generates the fluorescence separation image based on the image separation information and the target fluorescence image acquired in the second optical state.
  • the embodiment of the separated image acquisition method is a separated image acquisition method including: [1] a selection step of selecting desired image separation information from a storage unit storing a plurality of image separation information for obtaining a fluorescence separation image obtained by separating a fluorescence image based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics; an irradiation step of irradiating a sample with each of the excitation light of the plurality of wavelengths; an acquisition step of acquiring a target fluorescence image in at least one optical state of the plurality of optical states for each of the plurality of fluorescences generated from the sample by each of the excitation light of the plurality of wavelengths via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges; and a generation step of generating a fluorescence separation image for each of the plurality of fluorescences based on the target fluorescence image acquired in the acquisition step and the image separation information selected in the
  • the separated image acquisition method of the embodiment may be [2] "the separated image acquisition method described in [1] above, in which the image separation information is stored in association with the image acquisition conditions when the plurality of reference fluorescent images were acquired, and in the selection step, the image separation information associated with the image acquisition conditions corresponding to the image acquisition conditions when the target fluorescent image was acquired is selected.”
  • the separated image acquisition method of the embodiment may be [3] "the separated image acquisition method described in [1] or [2] above, in which in the generating step, corrected image separation information is acquired, which is image separation information corrected based on the target fluorescent image and the selected image separation information.”
  • the separated image acquisition method of the embodiment may be [4] "the separated image acquisition method described in [3] above, in which, in the generating step, the fluorescent separated image is generated using the corrected image separation information and the target fluorescent image.”
  • the separated image acquisition method of the embodiment may be [5] "the separated image acquisition method described in [1] or [2] above, in which, in the generating step, the fluorescent separated image is corrected based on the target fluorescent image and the image separation information.”
  • the separated image acquisition method of the embodiment may be [6] "the separated image acquisition method described in [1] or [2] above, in which in the generating step, the fluorescence separated image generated based on the target fluorescence image and the image separation information is input to a trained inference model, and the output of the trained inference model is obtained as the corrected fluorescence separated image.”
  • the separated image acquisition method of the embodiment may be [7] "the separated image acquisition method described in [1] or [2] above, in which, in the generating step, the target fluorescent image and the image separation information are input to a trained inference model, and the output of the trained inference model is obtained as the corrected fluorescent separated image.”
  • the separated image acquisition method of the embodiment may be [8] "the separated image acquisition method described in any one of [1] to [7] above, in which the plurality of optical states includes at least a first optical state in which the fluorescence is transmitted through an optical filter, and a second optical state in which the fluorescence is not transmitted through the optical filter.”
  • the separated image acquisition method of the embodiment may be [9] "the separated image acquisition method described in [8] above, in which in the generating step, a fluorescent separated image is generated based on the image separation information and the target fluorescent image acquired in the second optical state.”
  • the separated image acquisition device of the embodiment is [10] "a storage unit that stores image separation information for obtaining a fluorescence separation image obtained by separating a fluorescence image based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics; an irradiation device that irradiates a sample with each of excitation light having a plurality of wavelength distributions; an image acquisition device that acquires a target fluorescence image in at least one of the plurality of optical states for each of a plurality of fluorescences generated from the sample by each of the excitation light of a plurality of wavelengths via a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges; and an image processing device that processes the fluorescence image, wherein the image processing device selects desired image separation information from the plurality of image separation information stored in the storage unit, and generates a fluorescence separation image for each of the plurality of fluorescences based on the target fluorescence image acquired
  • the separated image acquisition device of the embodiment may be [11] "the separated image acquisition device described in [10] above, in which the storage unit stores the image separation information in association with the image acquisition conditions when the multiple reference fluorescent images were acquired, and the image processing device selects the image separation information associated with the image acquisition conditions corresponding to the image acquisition conditions when the target fluorescent image was acquired.”
  • the separated image acquisition device of the embodiment may be [12] "the separated image acquisition device described in [10] or [11] above, in which the image processing device acquires corrected image separation information, which is image separation information corrected based on the target fluorescent image and the selected image separation information, in generating the fluorescent separation image.”
  • the separated image acquisition device of the embodiment may be [13] "the separated image acquisition device described in [12] above, in which the image processing device generates the fluorescent separation image using the corrected image separation information and the target fluorescent image in generating the fluorescent separation image.”
  • the separated image acquisition device of the embodiment may be [14] "the separated image acquisition device described in [10] or [11] above, in which the image processing device corrects the fluorescent separation image based on the target fluorescent image and the image separation information in generating the fluorescent separation image.”
  • the separated image acquisition device of the embodiment may be [15] "the separated image acquisition device described in [10] or [11] above, in which, in generating the fluorescent separation image, the image processing device inputs the fluorescent separation image generated based on the target fluorescent image and the image separation information into a trained inference model, and obtains the output of the trained inference model as the corrected fluorescent separation image.”
  • the separated image acquisition device of the embodiment may be [16] "the separated image acquisition device described in [10] or [11] above, in which the image processing device inputs the target fluorescent image and the image separation information into a trained inference model in generating the fluorescent separated image, and obtains the output of the trained inference model as the corrected fluorescent separated image.”
  • the separated image acquisition device of the embodiment may be [17] "the separated image acquisition device described in any one of [10] to [16] above, in which the plurality of optical states includes at least a first optical state in which the fluorescence is transmitted through an optical filter, and a second optical state in which the fluorescence is not transmitted through the optical filter.”
  • the separated image acquisition device of the embodiment may be [18] "the separated image acquisition device described in [17] above, in which the image processing device generates the fluorescent separation image based on the image separation information and the target fluorescent image acquired in the second optical state in generating the fluorescent separation image.”
  • the separated image acquisition program of the embodiment is [19] "a separated image acquisition program for generating a fluorescence separation image based on a plurality of reference fluorescence images in a plurality of optical states having different wavelength characteristics, which are acquired through a fluorescence filter unit having a plurality of reflection wavelength ranges and a plurality of transmission wavelength ranges for each of a plurality of fluorescences generated from the sample by each of the excitation lights of a plurality of wavelengths by irradiating the sample with each of the excitation lights of a plurality of wavelengths, and causes a computer to execute a storage process for storing a plurality of pieces of image separation information for obtaining a fluorescence separation image obtained by separating the fluorescence, which is acquired based on the plurality of reference fluorescence images, a selection process for selecting a desired image separation information from the stored plurality of image separation information, and a generation process for generating a fluorescence separation image based on a target fluorescence image acquired in at least one of the pluralit
  • 1... fluorescent dye image acquisition system 3... image acquisition device, 5, 5A... image processing device, 7... excitation light source (irradiation device), 9a... light source side filter set, 9b... camera side filter set (fluorescence filter section), 11... dichroic mirror, 15... camera (image acquisition device), 13... wavelength information acquisition optical system (optical filter), 201... information search section (selection section), 202... image acquisition section, 203... matrix acquisition section, 204... matrix correction section, 204A... image correction section, 205... image generation section, 205A... image output section, 206... image separation information storage section (memory section), C1 , C2 , C3 ... fluorescent dyes, GC1 to GC6 ... fluorescent images, PGr01 to PGr03 , PGr1 to PGr6 ... pixel group, S... sample.

Landscapes

  • Health & Medical Sciences (AREA)
  • Immunology (AREA)
  • Chemical & Material Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Biochemistry (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Analytical Chemistry (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Pathology (AREA)
  • Optics & Photonics (AREA)
  • Chemical Kinetics & Catalysis (AREA)
  • Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)
PCT/JP2023/046652 2023-02-24 2023-12-26 分離画像取得方法、分離画像取得装置、及び分離画像取得プログラム Ceased WO2024176615A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
CN202380094657.6A CN120752513A (zh) 2023-02-24 2023-12-26 分离图像取得方法、分离图像取得装置、及分离图像取得程序
EP23924259.7A EP4653848A1 (en) 2023-02-24 2023-12-26 Separated image acquisition method, separated image acquisition device, and separated image acquisition program
JP2024537453A JP7566216B1 (ja) 2023-02-24 2023-12-26 分離画像取得方法、分離画像取得装置、及び分離画像取得プログラム
JP2024172416A JP2024174139A (ja) 2023-02-24 2024-10-01 分離画像取得方法、分離画像取得装置、及び分離画像取得プログラム

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2023-027522 2023-02-24
JP2023027522 2023-02-24

Publications (1)

Publication Number Publication Date
WO2024176615A1 true WO2024176615A1 (ja) 2024-08-29

Family

ID=92500827

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/046652 Ceased WO2024176615A1 (ja) 2023-02-24 2023-12-26 分離画像取得方法、分離画像取得装置、及び分離画像取得プログラム

Country Status (4)

Country Link
EP (1) EP4653848A1 (https=)
JP (2) JP7566216B1 (https=)
CN (1) CN120752513A (https=)
WO (1) WO2024176615A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2026042343A1 (ja) * 2024-08-20 2026-02-26 浜松ホトニクス株式会社 生成方法、生成装置、及びプログラム

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2005036143A1 (ja) * 2003-10-10 2005-04-21 Hamamatsu Photonics K.K. 蛍光色素の濃度を定量する方法およびシステム
JP2008541179A (ja) * 2005-05-18 2008-11-20 オリンパス・ソフト・イメージング・ソリューションズ・ゲゼルシャフト・ミト・ベシュレンクテル・ハフツング 多色画像、特に顕微鏡多色透過画像におけるスペクトル重畳またはカラー重畳画像寄与の分離
US20210208076A1 (en) * 2020-01-03 2021-07-08 Korea Advanced Institute Of Science And Technology Method and Apparatus for Multiplexed Imaging of Spectrally-Similar Fluorophores
WO2023026742A1 (ja) * 2021-08-25 2023-03-02 浜松ホトニクス株式会社 色素画像取得方法、色素画像取得装置、及び色素画像取得プログラム

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2005036143A1 (ja) * 2003-10-10 2005-04-21 Hamamatsu Photonics K.K. 蛍光色素の濃度を定量する方法およびシステム
JP2008541179A (ja) * 2005-05-18 2008-11-20 オリンパス・ソフト・イメージング・ソリューションズ・ゲゼルシャフト・ミト・ベシュレンクテル・ハフツング 多色画像、特に顕微鏡多色透過画像におけるスペクトル重畳またはカラー重畳画像寄与の分離
US20210208076A1 (en) * 2020-01-03 2021-07-08 Korea Advanced Institute Of Science And Technology Method and Apparatus for Multiplexed Imaging of Spectrally-Similar Fluorophores
WO2023026742A1 (ja) * 2021-08-25 2023-03-02 浜松ホトニクス株式会社 色素画像取得方法、色素画像取得装置、及び色素画像取得プログラム

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
MCRAE TRISTAN D., OLEKSYN DAVID, MILLER JIM, GAO YU-RONG: "Robust blind spectral unmixing for fluorescence microscopy using unsupervised learning", PLOS ONE, vol. 14, no. 12, 2 December 2019 (2019-12-02), US , pages 1 - 24, XP093202947, ISSN: 1932-6203, DOI: 10.1371/journal.pone.0225410 *
See also references of EP4653848A1 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2026042343A1 (ja) * 2024-08-20 2026-02-26 浜松ホトニクス株式会社 生成方法、生成装置、及びプログラム

Also Published As

Publication number Publication date
EP4653848A1 (en) 2025-11-26
CN120752513A (zh) 2025-10-03
JP7566216B1 (ja) 2024-10-11
JP2024174139A (ja) 2024-12-13
JPWO2024176615A1 (https=) 2024-08-29

Similar Documents

Publication Publication Date Title
JP7537029B2 (ja) 色素画像取得方法、色素画像取得装置、及び色素画像取得プログラム
McRae et al. Robust blind spectral unmixing for fluorescence microscopy using unsupervised learning
JP7404906B2 (ja) 情報処理装置、及び顕微鏡システム
JP2025087824A (ja) 情報処理装置及び情報処理方法
JP7566216B1 (ja) 分離画像取得方法、分離画像取得装置、及び分離画像取得プログラム
Begelman et al. Blind decomposition of transmission light microscopic hyperspectral cube using sparse representation
CN115376126A (zh) 显微高光谱图像分类模型的构建方法、分类方法及装置
CN117274236B (zh) 基于高光谱图像的尿液成分异常检测方法及系统
JP7549759B1 (ja) 蛍光画像取得方法、蛍光画像取得装置、及び蛍光画像取得プログラム
JP7564408B1 (ja) 色素データ取得方法、色素データ取得装置、及び色素データ取得プログラム
WO2026042343A1 (ja) 生成方法、生成装置、及びプログラム
WO2026042344A1 (ja) 色素データ取得方法、色素データ取得装置、及び色素データ取得プログラム
EP4653850A1 (en) Image acquisition device
Tyagi et al. Spectral band attention networks for efficient multi-feature fusion in hyperspectral and RGB data with ensemble deep learning networks
JP2026036378A (ja) 色素データ取得方法、色素データ取得装置、及び色素データ取得プログラム
Iqbal et al. Hyperspectral imaging and machine learning for soybean SDS severity classification
WO2026042403A1 (ja) 画像取得方法、装置、及びプログラム
Wang et al. A color classification system for sunglass lenses based on YCrCb-MST hyperspectral reconstruction
US20250377303A1 (en) Computer-implemented multispectral imaging method and system
Vasas et al. Recording animal-view videos of the natural world
GORLINI et al. Hyperspectral imaging and deep learning for automatic food quality inspection
Ingold et al. HAMscope: a snapshot Hyperspectral Autofluorescence Miniscope for real-time molecular imaging
Rabinovich et al. Quantitative spectral decomposition for stained tissue analysis
Ortiz-de-Solorzano et al. Spectral unmixing of multiply stained fluorescence samples T
Orlíček Acceleration of hyperspectral data processing algorithms

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 2024537453

Country of ref document: JP

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23924259

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202380094657.6

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

WWP Wipo information: published in national office

Ref document number: 202380094657.6

Country of ref document: CN

WWP Wipo information: published in national office

Ref document number: 2023924259

Country of ref document: EP