CN113508290A

CN113508290A - Information processing apparatus and microscope system

Info

Publication number: CN113508290A
Application number: CN202080017375.2A
Authority: CN
Inventors: 池田宪治; 岸井典之; 中钵秀弥; 田口步; 中川和博; 安川咲湖
Original assignee: Sony Group Corp
Current assignee: Sony Group Corp
Priority date: 2019-03-04
Filing date: 2020-02-26
Publication date: 2021-10-15
Also published as: JP2020144109A; JP7404906B2

Abstract

The present invention makes it possible to perform fluorescence separation more appropriately. An information processing apparatus according to one embodiment is provided with: a fluorescence signal acquisition unit (112) that acquires a plurality of fluorescence spectra corresponding to a plurality of excitation lights, respectively, the plurality of excitation lights having mutually different wavelengths, and that irradiates a fluorescence-stained sample (30) prepared by staining a sample (20) with a fluorescent reagent (10); a linking unit (131) that links at least some of the plurality of fluorescence spectra in a wavelength direction to generate linked fluorescence spectra; a separation processing unit (132) that separates the linked fluorescence spectrum into a spectrum of each fluorescent material using a reference spectrum including a linked autofluorescence reference spectrum in which spectra of the autofluorescence materials in the sample are linked in a wavelength direction and a linked fluorescence reference spectrum in which spectra of the fluorescent materials in the fluorescently stained sample are linked in the wavelength direction; and an extraction unit (132) that updates the linked autofluorescence reference spectra using the spectrum of each fluorescent material separated by the separation processing unit.

Description

Information processing apparatus and microscope system

Technical Field

The present disclosure relates to an information processing apparatus and a microscope system.

Background

In recent years, with the development of cancer immunotherapy and the like, multiple markers of fluorescence and immunostaining have been advanced. For example, a method of extracting an autofluorescence spectrum from a non-stained section of the same tissue piece and then performing fluorescence separation on the stained section using the autofluorescence spectrum has been performed.

Further, for example, patent document 1 below discloses a technique of approximating a fluorescence spectrum obtained by irradiating microparticles multi-labeled with a plurality of kinds of fluorescent pigments with excitation light to a linear sum of single-dyeing spectra obtained by microparticles labeled with each kind of fluorescent pigment, respectively.

CITATION LIST

Patent document

Patent document 1: japanese patent application laid-open No. 2012-18108

Disclosure of Invention

Problems to be solved by the invention

However, according to these techniques or methods, there are cases where fluorescence separation cannot be appropriately performed. For example, in the case where an autofluorescence spectrum is extracted from a non-stained section of the same tissue piece and then fluorescence separation of the stained section is performed using the autofluorescence spectrum, a physician is required to extract the autofluorescence spectrum from an appropriate space in the non-stained section, and therefore, the accuracy of the fluorescence separation depends on the work done by the physician. Further, since fluorescence separation is performed for each excitation wavelength, a separation result is output for each excitation wavelength, so that the spectrum obtained as the separation result is not uniquely determined.

Accordingly, the present disclosure has been made in view of the above circumstances, and provides a novel and improved information processing apparatus and microscope system capable of more appropriately performing fluorescence separation.

Solution to the problem

According to an embodiment of the present disclosure, an information processing apparatus includes: a fluorescence signal acquisition unit that acquires a plurality of fluorescence spectra corresponding to each of a plurality of excitation lights having different wavelengths and irradiated to a fluorescently-stained sample generated by staining the sample with a fluorescent reagent; a link unit that generates a linked fluorescence spectrum by linking at least a part of the plurality of fluorescence spectra to each other in a wavelength direction; a separation unit that separates the linked fluorescence spectrum into a spectrum for each fluorescent substance using a reference spectrum including a linked autofluorescence reference spectrum in which spectra of the autofluorescent substances in the sample are linked to each other in a wavelength direction and a linked fluorescence reference spectrum in which spectra of the fluorescent substances in the fluorescently-stained sample are linked to each other in the wavelength direction; and an extraction unit that updates the linked autofluorescence reference spectrum using the spectrum of each fluorescent substance separated by the separation unit.

Drawings

Fig. 1 is a block diagram showing a configuration example of an information processing system according to a first embodiment.

Fig. 2 is a diagram showing a specific example of the fluorescence spectrum acquired by the fluorescence signal acquiring unit.

Fig. 3 is a diagram for describing a method of generating a linked fluorescence spectrum by a linking unit.

FIG. 4 is a graph showing fluorescence spectra of AF546 and AF555 with a wavelength resolution of 8 nm.

FIG. 5 is a graph showing fluorescence spectra of AF546 and AF555 with a wavelength resolution of 1 nm.

Fig. 6 is a diagram illustrating an example of linked fluorescence spectra generated from the fluorescence spectra illustrated in a to D of fig. 3.

Fig. 7 is a block diagram showing a more specific configuration example of the separation processing unit according to the embodiment of the first embodiment.

Fig. 8 is a diagram showing a specific example of the linked autofluorescence reference spectrum.

Fig. 9 is a diagram showing a specific example of the linked fluorescence reference spectrum.

Fig. 10 is a block diagram showing a configuration example of a microscope system in a case where the information processing system according to the first embodiment is implemented as the microscope system.

Fig. 11 is a flowchart showing an example of the flow of the process of fluorescence separation by the information processing apparatus according to the first embodiment.

Fig. 12 is a block diagram showing a more specific configuration example of the separation processing unit according to the second embodiment.

Fig. 13 is a diagram for describing an overview of non-negative matrix factorization.

Fig. 14 is a diagram for describing an overview of clustering.

Fig. 15 is a flowchart showing an example of the flow of the process of fluorescence separation by the information processing apparatus according to the second embodiment.

Fig. 16 is a diagram for describing a method of calculating the number of fluorescent molecules (or the number of antibodies) in the imaging element 1[ pixel ] in the modification.

Fig. 17 is a block diagram showing a schematic configuration example of a separation processing unit according to the third embodiment.

Fig. 18 is a diagram showing an example (excitation wavelength: 392nm) of a sample image input to the matrix a in the third embodiment.

Fig. 19 is a diagram showing an example (excitation wavelength: 470nm) of a sample image input to the matrix a in the third embodiment.

Fig. 20 is a diagram showing an example (excitation wavelength: 515nm) of a sample image input to the matrix a in the third embodiment.

Fig. 21 is a diagram showing an example (excitation wavelength: 549nm) of a sample image input to the matrix a in the third embodiment.

Fig. 22 is a diagram showing an example of a sample image input to the matrix a in the third embodiment (excitation wavelength: 628 nm).

Fig. 23 is a diagram showing an example of a fluorescence separation image acquired by NMF as a matrix W in the case where the sample images shown in fig. 18 to 22 are input in the third embodiment (section 1).

Fig. 24 is a diagram showing an example of a fluorescence separation image acquired by NMF as a matrix W in the case where the sample images shown in fig. 18 to 22 are input in the third embodiment (section 2).

Fig. 25 is a diagram showing an example of a fluorescence separation image acquired by NMF as a matrix W in the case where the sample images shown in fig. 18 to 22 are input in the third embodiment (section 3).

Fig. 26 is a diagram showing an example of a fluorescence separation image acquired by NMF as a matrix W in the case where the sample images shown in fig. 18 to 22 are input in the third embodiment (section 4).

Fig. 27 is a diagram showing an example of a fluorescence separation image acquired by NMF as a matrix W in the case where the sample images shown in fig. 18 to 22 are input in the third embodiment (section 5).

Fig. 28 is a diagram showing an example of a fluorescence separation image acquired by NMF as a matrix W in the case where the sample images shown in fig. 18 to 22 are input in the third embodiment (section 6).

Fig. 29 is a diagram showing an example of a fluorescence separation image acquired by NMF as a matrix W in the case where the sample images shown in fig. 18 to 22 are input in the third embodiment (section 7).

Fig. 30 is a flowchart for describing an NMF flow according to the fourth embodiment.

Fig. 31 is a diagram for describing a flow of processing in the first cycle of the NMF shown in fig. 30.

Fig. 32 is a diagram showing an example of the initial value of the dyeing fluorescence spectrum.

Fig. 33 is a diagram showing an example of a staining fluorescence spectrum after NMF is performed according to the fourth embodiment.

Fig. 34 is a diagram showing an example of a spectrum of a fluorescent substance extracted by a method not using a non-stained specimen according to the fourth embodiment.

Fig. 35 is a diagram showing an example of a spectrum of a fluorescent substance extracted in the case of using a non-stained specimen.

Fig. 36 is a diagram showing an example of a measurement system of an information processing system according to the sixth embodiment.

Fig. 37 is a flowchart showing an example of the operation of the processing unit according to the sixth embodiment.

Fig. 38 is a diagram for describing processing performed by the processing unit in each step in fig. 37 (part 1).

Fig. 39 is a diagram for describing processing performed by the processing unit in each step in fig. 37 (part 2).

Fig. 40 is a diagram for describing processing performed by the processing unit in each step of fig. 37 (part 3).

Fig. 41 is a flowchart showing an example of the operation of the processing unit according to the first modification of the sixth embodiment.

Fig. 42 is a block diagram showing an example of a hardware configuration of an information processing apparatus according to each embodiment and modification.

Detailed Description

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration will be denoted by the same reference numerals, and overlapping description will be omitted.

Note that description will be made in the following order.

1. First embodiment

1.1. Example of configuration

1.2. Example Process flow

2. Second embodiment

2.1. Example Process flow

2.2. Principal Component Analysis (PCA) is not suitable as a method for extracting a linked autofluorescence reference spectrum from a non-stained section

2.3. Application example

3. Modifications of the invention

4. Third embodiment

5. Fourth embodiment

5.1. Method for fixing dyeing fluorescence spectrum in minimum mean square residual D by using recurrence formula

5.2. Fixing method for dyeing fluorescence spectrum in minimum mean square residual D by using DFP method, BFGS method and the like

6. Fifth embodiment

6.1. Processing overview of a processing Unit

6.2. Configuration example of measurement System

6.3. Example of operation

6.4.1. First modification

6.4.2. Second modification

6.5. Effect

7. Hardware configuration example

8. Conclusion

<1. first embodiment >

First, a first embodiment of the present disclosure will be described.

(1.1. configuration example)

A configuration example of an information processing system according to the present embodiment will be described with reference to fig. 1. As shown in fig. 1, the information processing system according to the present embodiment includes an information processing apparatus 100 and a database 200, and has a fluorescent reagent 10, a specimen 20, and a fluorescently stained specimen 30 as inputs to the information processing system.

(fluorescent reagent 10)

The fluorescent reagent 10 is a chemical for staining the specimen 20. The fluorescent reagent 10 is, for example, a fluorescent antibody (including a primary antibody for direct labeling or a secondary antibody for indirect labeling), a fluorescent probe, a nuclear staining reagent, or the like, but the type of the fluorescent reagent 10 is not limited thereto. Further, the fluorescent reagent 10 is managed by attaching identification information (hereinafter referred to as "reagent identification information 11") capable of identifying the fluorescent reagent 10 (or the production lot of the fluorescent reagent 10) to the fluorescent reagent 10. The reagent identification information 11 is, for example, barcode information (one-dimensional barcode information, two-dimensional barcode information, etc.), but is not limited thereto. The properties of the fluorescent reagent 10 are different for each production lot according to the production method, the state of the cells from which the antibody is obtained, and the like, even if the products are identical to each other. For example, in the fluorescent reagent 10, the spectrum, the quantum yield, the fluorescence labeling rate, and the like are different for each production lot. Therefore, in the information processing system according to the present embodiment, the fluorescent reagent 10 is managed for each production lot by attaching the reagent identification information 11 to the fluorescent reagent 10. Therefore, the information processing apparatus 100 can perform fluorescence separation in consideration of a slight difference in the property occurring per production lot.

(sample 20)

The sample 20 is prepared from a clinical sample or a tissue sample collected from a human body for the purpose of pathological diagnosis or the like. The specimen 20 may be a tissue slice, a cell, or a microparticle, and there is no particular limitation with respect to the specimen 20, the type of tissue used (e.g., organ, etc.), the type of target disease, the attributes of the target person (e.g., age, sex, blood type, race, etc.), or the lifestyle of the target person (e.g., eating habits, exercise habits, smoking habits, etc.). Note that the tissue section may include, for example, a section before staining of a tissue section to be stained (hereinafter also simply referred to as a section), a section adjacent to the stained section, a section different from the stained section in the same piece (sampled from the same position as the stained section), a section in a different piece in the same tissue (sampled from a position different from the stained section), a section collected from a different patient, and the like. Further, the samples 20 are managed by attaching identification information (hereinafter referred to as "sample identification information 21") capable of identifying each sample 20 to the sample 20. The sample identification information 21 is, for example, barcode information or the like (one-dimensional barcode information, two-dimensional barcode information or the like), similar to the reagent identification information 11, but is not limited thereto. The attributes of the sample 20 differ according to the type of tissue used, the type of disease targeted, the attributes of the targeted person, the lifestyle of the targeted person, and the like. For example, in the sample 20, a measurement channel, a spectrum, and the like differ depending on the type of tissue used, and the like. Therefore, in the information processing system according to the present embodiment, the samples 20 are managed individually by attaching the sample identification information 21 to the samples 20. Therefore, the information processing apparatus 100 can perform fluorescence separation in consideration of a slight difference in the property occurring in each sample 20.

(fluorescent staining sample 30)

A fluorescently stained sample 30 is produced by staining the sample 20 with a fluorescent reagent 10. In the present embodiment, it is assumed that the fluorescently stained sample 30 is produced by staining the sample 20 with one or more kinds of the fluorescent reagents 10, but the number of the fluorescent reagents 10 used for staining is not particularly limited. Further, the staining method is determined by a combination of the specimen 20 and the fluorescent reagent 10, or the like, and is not particularly limited.

(information processing apparatus 100)

As shown in fig. 1, the information processing apparatus 100 includes an acquisition unit 110, a storage unit 120, a processing unit 130, a display unit 140, a control unit 150, and an operation unit 160. The information processing apparatus 100 may be, for example, a fluorescence microscope or the like, but is not necessarily limited thereto, and may include various apparatuses. For example, the information processing apparatus 100 may be a Personal Computer (PC) or the like.

(obtaining unit 110)

The acquisition unit 110 is a configuration that acquires information for various processes of the information processing apparatus 100. As shown in fig. 1, the acquisition unit 110 includes an information acquisition unit 111 and a fluorescence signal acquisition unit 112.

(information acquisition Unit 111)

The information acquisition unit 111 is a configuration that acquires information on the fluorescent reagent 10 (hereinafter referred to as "reagent information") or information on the sample 20 (hereinafter referred to as "sample information"). More specifically, the information acquiring unit 111 acquires the reagent identification information 11 attached to the fluorescent reagent 10 for generating the fluorescently stained sample 30 and the sample identification information 21 attached to the sample 20. For example, the information acquisition unit 111 acquires the reagent identification information 11 and the specimen identification information 21 using a barcode reader or the like. Then, the information acquiring unit 111 acquires the reagent information based on the reagent identification information 11 and the sample information based on the sample identification information 21 from the database 200, respectively. The information acquisition unit 111 stores the acquired information in the information storage unit 121 as described below.

Here, in the present embodiment, it is assumed that the sample information includes a linked autofluorescence reference spectrum in which spectra of the autofluorescent substance in the sample 20 are linked to each other in the wavelength direction, and the reagent information includes a linked fluorescence reference spectrum in which spectra of the fluorescent substance in the fluorescently-stained sample 30 are linked to each other in the wavelength direction. Note that the linked autofluorescence reference spectrum and the linked fluorescence reference spectrum are collectively referred to as "reference spectrum".

(fluorescent Signal acquiring Unit 112)

The fluorescent signal acquiring unit 112 is a configuration that acquires a plurality of fluorescent signals each corresponding to a plurality of excitation lights having different wavelengths when the fluorescence-stained sample 30 (a sample produced by staining the sample 20 with the fluorescent reagent 10) is irradiated with the plurality of excitation lights. More specifically, the fluorescence signal acquiring unit 112 receives light and outputs a detection signal according to the amount of received light to acquire a fluorescence spectrum of the fluorescently stained sample 30 based on the detection signal. Here, the content (including the excitation wavelength, intensity, and the like) of the excitation light is determined based on the reagent information and the like (in other words, information on the fluorescent reagent 10 and the like). Note that the fluorescence signal mentioned here is not particularly limited as long as it is a signal derived from fluorescence, and may be, for example, a fluorescence spectrum.

A to D of fig. 2 are specific examples of the fluorescence spectrum acquired by the fluorescence signal acquisition unit 112. In A to D of FIG. 2, specific examples of fluorescence spectra obtained in the case where the fluorescence-stained specimen 30 includes four types of fluorescent substances (e.g., DAPI, CK/AF488, PgR/AF594, and ER/AF647) and is irradiated with excitation light having excitation wavelengths of 392[ nm ] (A of FIG. 2), 470[ nm ] (B of FIG. 2), 549[ nm ] (C of FIG. 2), and 628[ nm ] (D of FIG. 2), respectively, are shown. It should be noted that energy is released for fluorescence emission so that the fluorescence wavelength is shifted to the longer wavelength side (stokes shift) than the excitation wavelength. Further, the excitation wavelengths of the fluorescent substance included in the fluorescent-stained sample 30 and the irradiated excitation light are not limited to those described above. The fluorescence signal acquisition unit 112 stores the acquired fluorescence spectrum in the fluorescence signal storage unit 122 as described later.

(storage unit 120)

The storage unit 120 is a configuration that stores information for various processes of the information processing apparatus 100 or information output by various processes. As shown in fig. 1, the storage unit 120 includes an information storage unit 121 and a fluorescent signal storage unit 122.

(information storage Unit 121)

The information storage unit 121 is a configuration that stores the reagent information and the sample information acquired by the information acquisition unit 111.

(fluorescent Signal storage Unit 122)

The fluorescence signal storage unit 122 is a configuration that stores the fluorescence signal of the fluorescence-stained sample 30 acquired by the fluorescence signal acquisition unit 112.

(processing unit 130)

The processing unit 130 is a configuration that performs various processes including fluorescence separation process. As shown in fig. 1, the processing unit 130 includes a linking unit 131, a separation processing unit 132, and an image generating unit 133.

(linking unit 131).

The link unit 131 is a configuration that generates a linked fluorescence spectrum by linking at least a part of the plurality of fluorescence spectra acquired by the fluorescence signal acquisition unit 112 to each other in the wavelength direction. For example, the linking unit 131 extracts data of a predetermined width in each fluorescence spectrum so as to include the maximum value of the fluorescence intensity of each of the four fluorescence spectra (a to D of fig. 3) acquired by the fluorescence signal acquisition unit 112 as described above. The width of the wavelength band in which the link unit 131 extracts data may be determined based on reagent information, excitation wavelength, fluorescence wavelength, or the like, and may be different from each other for each fluorescent substance (in other words, the width of the wavelength band in which the link unit 131 extracts data may be different from each other for each fluorescence spectrum shown in a to D of fig. 3). Then, as shown in E of fig. 3, the link unit 131 generates one linked fluorescence spectrum by linking the extracted data to each other in the wavelength direction. It should be noted that since the linked fluorescence spectra include data extracted from a plurality of fluorescence spectra, the wavelength is not continuous at the boundary of each linked data.

At this time, the linking unit 131 performs the above-described linking after aligning the intensities of the excitation lights corresponding to each of the plurality of fluorescence spectra with each other (in other words, after correcting the plurality of fluorescence spectra) based on the intensities of the excitation lights. More specifically, the linking unit 131 performs the above-described linking after aligning the intensities of the excitation light corresponding to each of the plurality of fluorescence spectra with each other by dividing each fluorescence spectrum by the excitation power density (which is the intensity of the excitation light). Therefore, a fluorescence spectrum in the case of irradiating excitation light of the same intensity is obtained. Further, in the case where the intensities of the irradiated excitation lights are different from each other, the intensities of the spectra absorbed by the fluorescent-stained sample 30 (hereinafter referred to as "absorption spectra") are also different from each other according to the intensity of the irradiated excitation lights. Therefore, as described above, the absorption spectrum can be appropriately evaluated by aligning the intensities of the excitation light corresponding to each of the plurality of fluorescence spectra with each other.

As described above, the intensity of the excitation light in this specification may be excitation power or excitation power density. The excitation power or excitation power density may be power or power density obtained by actually measuring the excitation light emitted from the light source 104, or may be power or power density obtained from a driving voltage applied to the light source 104. Note that the intensity of excitation light in this specification may be a value obtained by correcting the above-described excitation power density using the absorption rate of each excitation light of a slice as an observation target, the amplification factor of a detection signal in a detection system (fluorescence signal acquisition unit 112 or the like) that detects fluorescence emitted from the slice, or the like. That is, the intensity of the excitation light in the present specification may be a power density of the excitation light that actually contributes to the excitation of the fluorescent substance, a value obtained by correcting the power density with an amplification factor of the detection system, or the like. By taking into account the absorbance, amplification factor, and the like, the intensity of excitation light that varies according to variations in machine conditions, environment, and the like can be appropriately corrected, and thus linked fluorescence spectra that enable more accurate color separation can be produced.

Note that the correction value based on the intensity of the excitation light of each fluorescence spectrum (also referred to as an intensity correction value) is not limited to a value for aligning the intensities of the excitation lights corresponding to each of the plurality of fluorescence spectra with each other, and various modifications may be made. For example, the signal intensity of a fluorescence spectrum having an intensity peak on the long wavelength side tends to be lower than that of a fluorescence spectrum having an intensity peak on the short wavelength side. Therefore, in the case where the linked fluorescence spectrum includes a fluorescence spectrum having an intensity peak on the long wavelength side and a fluorescence spectrum having an intensity peak on the short wavelength side, there is a case where the fluorescence spectra having an intensity peak on the long wavelength side are hardly added and only the fluorescence spectrum having an intensity peak on the short wavelength side is extracted. In this case, for example, by setting the intensity correction value of the fluorescence spectrum having an intensity peak on the long wavelength side to a large value, the separation accuracy of the fluorescence spectrum on the short wavelength side can be improved.

Further, the link unit 131 may correct the wavelength resolution of each of the plurality of fluorescence spectra to be linked to each other independently of the other fluorescence spectra. For example, the fluorescence spectrum of AF546 and the fluorescence spectrum of AF555 have almost the same spectral shape and peak wavelength, and the difference between the fluorescence spectrum of AF546 and the fluorescence spectrum of AF555 is that the fluorescence spectrum of AF555 has a shoulder at the bottom on the high wavelength side, while the fluorescence spectrum of AF546 has no shoulder. Thus, in the case where the two fluorescence spectra are close to each other, there arises a problem that it is difficult to color-separate the two fluorescence spectra from each other by spectrum extraction.

This problem can be solved by improving the wavelength resolution of the fluorescence spectrum of the link. FIG. 4 is a diagram showing fluorescence spectra of AF546 and AF555 with a wavelength resolution of 8nm, and FIG. 5 is a diagram showing fluorescence spectra of AF546 and AF555 with a wavelength resolution of 1 nm. As shown in fig. 4, the spectral shape and peak wavelength of AF546 and the spectral shape and peak wavelength of AF555 substantially coincide with each other with a wavelength resolution of 8 nm. Therefore, it is practically difficult to color-separate these spectral shapes and peak wavelengths from each other using, for example, the least squares method. On the other hand, in the case where the wavelength resolution is 8 times the wavelength resolution shown in fig. 4, i.e., 1nm, as shown in fig. 5, the spectral shape and peak wavelength of AF546 and the spectral shape and peak wavelength of AF555 can be clearly separated from each other. This shows that even in the case of using a plurality of fluorescence spectra having close spectral shapes and peak wavelengths, color separation can be performed using the plurality of fluorescence spectra by increasing the wavelength resolution.

However, when the wavelength resolution is increased, the data amount of the linked fluorescence spectrum becomes large, so that the required memory capacity, the calculation cost in the fluorescence separation process, and the like increase. Therefore, the link unit 131 corrects a fluorescence spectrum that is assumed to be difficult to be color-separated among a plurality of fluorescence spectra to be linked to each other so that the wavelength resolution of the fluorescence spectrum becomes high, and corrects a fluorescence spectrum that is assumed to be easy to be color-separated among a plurality of fluorescence spectra so that the wavelength resolution of the fluorescence spectrum becomes low. Therefore, the accuracy of color separation can be improved while suppressing an increase in the amount of data.

Here, a method of generating the linked fluorescence spectrum by the linking unit 131 will be described by a specific example. In the present specification, similarly to the method of generating a linked fluorescence spectrum described with reference to fig. 3, a case of linking four fluorescence spectra obtained by irradiating a fluorescence-stained specimen 30 including four types (e.g., DAPI, CK/AF488, PgR/AF594, and ER/AF647) of fluorescent substances with excitation light each having an excitation wavelength of 392nm, 470nm, 549nm, and 628nm is exemplified.

Fig. 6 is a diagram illustrating an example of linked fluorescence spectra generated from the fluorescence spectra illustrated in a to D of fig. 3. As shown in fig. 6, the link unit 131 extracts a fluorescence spectrum SP1 in a wavelength band in which the excitation wavelength is 392nm or more and 591nm or less from the fluorescence spectrum shown in a of fig. 3, extracts a fluorescence spectrum SP2 in a wavelength band in which the excitation wavelength is 470nm or more and 669nm or less from the fluorescence spectrum shown in B of fig. 3, extracts a fluorescence spectrum SP3 in a wavelength band in which the excitation wavelength is 549nm or more and 748nm or less from the fluorescence spectrum shown in C of fig. 3, and extracts a fluorescence spectrum SP4 in a wavelength band in which the excitation wavelength is 628nm or more and 827nm or less from the fluorescence spectrum shown in D of fig. 3. Next, the link unit 131 corrects the wavelength resolution of the extracted fluorescence spectrum SP1 to 16nm (no intensity correction), corrects the intensity of the fluorescence spectrum SP2 to 1.2 times, and corrects the wavelength resolution of the fluorescence spectrum SP2 to 8nm, corrects the intensity of the fluorescence spectrum SP3 to 1.5 times (no wavelength resolution correction), corrects the intensity of the fluorescence spectrum SP4 to 4.0 times, and corrects the wavelength resolution of the fluorescence spectrum SP4 to 4 nm. Then, the link unit 131 generates the linked fluorescence spectra as shown in fig. 6 by sequentially linking the fluorescence spectra SP1 to SP4 to each other after the correction.

Note that, in fig. 6, a case is shown in which the link unit 131 has extracted and linked the fluorescence spectra SP1 to SP4 having a predetermined bandwidth (width of 200nm in fig. 6) from the excitation wavelength when each fluorescence spectrum has been acquired, but the bandwidths of the fluorescence spectra extracted by the link unit 131 do not need to coincide with each other in each fluorescence spectrum and may be different from each other. That is, the region extracted from each fluorescence spectrum by the link unit 131 only needs to be a region including the peak wavelength of each fluorescence spectrum, and the wavelength band and the bandwidth of each fluorescence spectrum may be appropriately changed. In this case, a shift in spectral wavelength due to stokes shift can be considered. In this way, by narrowing down the wavelength band to be extracted, the data amount can be reduced, and therefore the fluorescence separation process can be performed at higher speed.

(separation processing unit 132)

The separation processing unit 132 is a configuration that separates the linked fluorescence spectra of each molecule. Fig. 7 is a block diagram showing a more specific configuration example of the separation processing unit according to the present embodiment. As shown in fig. 7, the separation processing unit 132 includes a color separation unit 1321 and a spectrum extraction unit 1322.

The color separation unit 1321 includes, for example, a first color separation unit 1321a and a second color separation unit 1321b, and performs color separation on the linked fluorescence spectrum of the stained section (also referred to as a stained specimen) input from the linking unit 131 for each molecule.

The spectrum extraction unit 1322 is a configuration that raises the linked autofluorescence reference spectrum so as to be able to obtain a more accurate color separation result, and adjusts the linked autofluorescence reference spectrum included in the sample information input from the information storage unit 121 based on the color separation result of the color separation unit 1321 so as to obtain a more accurate color separation result.

More specifically, the first color separation unit 1321a separates the linked fluorescence spectrum into a spectrum of each molecule by performing a color separation process using the linked fluorescence reference spectrum included in the reagent information and the linked autofluorescence reference spectrum included in the sample information, which are input from the information storage unit 121, with respect to the linked fluorescence spectrum of the stained sample input from the linking unit 131. Note that, for example, a Least Squares Method (LSM), a Weighted Least Squares Method (WLSM), or the like may be used for the color separation process.

The spectrum extraction unit 1322 improves the linked autofluorescence reference spectrum with respect to the linked autofluorescence reference spectrum input from the information storage unit 121 by performing spectrum extraction processing using the color separation result input from the first color separation unit 1321a, and adjusting the linked autofluorescence reference spectrum based on the result of the spectrum extraction processing, so as to obtain a more accurate color separation result. Note that, for example, non-Negative Matrix Factorization (NMF), singular value decomposition (singular value decomposition), or the like may be used for the spectrum extraction process.

The second color separation unit 1321b separates the linked fluorescence spectrum into a spectrum of each molecule by performing a color separation process using the adjusted linked autofluorescence reference spectrum input from the spectrum extraction unit 1322 with respect to the linked fluorescence spectrum of the stained sample input from the linking unit 131. Note that, for example, like the first color separation unit 1321a, a Least Squares Method (LSM), a Weighted Least Squares Method (WLSM), or the like may be used for the color separation process.

Note that, the case where the adjustment of the linked autofluorescence reference spectrum has been performed once has been exemplified in fig. 7, but the present disclosure is not limited thereto, and a final color separation result may be acquired after the color separation result of the second color separation unit 1321b is input to the spectrum extraction unit 1322 and the process for performing the adjustment of the linked autofluorescence reference spectrum again is repeated one or more times in the spectrum extraction unit 1322.

In fig. 8, a specific example of an autofluorescence reference spectrum linked in the case where the autofluorescent substance is hemoglobin, archidonic acid, catalase, collagen, FAD, NADPH, and prolong diamond is shown. In fig. 9, a specific example of the fluorescence reference spectrum linked in the case where the fluorescent substances are CK, ER, PgR, and DAPI is shown. The linked fluorescence reference spectrum and the linked autofluorescence reference spectrum may be generated by a method similar to (but not necessarily limited to) the method in which the linking unit 131 generates the linked fluorescence spectrum. More specifically, the linked fluorescence reference spectrum and the linked autofluorescence reference spectrum may be generated by linking data having a predetermined wavelength bandwidth with each other in the wavelength direction among a plurality of spectra acquired by a plurality of excitation lights having the same excitation wavelength as when the linked fluorescence spectrum is generated. At this time, it is assumed (but not necessarily limited thereto) that the intensities of the excitation lights corresponding to each of the plurality of spectra are aligned with each other based on the intensity of the excitation light (e.g., excitation power density). Note that the method of generating the linked fluorescence reference spectrum and the linked autofluorescence reference spectrum is not necessarily limited to the above-described method. For example, the linked fluorescence reference spectrum and the linked autofluorescence reference spectrum may be generated based on theoretical values, catalog values, and the like of the spectrum of each substance.

Next, calculation regarding the least square method will be described. The least squares method is to calculate a color mixing ratio by fitting the linked fluorescence spectrum generated by the linking unit 131 to the reference spectrum. Note that the color mixing ratio is an index indicating the degree to which the respective substances are mixed with each other. The following equation (1) is an equation representing a residual obtained by subtracting the reference spectrum (St) (the linked fluorescence reference spectrum and the linked autofluorescence reference spectrum) mixed at the color mixing ratio a from the linked fluorescence spectrum (Signal). Note that "Signal (1 × the number of channels)" in equation (1) indicates that the linked fluorescence spectrum (Signal) exists in the number of channels of wavelength (for example, Signal is a matrix indicating the linked fluorescence spectrum). Further, "St (number of substances × number of channels)" indicates that the reference spectrum exists in the number of wavelength channels per substance (fluorescent substance and autofluorescent substance) (for example, St is a matrix indicating the reference spectrum). Further, "a (1 × number of substances)" indicates that a color mixing ratio a is provided for each substance (fluorescent substance and autofluorescent substance) (for example, a is a matrix indicating the color mixing ratio of each reference spectrum in the linked fluorescence spectrum).

[ equation 1]

Signal (1 × number of channels) -a (1 × number of material) × ST (number of material × number of channels) (1)

Then, the first color separation unit 1321a or the second color separation unit 1321b calculates a color mixture rate a of each substance at which the sum of squares of the residual equation (1) is minimum. Since the sum of squares of the residuals becomes minimum in the case where the result of the partial differentiation with respect to the color mixture rate a is 0 for equation (1) representing the residuals, the first color separation unit 1321a or the second color separation unit 1321b calculates the color mixture rate a of each substance at which the sum of squares of the residuals becomes minimum by solving the following equation (2). Note that "St" in equation (2) represents a transposed matrix of the reference spectrum St. Further, "inv (St × St ') represents an inverse matrix of St × St'.

[ equation 2]

Here, specific examples of each value of the above equation (1) are represented by the following equations (3) to (5). In the examples of equations (3) to (5), the case where the reference spectra (St) of the three types of substances (the number of substances is 3) are mixed with each other at different color mixing ratios a in the linked fluorescence spectrum (Signal) is shown.

[ equation 3]

[ equation 4]

a＝(3 2 1) (4)

[ equation 5]

Signal＝a*St＝(170.1 351 410 215 78) (5)

Then, a specific example of the calculation result of the above equation (2) by each value of equations (3) and (5) is represented by the following equation (6). As can be seen from equation (6), "a ═ 321" (i.e., the same value as equation (4) above) is correctly calculated as a calculation result.

[ equation 6]

a＝Signal*Si*inv(St*St′)＝(3 2 1) (6)

As described above, the first color separation unit 1321a or the second color separation unit 1321b may output a unique spectrum as a separation result (the separation result is not different for each excitation wavelength) by performing fluorescence separation processing using the reference spectra (the linked autofluorescence reference spectrum and the linked fluorescence reference spectrum) linked in the wavelength direction. Thus, the correct spectrum can be more easily obtained by the physician. Further, a reference spectrum (linked autofluorescence reference spectrum) regarding autofluorescence used for separation is automatically acquired, and fluorescence separation processing is performed, so that a physician does not need to extract a spectrum corresponding to autofluorescence from an appropriate space of a non-stained section.

Note that, as described above, the first color separation unit 1321a or the second color separation unit 1321b may extract the spectrum of each fluorescent substance from the linked fluorescence spectrum by performing calculation with respect to a weighted least square method instead of the least square method. In the weighted least squares method, weights are assigned to emphasize errors at low signal levels by utilizing the fact that noise of the linked fluorescence spectrum (signal) as a measurement value has a poisson distribution. However, the upper limit value that is not weighted by the weighted least square method is set as the shift value. The displacement value is determined by the characteristics of the sensor used for measurement, and needs to be individually optimized in the case where the imaging element is used as a sensor. In the case of performing the weighted least squares method, the reference spectrum St in the above equations (1) and (2) is replaced with St _ represented by the following equation (7). Note that equation (7) below means that St _iscalculated by dividing (in other words, element division) each element (each component) of St represented by a matrix by each corresponding element (each component) in "Signal + displacement value (Signal + Offset value)" also represented by a matrix.

[ equation 7]

Here, in the case where the value of the displacement is 1 and the values of the reference spectrum St and the linked fluorescence spectrum signal are represented by the above equations (3) and (5), respectively, a specific example of St _ represented by the above equation (7) is represented by the following equation (8).

[ equation 8]

Then, a specific example of the calculation result of the color mixture ratio a in this case is represented by the following equation (9). As can be seen from equation (9), "a ═ 321" is correctly calculated as a calculation result.

[ equation 9]

a＝Signal*St_′*inv(St*St_′)＝(3 2 1) (9)

(image generating unit 133)

The image generation unit 133 is a configuration that generates image information based on the result of separation of the linked fluorescence spectra by the separation processing unit 132. For example, the image generation unit 133 may generate image information using a fluorescence spectrum corresponding to one or more fluorescent molecules, or generate image information using an autofluorescence spectrum corresponding to one or more autofluorescent molecules. Note that the number or combination of fluorescent molecules or autofluorescent molecules used by the image generation unit 133 to generate image information is not particularly limited. Further, in the case where various processes (e.g., segmentation, calculation of a signal-to-noise ratio value, and the like) are performed using the separated fluorescence spectrum or autofluorescence spectrum, the image generation unit 133 may generate image information indicating the results of those processes.

(display unit 140)

The display unit 140 is a configuration that displays the image information generated by the image generation unit 133 on a display to present the image information to a physician. Note that the type of display used as the display unit 140 is not particularly limited. Further, although not described in detail in the present embodiment, the image information generated by the image generation unit 133 may be projected by a projector or may be printed by a printer to be presented to a physician (in other words, a method of outputting the image information is not particularly limited).

(control unit 150)

The control unit 150 is a functional configuration that comprehensively controls general processing performed by the information processing apparatus 100. For example, the control unit 150 controls the start, end, and the like of various processes as described above (for example, adjustment processing of the placement position of the fluorescence-stained sample 30, irradiation processing of the fluorescence-stained sample 30 with excitation light, acquisition processing of a spectrum, generation processing of a linked fluorescence spectrum, fluorescence separation processing, generation processing of image information, display processing of image information, and the like) based on operation input performed by a physician via the operation unit 160. Note that the control content of the control unit 150 is not particularly limited. For example, the control unit 150 may control processing (e.g., processing regarding an Operating System (OS)) that is typically executed on a general-purpose computer, a PC, a tablet computer, or the like.

(operation unit 160)

The operation unit 160 is a configuration that receives an operation input from a doctor. More specifically, the operation unit 160 includes various input devices, for example, a keyboard, a mouse, buttons, a touch panel, a microphone, and the like, and the doctor can perform various inputs to the information processing apparatus 100 by operating these input devices. Information on operation input performed via the operation unit 160 is provided to the control unit 150.

(database 200)

The database 200 is a device that manages reagent information, sample information, and the like. More specifically, the database 200 manages the reagent identification information 11 and the reagent information, and the sample identification information 21 and the sample information in association with each other. Accordingly, the information acquiring unit 111 can acquire reagent information based on the reagent identification information 11 of the fluorescent reagent 10 from the database 200, and acquire sample information based on the sample identification information 21 of the sample 20.

The reagent information managed by the database 200 is assumed to include information of linked fluorescence reference spectra unique to a fluorescent substance possessed by the fluorescent reagent 10 and a measurement channel (but is not necessarily limited thereto). The "measurement channel" is a concept indicating a fluorescent substance included in the fluorescent reagent 10, and is a concept of CK, ER, PgR, and DAPI in the example of fig. 9. Since the number of fluorescent substances varies depending on the fluorescent reagent 10, a measurement channel is managed as reagent information in association with each fluorescent reagent 10. Further, as described above, the linked fluorescence reference spectrum included in the reagent information is a spectrum generated by linking the fluorescence spectrum of each fluorescent substance included in the measurement channel to each other in the wavelength direction.

Further, the sample information managed by the database 200 is assumed to include information of the linked autofluorescence reference spectrum and measurement channel unique to the autofluorescent substance possessed by the sample 20 (but is not necessarily limited thereto). The "measurement channel" is a concept indicating a spontaneous fluorescent substance included in the sample 20, and is a concept of hemoglobin, archidonic acid, catalase, collagen, FAD, NADPH, and prolong diamond in the example of fig. 8. Since the number of autofluorescence varies depending on the samples 20, a measurement channel is managed in association with each sample 20 as sample information. Further, as described above, the linked autofluorescence reference spectrum included in the sample information is a spectrum generated by linking the autofluorescence spectra to each other in the wavelength direction of each autofluorescent substance included in the measurement channel. Note that the information managed by the database 200 is not necessarily limited to those described above.

The configuration example of the information processing system according to the present embodiment has been described above. Note that the configuration described above with reference to fig. 1 is merely an example, and the configuration of the information processing system according to the present embodiment is not limited to such an example. For example, the information processing apparatus 100 may not necessarily include all the configurations shown in fig. 1, or may include a configuration not shown in fig. 1.

Here, the information processing system according to the present embodiment may include an imaging apparatus (e.g., including a scanner or the like) that acquires a fluorescence spectrum and an information processing apparatus that performs processing using the fluorescence spectrum. In this case, the fluorescent signal acquisition unit 112 shown in fig. 1 may be implemented by an imaging device, and other configurations may be implemented by an information processing device. Further, the information processing system according to the present embodiment may include an imaging device that acquires a fluorescence spectrum and software for processing using the fluorescence spectrum. In other words, an information handling system may not have a physical configuration (e.g., memory, processor, etc.) that stores or executes software. In this case, the fluorescent signal acquisition unit 112 shown in fig. 1 may be realized by an imaging device, and other configurations may be realized by an information processing device on which software is executed. Then, the software may be provided to the information processing apparatus via a network (e.g., from a website, a cloud server, or the like), or may be provided to the information processing apparatus via any storage medium (e.g., an optical disk, or the like). Further, the information processing apparatus on which the software is executed may be various servers (e.g., a cloud server or the like), a general-purpose computer, a PC, a tablet computer, or the like. Note that the method of providing software to the information processing apparatus and the type of the information processing apparatus are not limited to those described above. Further, it should be noted that the configuration of the information processing system according to the present embodiment is not necessarily limited to the above-described configuration, and a so-called configuration that can be thought by those skilled in the art may be applied based on the technical level at the time of use.

The above-described information processing system may be implemented as, for example, a microscope system. Therefore, next, a configuration example of a microscope system in a case where the information processing system according to the present embodiment is implemented as a microscope system will be described with reference to fig. 10.

As shown in fig. 10, the microscope system according to the present embodiment includes a microscope 101 and a data processing unit 107.

The microscope 101 includes a stage 102, an optical system 103, a light source 104, a stage driving unit 105, a light source driving unit 106, and a fluorescence signal acquiring unit 112.

The stage 102 has a placing surface on which the fluorochrome specimen 30 can be placed, and the stage 102 is movable in a direction parallel to the placing surface (x-y plane direction) and a direction perpendicular to the placing surface (z-axis direction) by the drive of the stage drive unit 105. The fluorescence-stained specimen 30 has a thickness of, for example, several micrometers to several tens micrometers in the Z direction, and is sandwiched between a slide glass SG and a cover glass (not shown), and fixed by a predetermined fixing method.

An optical system 103 is disposed above the stage 102. The optical system 103 includes an objective lens 103A, an image forming lens 103B, a dichroic mirror 103C, an emission filter 103D, and an excitation filter 103E. The light source 104 is, for example, a bulb (e.g., a mercury lamp or the like), a Light Emitting Diode (LED), or the like, and irradiates the fluorescent label attached to the fluorescent-dyed sample 30 with excitation light by driving of the light source driving unit 106.

In the case of obtaining a fluorescent image of the fluorescent-stained sample 30, the excitation filter 103E generates excitation light by transmitting only light of an excitation wavelength that excites a fluorescent dye among light emitted from the light source 104. The dichroic mirror 103C reflects the excitation light transmitted through the excitation filter and then incident on the dichroic mirror 103C, and guides the excitation light to the objective lens 103A. The objective lens 103A focuses the excitation light on the fluorescent-stained specimen 30. Then, the objective lens 103A and the image forming lens 103B magnify the image of the fluorescent-stained specimen 30 to a predetermined magnification, and form the magnified image on the image forming surface of the fluorescent signal acquiring unit 112.

When the fluorescence-stained sample 30 is irradiated with excitation light, the stain of each tissue bound to the fluorescence-stained sample 30 emits fluorescence. The fluorescence is transmitted through the dichroic mirror 103C via the objective lens 103A, and reaches the image forming lens 103B via the emission filter 103D. The emission filter 103D absorbs light that has been amplified by the above-described objective lens 103A and has been transmitted through the excitation filter 103E, and transmits only a part of the colored light. As described above, the image of the color light from which the external light is lost is enlarged by the image forming lens 103B and formed on the fluorescent signal obtainment unit 112.

The data processing unit 107 is a configuration that drives the light source 104, acquires a fluorescence image of the fluorescently stained sample 30 using the fluorescence signal acquisition unit 112, and performs various processes using the fluorescence image. More specifically, the data processing unit 107 can function as some or all of the information acquisition unit 111, the storage unit 120, the processing unit 130, the display unit 140, the control unit 150, and the operation unit 160 or the database 200 of the information processing apparatus 100 described with reference to fig. 1. For example, the data processing unit 107 functions as the control unit 150 of the information processing apparatus 100 to control the driving of the stage driving unit 105 and the light source driving unit 106, or to control the spectrum acquisition by the fluorescence signal acquisition unit 112. Further, the data processing unit 107 functions as the processing unit 130 of the information processing apparatus 100 to generate a linked fluorescence spectrum, separate the linked fluorescence spectrum of each molecule, or generate image information based on the separation result.

The configuration example of the microscope system in the case where the information processing system according to the present embodiment is implemented as the microscope system has been described above. Note that the configuration described above with reference to fig. 10 is merely an example, and the configuration of the microscope system according to the present embodiment is not limited to such an example. For example, the microscope system may not necessarily include all of the configurations shown in fig. 10, or may include configurations not shown in fig. 10.

(1.2 example Process flow)

The configuration example of the information processing system according to the present embodiment has been described above. Next, a flow example of a series of processes accompanying fluorescence separation by the information processing apparatus 100 will be described with reference to fig. 11. Fig. 11 is a flowchart showing an example of the flow of a series of processes accompanying fluorescence separation by the information processing apparatus 100.

In step S1000, the fluorescence signal acquisition unit 112 of the information processing apparatus 100 acquires a fluorescence spectrum. More specifically, the fluorescently stained sample 30 is irradiated with a plurality of excitation lights having different excitation wavelengths, and the fluorescence signal acquisition unit 112 acquires a plurality of fluorescence spectra corresponding to each excitation light. Then, the fluorescence signal acquisition unit 112 stores the acquired fluorescence spectrum in the fluorescence signal storage unit 122.

In step S1004, the linking unit 131 generates a linked fluorescence spectrum by linking at least a part of the plurality of fluorescence spectra stored in the fluorescence signal storage unit 122 to each other in the wavelength direction. More specifically, the linking unit 131 generates one linked fluorescence spectrum by extracting data of a predetermined width in each fluorescence spectrum to include a maximum value of the fluorescence intensity of each of the plurality of fluorescence spectra, and linking the data to each other in the wavelength direction.

In step S1008, the separation processing unit 132 separates the linked fluorescence spectrum for each molecule (performs fluorescence separation). More specifically, the separation processing unit 132 separates the fluorescence spectrum of the link for each molecule by performing the processing described with reference to fig. 7.

In the subsequent process, for example, the image generating unit 133 generates image information using the separated fluorescence spectrum corresponding to one or more fluorescent molecules (or the separated autofluorescence spectrum corresponding to autofluorescent molecules), and the display unit 140 displays the image information on a display to present the image information to a physician.

<2 > second embodiment

The first embodiment of the present disclosure has been described above. Next, a second embodiment of the present disclosure will be described.

The information processing apparatus 100 according to the first embodiment has performed fluorescence separation processing using the linked autofluorescence reference spectrum (and the linked fluorescence reference spectrum) prepared in advance. On the other hand, the information processing apparatus 100 according to the second embodiment performs fluorescence separation processing using the actually measured linked autofluorescence reference spectrum.

More specifically, the spectrum extraction unit 1322 of the separation processing unit 132 according to the second embodiment extracts a linked autofluorescence reference spectrum of each autofluorescent substance from the linked autofluorescence spectrum, generates the linked autofluorescence spectrum by linking at least a part of the plurality of autofluorescence spectra to each other in the wavelength direction, obtains the plurality of autofluorescence spectra by irradiating a section with a plurality of excitation lights having different excitation wavelengths, and is the same as or similar to the sample 20. Then, the spectrum extraction unit 1322 performs fluorescence separation processing using the extracted linked autofluorescence reference spectrum and the linked fluorescence reference spectrum (the linked fluorescence reference spectrum is similar to the linked fluorescence reference spectrum of the first embodiment) as reference spectra.

Fig. 12 is a block diagram showing a more specific configuration example of the separation processing unit according to the present embodiment. As shown in fig. 12, the separation processing unit 132 according to the present embodiment has a configuration similar to that of the separation processing unit 132 described with reference to fig. 7 in the first embodiment.

In this configuration, instead of the linked autofluorescence reference spectra included in the sample information, the linked fluorescence spectra (also referred to as linked autofluorescence spectra) of the non-stained section (also referred to as non-stained sample) input from the linking unit 131 are input to the spectrum extraction unit 1322.

The spectrum extraction unit 1322 performs spectrum extraction processing by using the color separation result of the linked autofluorescence spectrum for the non-stained sample input from the linking unit 131 input from the first color separation unit 1321a, and adjusts the linked autofluorescence reference spectrum based on the result of the spectrum extraction processing, thereby improving the linked autofluorescence reference spectrum to obtain a more accurate color separation result. For example, similar to the first embodiment, non-Negative Matrix Factorization (NMF), Singular Value Decomposition (SVD), or the like may be used for the spectrum extraction process. Further, other operations may be similar to those of the separation processing unit 132 according to the first embodiment, and thus a detailed description thereof will be omitted.

Note that either of the non-stained section and the stained section may be used for the same or similar section as the sample 20 used for extracting the linked autofluorescence reference spectra. For example, in the case of using a non-stained section, a section before staining is used as a stained section, a section adjacent to the stained section, a section different from the stained section in the same piece (sampling from the same position as the stained section), a section in a different piece in the same tissue (sampling from a position different from the stained section), or the like can be used.

Further, in the case of using the stained section, by performing the fluorescence separation process by the method according to the third embodiment described later, it is also possible to obtain the color separation result of each molecule directly from the linked fluorescence spectrum without extracting the linked autofluorescence reference spectrum.

Here, principal component analysis (hereinafter referred to as "PCA") can be generally used as a method of extracting an autofluorescence spectrum from a non-stained section, but PCA is not applicable to a case where autofluorescence spectra linked to each other in the wavelength direction are used for processing as in the present embodiment. Therefore, the spectrum extraction unit 1322 according to the present embodiment extracts a linked autofluorescence reference spectrum from a non-stained section by performing non-negative matrix factorization (hereinafter, NMF) instead of PCA. Note that the reason why PCA is not suitable as a method of extracting a linked autofluorescence reference spectrum from a non-stained section will be described in detail later.

Fig. 13 is a diagram for describing an overview of NMF. As shown in fig. 13, the NMF decomposes a non-negative N-row M-column (N × M) matrix a into a non-negative N-row k-column (N × k) matrix W and a non-negative k-row M-column (k × M) matrix H. The matrix W and the matrix H are determined such that a mean square residual D between the matrix a and a product (W × H) of the matrix W and the matrix H becomes minimum. In the present embodiment, the matrix a corresponds to the spectrum before the extraction of the linked autofluorescence reference spectrum (N is the number of pixels, M is the number of wavelength channels), and the matrix H corresponds to the extracted linked autofluorescence reference spectrum (k is the number of linked autofluorescence reference spectra (in other words, the number of autofluorescent substances), M is the number of wavelength channels). Here, the mean square residual D is represented by the following equation (10). Note that "norm (D, 'fro')" refers to the Frobenius norm of the mean squared residual D.

[ equation 10]

Factorization of NMF uses an iterative method starting from random initial values of matrix W and matrix H. At NMF, the value of k (the number of linked autofluorescence reference spectra) is mandatory, but the initial values of matrix W and matrix H may be set as options instead of mandatory, and the solution is constant when setting the initial values of matrix W and matrix H. On the other hand, in the case where the initial values of the matrix W and the matrix H are not set, the initial values are randomly set, and the solution is not a constant.

The nature of the sample 20 differs according to the type of tissue used, the type of target disease, the attributes of the target person, the lifestyle of the target person, etc., and the autofluorescence spectrum of the sample 20 also differs. Therefore, as described above, the information processing apparatus 100 according to the second embodiment can realize more accurate fluorescence separation processing by actually measuring the linked autofluorescence reference spectrum of each sample 20.

Note that, as described above, the matrix a as an input to the NMF is a matrix including the same number of rows as the number of pixels N of the sample image (Hpix × Vpix) and the same number of columns as the number of wavelength channels M. Therefore, in the case where the number of pixels of the sample image is large or in the case where the number M of wavelength channels is large, the matrix a becomes a very large matrix, so that the calculation cost of the NMF increases and the processing time becomes long.

In this case, for example, as shown in fig. 14, by clustering the number of pixels N of the sample image (═ Hpix × Vpix) into a class of a specified number N (< Hpix × Vpix), redundancy in processing time due to enlargement of the matrix a can be suppressed.

In clustering, for example, similar spectra in the wavelength direction and the intensity direction in the sample image are classified into the same class. Thus, an image having a smaller number of pixels than that of the sample image is generated, and thus the scale of the matrix a' can be reduced using the image as an input.

(2.1. example Process flow)

Next, a flow example of a series of processes accompanying fluorescence separation by the information processing apparatus 100 according to the second embodiment will be described with reference to fig. 15. Fig. 15 is a flowchart showing an example of the flow of a series of processes accompanying fluorescence separation by the information processing apparatus 100 according to the second embodiment.

In step S1100 and step S1104, similarly to the flow example of the processing in the first embodiment (step S1000 and step S1004 of fig. 11), the fluorescence signal acquisition unit 112 acquires a plurality of fluorescence spectra corresponding to excitation light having different excitation wavelengths, and the linking unit 131 generates a linked fluorescence spectrum by linking at least a part of the plurality of fluorescence spectra to each other in the wavelength direction.

In step S1108, the spectrum extraction unit 1322 extracts a linked autofluorescence reference spectrum by performing NMF using the linked autofluorescence spectrum generated by linking at least a part of a plurality of autofluorescence spectra, which are obtained by irradiating the non-stained section with a plurality of excitation lights having different excitation wavelengths, to each other in the wavelength direction.

In step S1112, the color separation unit 1321 performs fluorescence separation processing using the linked autofluorescence reference spectrum and the linked fluorescence reference spectrum extracted as described above (the linked fluorescence reference spectrum is similar to that of the first embodiment) as reference spectra.

In the subsequent process, similarly to the first embodiment, for example, the image generating unit 133 generates image information using the separated fluorescence spectrum corresponding to one or more fluorescent molecules (or the separated autofluorescence spectrum corresponding to autofluorescent molecules), and the display unit 140 displays the image information on a display to present the image information to a physician.

(2.2. reason PCA is not suitable as a method for extracting a linked autofluorescence reference spectrum from a non-stained section)

The flow example of a series of processes accompanying fluorescence separation by the information processing apparatus 100 according to the second embodiment has been described above. Then, details of the reason why PCA is not suitable as a method of extracting a linked autofluorescence reference spectrum from a non-stained section will be described.

First, when the fluorescence spectrum of a certain pixel i in the number of pixels n is ai and the resolution is m for one excitation wavelength, the fluorescence spectrum ai is represented by the following equation (11) (mth order vector).

[ equation 11]

Similarly, fluorescence spectra of other excitation wavelengths are represented as bi, ci, and di as an m-order vector (here, a case where the number of types of excitation wavelengths is 4 is assumed as an example) at ai (ai1, ai2, to aim) (11). Then, a matrix in which these vectors with respect to all pixels (pixel 1 to pixel n) are integrated with each other is represented by the following equation (12) (n rows and 4m columns matrix P). Since the number of pixels is significantly (substantially) larger than the wavelength resolution, the rank of the matrix P represented by equation (12) is 4m at the maximum, and there are up to 4m eigenvalues and eigenvectors.

[ equation 12]

Here, the Singular Value Decomposition (SVD) represented by the following equation (13) may be performed on an n-row and m-column (n × m) real matrix a of rank k. U and V in equation (13) both represent singular matrices and form a normal matrix system (i.e.,^tU＝U^-1and UU^-11). Further, in the case where the real matrix a is a square matrix having different eigenvalues, U and V are eigenvectors.

[ equation 13]

A＝UD^tV (13)

The independent factors of the real matrix a may be analyzed by calculating singular (eigenvalue) vectors by performing Eigenvalue Decomposition (ED) or Singular Value Decomposition (SVD) on the real matrix a. In the case where the real matrix a is a square matrix and has different eigenvalues,^tthe eigenvalue of AA is the square of the eigenvalue of A, and^tthe feature vector of AA is equal to that of a (see the following equation (14), where a is VD in equation (14)^tV)。

[ equation 14]

^tAA＝^t(VD^tV)(VD^tV)＝V^tD^tVVD^tV＝V(DD)^tV (14) the spectrum obtained in the present embodiment is not a square matrix, but since the spectrum can be considered to be determined by a linear combination of elements configuring autofluorescence, it is considered that the spectrum can be convoluted into a square matrix by copying or linear conversion. Even in the case where there is an error,^tthe eigenvalues with AA ═ 0 are also the least squares solution of a. Thus, by obtaining^tAA, the independent components in the spectrum (eigenvectors) can be calculated. Further, equations (15) and (16) below are apparent because the singular value decomposition is established. However, in the case where the rank is not satisfied, L and R become subsets of the feature vector, so that all points cannot be represented.

[ equation 15]

Anm＝LnrRrm (15)

[ equation 16]

PCA is equivalent to obtaining eigenvalues and eigenvectors of the variance-covariance matrix of the data matrix. As expressed by equation (17) below, the variance-covariance matrix is a product between a matrix obtained by subtracting the average value from the data matrix and the transposed matrix. This is (product of data matrix and transpose-product of mean value per column).

[ equation 17]

[ equation 18]

As expressed by the following equations (19) to (23),^tthe feature vector of BB may become the feature vector of B to construct B, and^tBB and^tthe difference between AA is^taa (matrix of products of the averages of columns of A). Therefore, in the singular value decomposition of a, the eigenvectors of the arrangement points are obtained, and in the PCA, the eigenvectors representing the degree of change of the points are calculated(s) ((s))^tFeature vector sum of BB^tThe eigenvectors of AA are not equal to each other).

[ equation 19]

^tBBij＝∑(aki-ai)(akj-aj)＝∑(akiakj-aiaj)＝^tAAij-aiaj (19)

[ equation 20]

^tAAij＝∑akiakj (20)

[ equation 21]

a＝(a1，a2，～am) (21)

[ equation 22]

aiaj＝^taa (22)

[ equation 23]

^tBB＝^tAA-^taa (23)

At this time, in the case of a matrix such as the above equation (12) in which fluorescence spectra of each excitation wavelength are integrated with each other, in singular value decomposition, if they are independent from each other, they are not affected, but in PCA, a product term of the average values appears, and thus the eigenvectors are affected. Therefore, in order to perform PCA, each data set needs to be analyzed. As described above, in the case of performing processing using spectra linked to each other in the wavelength direction as in the present embodiment, PCA is not appropriate.

(2.3. application example)

Details of the reasons why PCA is not suitable as a method of extracting linked autofluorescence reference spectra from non-stained sections have been described above. Next, an application example according to the second embodiment will be described.

As described above, the spectrum extraction unit 1322 of the separation processing unit 132 according to the second embodiment extracts the linked autofluorescence reference spectrum by performing NMF using the linked autofluorescence spectrum generated by linking at least a part of the plurality of autofluorescence spectra to each other in the wavelength direction, the plurality of autofluorescence spectra being acquired by irradiating the non-stained section with the plurality of excitation lights having different excitation wavelengths. At this time, the spectrum extraction unit 1322 according to the application example can extract the linked autofluorescence reference spectrum by setting the initial value in the NMF (the initial value of the matrix H in fig. 13) using the autofluorescence spectrum acquired in advance according to the first embodiment or the like (more specifically, by setting the linked autofluorescence spectrum generated by linking at least part of the autofluorescence spectra with each other in the wavelength direction as the initial value in the NMF). Therefore, the spectrum obtained as a result of the separation is uniquely determined, and more accurate fluorescence separation can be performed.

<3. modified example >

The second embodiment of the present disclosure has been described above. Next, a modification of the present disclosure will be described.

Since the information obtained by the fluorescence separation process is the brightness (or fluorescence intensity) in the image information, a doctor may not be able to sufficiently perform quantitative analysis. More specifically, since a physician cannot obtain information such as the number of fluorescent molecules, the number of antibodies bound to the fluorescent molecules, and the like, it is difficult for the physician to compare the number of fluorescent molecules in a plurality of fluorescent substances, or to compare data imaged under different conditions from each other.

The present modification is made in view of the above-described problems, and the spectrum extraction unit 1322 according to this modification extracts the spectrum of each fluorescent substance from the linked fluorescence spectrum using the reference spectrum including the linked autofluorescence reference spectrum and the linked fluorescence reference spectrum calculated based on the number of fluorescent molecules or the number of antibodies bound to the fluorescent molecules. More specifically, the spectrum extraction unit 1322 according to this modification calculates the linked autofluorescence reference spectrum and the linked fluorescence reference spectrum of each fluorescent molecule or each antibody by dividing each linked autofluorescence reference spectrum and the linked fluorescence reference spectrum used in the above-described embodiment by the number of fluorescent molecules or the number of antibodies in the imaging element 1[ pixel ], and performs calculation with respect to the least square method (or weighted least square method) using the calculated linked autofluorescence reference spectrum and the linked fluorescence reference spectrum, thereby extracting the spectrum of each fluorescent substance from the linked fluorescence spectrum. Therefore, the separation processing unit 132 according to this modification can calculate the number of fluorescent molecules or the number of antibodies in the fluorescence-stained sample 30 as a result of the fluorescence separation processing.

Here, the calculation imaging element 1[ pixel ] will be described with reference to fig. 16]The number of fluorescent molecules (or the number of antibodies) in the sample. As shown in fig. 16, in the case where the imaging element and the specimen are disposed with the objective lens interposed therebetween, it is assumed to correspond to the imaging element 1[ pixel [ ]]The size of the bottom surface of the sample of (2) is 13/20[ mu ] m]×13/20[μm]. Then, when the thickness of the sample is assumed to be 10[ mu ] m]The volume of this cuboid [ m ]³]Using 13/20[ mu.m ]]×13/20[μm]×10[μm]Indicates (Note, volume [ L ]]Using 13/20[ mu.m ]]×13/20[μm]×10[μm]×10³Representation).

Then, when it is assumed that the concentration of the antibody (which may be, of course, the number of fluorescent molecules) included in the sample is uniform and 300[ nM ], the number of antibodies in the imaging element 1[ pixel ] is represented by the following equation (24).

[ equation 24]

As described above, the number of fluorescent molecules or the number of antibodies in the fluorescence-stained sample 30 is calculated as a result of the fluorescence separation process, so that a physician can compare the number of fluorescent molecules in a plurality of fluorescent substances, or compare data imaged under different conditions from each other. Further, although the brightness (or fluorescence intensity) is a continuous value, the number of fluorescent molecules or the number of antibodies is a discrete value. Therefore, the information processing apparatus 100 according to this modification can reduce the amount of data by outputting image information based on the number of fluorescent molecules or the number of antibodies.

Other configurations, operations, and effects may be similar to those in the above-described embodiments, and thus a detailed description thereof will be omitted here.

<4. third embodiment >

In the first and second embodiments described above, the case where the spectrum of each fluorescent substance is extracted from the linked autofluorescence spectrum by performing fluorescence separation processing using the linked autofluorescence reference spectrum (and the linked fluorescence reference spectrum) has been exemplified. On the other hand, in the third embodiment, a case where the fluorescence spectrum of each fluorescent substance is directly extracted from the stained section will be exemplified.

Fig. 17 is a block diagram showing a schematic configuration example of the separation processing unit according to the present embodiment. In the information processing apparatus 100 according to the present embodiment, the separation processing unit 132 is replaced with a separation processing unit 232 shown in fig. 17.

As shown in fig. 17, the separation processing unit 232 includes a color separation unit 2321, a spectrum extraction unit 2322, and a data set creation unit 2323.

The color separation unit 2321 performs color separation on the linked fluorescence spectrum of the stained section (also referred to as a stained sample) input from the linking unit 131 for each molecule.

The spectrum extraction unit 2322 is configured as follows: the autofluorescence spectrum is improved so that a more accurate color separation result can be obtained, and the linked autofluorescence reference spectrum included in the sample information input from the information storage unit 121 is adjusted so that a more accurate color separation result is obtained.

The dataset creation unit 2323 creates a dataset of the autofluorescence reference spectra from the spectrum extraction result input from the spectrum extraction unit 2322.

More specifically, the spectrum extraction unit 2322 performs spectrum extraction processing using non-Negative Matrix Factorization (NMF), Singular Value Decomposition (SVD), or the like on the linked autofluorescence reference spectrum input from the information storage unit 121, and inputs the result of the spectrum extraction processing to the dataset creation unit 2323. Note that, in the spectrum extraction process according to the present embodiment, the autofluorescence reference spectrum for each cell tissue and/or each type is extracted using, for example, a Tissue Microarray (TMA).

The data set creating unit 2323 creates a data set (hereinafter also referred to as an autofluorescence data set) necessary for the color separation unit 2321 to perform color separation processing, based on the autofluorescence reference spectrum for each cellular tissue and/or each type input from the spectrum extracting unit 2322, and inputs the created autofluorescence data set to the color separating unit 2321.

The color separation unit 2321 separates the linked fluorescence spectrum into a spectrum of each molecule by performing a color separation process using the linked fluorescence reference spectrum and the linked autofluorescence reference spectrum input from the information storage unit 121 and the autofluorescence data set for the linked fluorescence spectrum of the stained sample input from the linking unit 131 input from the data set creation unit 2323. Note that NMF or SVD may be used to perform the color separation process.

As the NMF performed by the color separation unit 2321 according to the present embodiment, for example, an NMF that is changed from the NMF (see fig. 13 and the like) as described in the second embodiment as follows when the autofluorescence spectrum is extracted from the non-stained section can be used.

That is, in the present embodiment, the matrix a corresponds to a plurality of sample images (N is the number of pixels, M is the number of wavelength channels) acquired from a stained section, the matrix H corresponds to the fluorescence spectrum of each extracted fluorescent substance (k is the number of fluorescence spectra (in other words, the number of fluorescent substances) and M is the number of wavelength channels), and the matrix W corresponds to the image of each fluorescent substance after fluorescence separation. Note that matrix D is the mean square residual.

Further, in the present embodiment, similarly to the second embodiment, the initial value of the NMF may be random. However, in the case where each result of NMF execution is different, an initial value needs to be set to prevent variation in the result.

Fig. 18 to 22 are diagrams showing an example of a sample image input to the matrix a in the present embodiment, and fig. 23 to 29 are diagrams showing an example of a fluorescence separation image acquired by the NMF as the matrix W in a case where the sample image shown in fig. 18 to 22 is input. Note that, in each of fig. 18 to 22, for the sake of simplifying the description, a case where the specimen 20 has been stained with a single fluorescent reagent 10 is shown. Further, it is assumed that fluorescence spectra of a total of eight kinds of fluorescent pigments of Archidonic acid, catalase, collagen, FAD, hemoglobin, NADPH, ProlongDiamond, and CK are given as initial values of NMF.

When the NMF is solved with the sample image acquired at each of the five excitation wavelengths (the number of wavelength channels (M) ═ 5) as shown in fig. 18 to 22 as a matrix a, seven fluorescence separation images as shown in fig. 23 to 29 are acquired as a matrix W, and the respective fluorescence spectra are acquired as a matrix H.

Note that in the case where the fluorescence separation process has been performed using an algorithm that changes the order of the respective spectra according to a calculation algorithm or an algorithm (e.g., NMF) that needs to change the order of the spectra to accelerate the process or improve the convergence of the result, it is possible to specify to which fluorescent pigment each fluorescence spectrum obtained as the matrix H corresponds by, for example, obtaining a pearson product-moment correlation coefficient (or cosine similarity) of each of all combinations.

Further, in the case where a default function (NMF) of MATLAB (registered trademark) has been used, even if an initial value is given, the order is changed and output is performed. This can be fixed by a self-function, but even if the order is changed due to the use of a default function, the correct combination of substance and fluorescence spectra can be obtained by using the pearson product-moment correlation coefficient (or cosine similarity), as described above.

As described above, by using the configuration in which the NMF is solved with the sample image acquired from the stained section as the matrix a, the fluorescence spectrum of each fluorescent substance can be directly extracted from the stained section without processes such as imaging of a non-stained section, generation of a linked autofluorescence reference spectrum, and the like. Therefore, the time and work cost required for the fluorescence separation process can be significantly reduced.

Further, in the present embodiment, the fluorescence spectrum of each fluorescent substance is extracted from the sample image obtained from the same stained section, and therefore, more accurate fluorescence separation results can be obtained as compared with, for example, the case where the autofluorescence spectrum obtained from a non-stained section different from the stained section is used.

Note that in the present embodiment, when the fluorescence spectrum of each fluorescent substance is extracted, a linked fluorescence spectrum may be used, or may not be linked. That is, in the present embodiment, the link unit 131 may or may not generate a linked fluorescence spectrum. In the case where the linked fluorescence spectrum is not generated by the linking unit 131, the extraction unit of the separation processing unit 132 performs processing for extracting the fluorescence spectrum of each fluorescent substance from the plurality of fluorescence spectra acquired by the fluorescence signal acquisition unit 112.

<5. fourth embodiment >

In the above-described third embodiment, the following method may be mentioned as a method of enhancing quantitative properties (e.g., with respect to the concentration of a coloring pigment, etc.).

Fig. 30 is a flowchart for describing an NMF flow according to the fourth embodiment. Fig. 31 is a diagram for describing a flow of processing in the first cycle of the NMF shown in fig. 30.

As shown in fig. 30, in the NMF according to the present embodiment, first, the variable i is reset to zero (step S401). The variable i represents the number of repetitions of factorization in NMF. Thus, the matrix H shown in (a) of fig. 31₀Corresponding to the initial values of matrix H. Note that, in the present example, the position of the stained fluorescence spectrum in the matrix H is the bottom row for clarity, but is not limited thereto, and may be variously changed to the top row, the middle row, and the like.

Next, in the NMF according to the present embodiment, similarly to the normal NMF, the non-negative N rows and M columns (N × M) matrix a is divided by the non-negative N rows and k columns (N × k) matrix W_iObtaining a non-negative k rows and M columns (k × M) matrix H_i+1(step S402). Thus, for example, in the first cycle, a matrix H as shown in (b) of fig. 31 is obtained₁。

Next, the matrix H obtained in step S402_i+1One row of the fluorescent staining spectra in (a) is replaced with the initial value of the fluorescent staining spectrum, i.e. matrix H₀One line dyeing inFluorescence spectrum (step S403). That is, in the present embodiment, the fluorescence staining spectrum in the matrix H is fixed to the initial value. For example, in the first cycle, the matrix H can be used₀Bottom row replacement matrix H in₁The bottom row in (a) of FIG. 31 shows the fluorescence spectrum of the dye fixed.

Next, in the NMF according to the present embodiment, by dividing the matrix a by the matrix H obtained in step S403_i+1To obtain a matrix W_i+1(step S404).

Thereafter, in the NMF according to the present embodiment, similarly to the normal NMF, it is determined whether or not the mean square residual D satisfies the predetermined branch condition (step S405), and in the case where the predetermined branch condition is satisfied (yes in step S405), the NMF uses the matrix H finally obtained_i+1And W_i+1Ending as a solution. On the other hand, in the case where the predetermined branching condition is not satisfied (no in step S405), the variable i is incremented by 1 (step S406), and then the process returns to step S402, and the next loop is executed.

Fig. 32 is a diagram showing an example of the initial value of the dyeing fluorescence spectrum. Fig. 33 is a diagram showing an example of a staining fluorescence spectrum after NMF is performed according to the present embodiment. As shown in fig. 32 and 33, it can be seen that the fluorescence spectrum of the staining remains a spectrum equivalent to the initial value even in the case where the NMF according to the present embodiment is performed.

Further, fig. 34 is a diagram showing an example of a spectrum of a fluorescent substance extracted by the method without using a non-stained sample according to the present embodiment, and fig. 35 is a diagram showing an example of a spectrum of a fluorescent substance extracted in the case of using a non-stained sample. Note that, in fig. 34 and 35, a case where CD8 has been used as a labeled antibody and Alexa Fluor 680 has been used as a fluorescent pigment is exemplified. As shown in fig. 34 and 35, according to the present embodiment, the spectrum of the fluorescent substance can be extracted with the same accuracy as in the case of using a non-stained specimen.

As described above, in the first method, in spectral extraction and color separation of a multi-stained pathological section image (specimen image), a stained specimen can be color-separated directly using NMF while ensuring the quantitative property of stained fluorescence, that is, while maintaining the spectrum of stained fluorescence, without imaging a non-stained specimen of the same tissue section for autofluorescence spectral extraction. Thus, for example, accurate color separation can be achieved as compared with the case where other samples are used. Further, the time and labor for imaging other samples and the like can be reduced.

Note that minimizing D ═ a-WH is used²The method of the recurrence formula of (c), the method using the quasi-newton method (Davidon-Fletcher-powell (dfp) method), the Broyden-Fletcher-golden farb-shanno (bfgs) method, and the like can be considered as a method of minimizing the mean square residual D. In these cases, the following method can be considered as a method of fixing the dyeing fluorescence spectrum to the initial value.

5.1. Method for fixing dyed fluorescence spectra in minimum mean square residual D using recursion formula

Using minimum D ═ A-WH²In the method of minimizing the mean square residual D by the recursive formula of (a), a loop process is performed which repeats a step including a multiplicative update formula represented by the following equations (25) and (26). Note that equations (25) and (26), a ═ a_i,j)_N×M，H＝(h_i,j)_k×M，W＝(w_i,j)_N×k. In addition to this, the present invention is,^th and^tw is the transpose of the sub-matrices h and w, respectively.

[ equation 25]

[ equation 26]

In such a loop process, in order to fix the stained fluorescence spectrum to the initial value, a method of inserting a step of executing the following equation (27) between a step of executing equation (25) and a step of executing equation (26) may be used. Note that equation (27) represents w corresponding to the update_i,j ^k+1Submatrix of the stained fluorescence spectrum in (1) and submatrix w_i,j(part) ^kOverwrite, the submatrix is the initial value of the stained fluorescence spectrum.

[ equation 27]

w_i，j ^k+1←w_i，j(part) ^k (27)

5.2. Method for fixing dyeing fluorescence spectrum in minimum mean square residual D by using DFP method, BFGS method and the like

Further, in the method of minimizing the mean square residual D using the DFP method, the BFGS method, or the like, when the mean square residual D of the minimization target is D (x), and x is a coordinate (at the k-th update, x_k＝(a1，a2，...an)_k) D (x) is minimized by the following steps. In the following steps, B denotes a Hessian matrix.

By x_k+1＝x_k-αB_k ^-1D’(x_k) Updating coordinates

New coordinate x_k+1Gradient displacement of

From y_k＝D’(x_k+1)–D’(x_k) Updating Hessian inverse matrix B_k+1 ^-1

Various methods (e.g., a DFP method represented by the following equation (28), a BFGF method represented by the following equation (29), etc.) may be applied to the update of the Hessian matrix Bk + 1.

[ equation 28]

[ equation 29]

Among such methods of minimizing the mean square residual D using the DFP method, the BFGS method, and the like, there are several methods as a method of fixing arbitrary coordinates, that is, a method of fixing the stained fluorescence spectrum to an initial value. For example, the dyed fluorescence spectrum may be fixed to the initial value by a method of performing the following process (1) or process (2) at the time of updating the coordinates.

(1)-αB_k ^-1D’(x_k) 0, i.e. partial differential D' (x)_k) Is replaced by zero

(2) Computing x after updating coordinates_k+1After that, the coordinate x to be obtained is forced_k+1Is replaced by x_k(or x)_kA part of)

<6 > fifth embodiment

Next, a sixth embodiment of the present disclosure will be described in detail with reference to the accompanying drawings.

Methods of separating and analyzing certain data into elements of configuration data and coefficients of elements, including machine learning, are widely used. As a method of decomposing data into elements (bases or spectra) (which are referred to as spectra in the present disclosure) and coefficients, there are various methods such as eigenvalue decomposition, singular value decomposition, non-Negative Matrix Factorization (NMF), and the like described in the above embodiments. It can be said that NMF for non-negative data in particular has a high similarity between the obtained solution and the actual spectrum (e.g. absorption spectrum, fluorescence spectrum of a material, etc.) and is advantageous in interpreting the data because both the spectrum and the coefficients are non-negative values.

As described above, the NMF is to represent the data matrix a by the sum of the product of the spectrum S (corresponding to the matrix H of fig. 13) and the coefficient C (corresponding to the matrix W of fig. 13) and the error f (corresponding to the mean square residual D of fig. 13), and to perform matrix factorization under non-negative constraint conditions so that the error (f ═ a-sxc @ is²) The method becomes minimal and has a feature that it is easy to approximate data by the minimum spectrum (low rank approximation). In this NMF, a calculation method using a recurrence formula is established, and a combination of S and C that minimizes the error f can be obtained by repeatedly calculating the following equation (30).

[ equation 30]

Note that Xij and Yij are values represented by the following equation (31), respectively.

[ equation 31]

Further, in equation (31), the matrix^tC and matrix^tS is the transpose of matrix C and matrix S, respectively.

Here, a case is considered where the matrix a has p data (corresponding to the number N of pixels of fig. 13) of the number w (corresponding to the number M of wavelength channels of fig. 13) and these data are approximated by N spectra of the number w of element components. In this case, the matrix a may be represented by the following equation (32).

[ equation 32]

A(p，w)＝S(n，w)×C(p，n) (32)

In equation (32), the matrix a (p, w) needs to be referred to when calculating Xij and Yij given by equation (31) above. Thus, every iteration of the calculation requires the calculation of all points p.

If the data to be analyzed is small-scale data, there is little problem in performing calculation for all points p for each iteration, but in the case of large-scale data having a very large number of data points, this becomes a factor of increasing the calculation time. Further, in the case where all p data cannot be expanded in a memory (e.g., RAM 903 in fig. 42 described later), a problem of frequent access to an external storage device (e.g., storage device 908 in fig. 42) may occur, so that the processing time becomes more redundant.

On the other hand, the inventors have found that the data can be obtained by a Gram matrix to the data matrix a^tAA performs a non-negative decomposition instead of the data matrix a to be decomposed to obtain a spectrum S.

Thus, in the present embodiment, the data matrix A is converted into a Gram matrix^tAA and to Gram matrix^tAA performs a non-negative decomposition to obtain a spectrum S as a solution. By converting the data matrix A into a Gram matrix^tAA, can make the treatment symmetricalThe matrix becomes a square matrix. Thus, for example, a data matrix a in which the number N of pixels is very large relative to the number M of wavelength channels is converted into an M × MGram matrix^tAA, and thus the number of data points can be significantly reduced to shorten the computation time and significantly reduce the amount of memory required for the computation. As a result, highly efficient analysis can be achieved.

(6.1. processing overview of processing Unit)

The information processing apparatus according to the present embodiment has, for example, a configuration similar to the information processing apparatus 100 (see fig. 1) according to the above-described embodiment, and the processing unit 130 (e.g., the separation processing unit 132) performs the following operations.

First, the processing unit 130 according to the present embodiment performs non-negative factorization or singular value decomposition of the data matrix a into a ═ sxc by calculating the Gram matrix of the matrix a in advance^tAA and Gram matrix to be calculated^tAA performs non-negative decomposition into^tSpectrum S was obtained with AA ═ sxe.

Secondly, the Gram matrix is calculated^tIn the course of AA, the processing unit 130 according to the present embodiment convolves each Gram matrix as in the following equation (33) by using a subset in which a (p, w) ═ a1(p1-pn1, w) + a2(pn1+1-pm, w) +. + Ao (pm +1-p, w)^tAqAq (q is an integer greater than or equal to 1 and less than or equal to n) to obtain a Gram matrix^tAA。

[ equation 33]

^tAA＝^tA1A1+^tA2A2+...^tAnAn (33)

Thirdly, the coefficient C is obtained by solving a ═ sxc using a spectrum S obtained by non-negative decomposition with respect to the above Gram matrix.

(6.2. configuration example of measurement System)

Next, a configuration example of a measurement system in the information processing apparatus 100 according to the present embodiment will be described. Fig. 36 is a diagram showing an example of a measurement system of the information processing system according to the present embodiment. Note that in fig. 36, an example of a measurement system when imaging a wide field of view of a fluorescence-stained sample 30 (or a sample 20 that is a non-stained sample), for example, full slide imaging (WSI) or the like, is shown. However, the measurement system according to the present embodiment is not limited to the measurement system shown in fig. 36, and various modifications may be made as long as it is a measurement system capable of acquiring image data (hereinafter referred to as wide-field image data) of a sufficient resolution of the entire imaging region or the region of interest, for example, a measurement system that images the entire imaging region or a necessary region (also referred to as a region of interest) at a time, a measurement system that acquires an image of the entire imaging region or the region of interest by line scanning, or the like.

As shown in fig. 36, the measurement system according to the present embodiment includes, for example, the information processing apparatus 100, an XY stage 501, an excitation light source 510, a beam splitter 511, an objective lens 512, a beam splitter 513, and a photodetector 514.

The XY stage 501 is a stage on which the fluorescent-stained sample 30 (or the sample 20) as an analysis target is placed, and may be, for example, a stage movable on a plane (XY plane) parallel to a placement surface of the fluorescent-stained sample 30 (or the sample 20).

The excitation light source 510 is a light source for exciting the fluorescent-stained specimen 30 (or the specimen 20), and emits a plurality of excitation lights having different wavelengths, for example, along a predetermined optical axis.

The beam splitter 511 includes, for example, a dichroic mirror or the like, reflects the excitation light from the excitation light source 510, and transmits the fluorescence from the fluorescence-stained sample 30 (or sample 20).

The objective lens 512 irradiates the fluorescence-stained sample 30 (or the sample 20) on the XY stage 501 with the excitation light reflected by the beam splitter 511.

The beam splitter 513 is configured by using one or more prisms, lenses, or the like, and scatters the fluorescence emitted from the fluorescence-stained sample 30 (or the sample 20) and transmitted through the objective lens 512 and the beam splitter 511 in a predetermined direction.

The photodetector 514 detects the light intensity of each wavelength of the fluorescence scattered by the spectroscope 513, and inputs a fluorescence signal (fluorescence spectrum and/or autofluorescence spectrum) obtained by the detection to the fluorescence signal acquisition unit 112 of the information processing apparatus 100.

In the configuration as described above, in the case where the entire imaging region exceeds a region that can be imaged once (hereinafter referred to as a field of view), for example, WSI, imaging of each field of view s is performed in turn by moving the field of view by moving the XY stage 501 for each imaging. Then, by tiling image data obtained by imaging each field of view (hereinafter referred to as field-of-view image data), wide-field image data of the entire imaging region is generated. The generated wide-field image data is stored in, for example, the fluorescence signal storage unit 122. Note that the tiling of the sight field image data may be performed by the acquisition unit 110 of the information processing apparatus 100, may be performed by the storage unit 120 of the information processing apparatus 100, or may be performed by the processing unit 130 of the information processing apparatus 100.

Then, the processing unit 130 according to the present embodiment acquires the coefficient C, that is, the fluorescence separation image of each fluorescent molecule (or the autofluorescence separation image of each autofluorescent molecule) by performing the above-described processing on the obtained wide-field image data.

(6.3. operation example)

Next, an operation example of the information processing apparatus 100 according to the present embodiment will be described. Note that the following description will focus on the operation of the processing unit 130.

Fig. 37 is a flowchart showing an example of the operation of the processing unit according to the present embodiment. Further, fig. 38 to 40 are diagrams for describing processing executed by the processing unit in each step of fig. 37.

As shown in fig. 37, first, the processing unit 130 according to the present embodiment generates wide-field image data of the entire imaging region (see, for example, wide-field image data a of fig. 38) by tiling field-of-view image data obtained by imaging each field of view (step S2001).

Next, the processing unit 130 acquires unit image data (for example, unit image data Aq in fig. 38 (q is an integer of 1 or more and n or less)) which is a part of the wide-field image data a from the wide-field image data a (step S2002). The unit image data Aq may be variously changed as long as it is image data of an area smaller than the wide-field image data a, for example, image data corresponding to one field of view, image data of a preset size, and the like. Note that the image data of a preset size may include image data of a size determined by the amount of data that the information processing apparatus 100 can process at one time.

Next, as shown in fig. 38, the processing unit 130 multiplies the data matrix (for clarity of description, the data matrix is referred to as a1) of the acquired unit image data Aq (for clarity of description, referred to as unit image data a1 in the following description) by the transpose matrix^tA1 generating a Gram matrix of unit image data A1^tA1a1 (step S2003).

Next, the processing unit 130 determines whether the Gram matrix of all the unit image data a1 to An has been completed^tA1A1 to^tAn is generated (step S2004), and steps S2002 to S2004 are repeatedly performed until the Gram matrix of all the unit image data a1 to An is completed^tA1A1 to^tGeneration of an (no in step S2004).

On the other hand, when the Gram matrix for all the unit image data A1 to An is completed^tA1A1 to^tUpon generation of an (yes in step S2004), the processing unit 130 obtains a Gram matrix from the obtained Gram matrix by using, for example, a least square method (or a weighted least square method)^tA1A1 to^tAn calculates an initial value of the coefficient C (step S2005).

Next, the processing unit 130 passes the generated Gram matrix^tA1A1 to^tAnn is added to calculate a Gram matrix of wide-field image data A^tAA (step S2006). Specifically, as described above, processing unit 130 convolves each Gram matrix as in equation (33) above by using a subset of a (p, w) ═ a1(p1-pn1, w) + a2(pn1+1-pm, w) +. + Ao (pm +1-p, w)^tAqAq (q is an integer of 1 or more and n or less) to obtain a Gram matrix^tAA。

Next, as shown in FIG. 39, the processing unit 130 calculates a Gram matrix by calculating^tNon-negative decomposition of AA^tSpectrum S is obtained by AA ═ sxe (step S2007). Note that the matrix E corresponds to a separate image fluorescence-separated from the wide-field image data a.

ThereafterAs shown in fig. 40, the processing unit 130 uses the sum of the sum matrix for the Gram matrix by the NMF through the least square method (or the weighted least square method)^tThe spectrum S obtained by AA is solved for a ═ sxc to acquire a coefficient C, that is, a fluorescence separation image of each fluorescent molecule (or an autofluorescence separation image of each autofluorescent molecule) (step S2008), and the present operation is ended thereafter.

Note that in the NMF of step S2007, non-negative factorization of data may be performed with a fixed specific spectrum.

(6.4.1. first modification)

Note that the case where the entire imaging region is set as the processing target region has been exemplified in fig. 37 to 40, but the processing target region is not limited thereto, and may be set as a region (region of interest) narrower than the entire imaging region. The region of interest may be, for example, a region of the projection analysis target, for example, a region where the fluorescence-stained specimen 30 (or the specimen 20) exists in the wide-field image data a, or the like. Further, for example, morphological information of the fluorescently stained sample 30 or sample 20 (e.g., cells, tissues, etc.) can be used to set the region of interest. Note that the morphological information may be a bright field image, a non-stained image, and staining information of the same tissue mass, or may be, for example, an expression map of the target in the sample 20. Further, the morphological information may be information generated using a technique such as segmentation (acquisition and labeling of an area in units of one pixel) in an image recognition technique of machine learning.

Fig. 41 is a flowchart showing an example of the operation of the processing unit according to the first modification of the present embodiment. As shown in fig. 41, first, the processing unit 130 according to the present first modification generates wide-field image data of the entire imaging area by tiling field-of-view image data obtained by imaging each field of view (step S2101). In the present first modification, the resolution of the wide-field image data a may be lower than that of image data as a processing target (for example, high-resolution image data described later).

Next, the processing unit 130 sets a monitoring area as a processing target area in the wide-field image data a (step S2102). As described above, the setting of the region of interest may be performed based on, for example, morphological information or the like. However, the setting of the region of interest may be automatically performed by the processing unit 130 based on the morphological information or the like, or may be manually performed by the user.

Next, the processing unit 130 requests, for example, the control unit 150 to acquire high resolution image data of the region of interest (step S2103). In response to such a request, the control unit 150 acquires the high-resolution image data of the region of interest by controlling the above-described measurement system (see fig. 36), the acquisition unit 110, and the storage unit 120. Note that the region of interest may be a wider range than one field of view.

Next, the processing unit 130 generates a Gram matrix of each unit image data Aq acquired from the high resolution image data of the region of interest by performing operations similar to, for example, steps S2002 to S2004 of fig. 37^tAqaqaqq (steps S2104 to S2106).

Next, similarly to step S2005 of fig. 37, for example, the processing unit 130 obtains the Gram matrix from the obtained Gram matrix by using the least square method (or the weighted least square method)^tA1A1 to^tAn calculates an initial value of the coefficient C (step S2107).

Next, the processing unit 130 passes the generated Gram matrix^tA1A1 to^tAnn is added to calculate a Gram matrix of wide-field image data A^tAA (step S2108), calculating the Gram matrix^tAA performs a non-negative decomposition into^tThe spectrum S is obtained (step S2109) with AA ═ sxe, and the coefficient C is acquired, that is, similarly to, for example, steps S2006 to S2008 of fig. 37, by using the matrix for Gram^tThe spectrum S obtained by the NMF of AA is solved by the least square method (or weighted least square method) for a ═ sxc (step S2110), a fluorescence separation image of each fluorescent molecule (or an autofluorescence separation image of each autofluorescent molecule) is acquired, and then the present operation is ended. Note that in the NMF of step S2109, non-negative factorization of data may be performed with a fixed specific spectrum.

(6.4.2. second modification)

Note that in the operation example shown in fig. 37 and its modification (fig. 41), the first example has been illustratedA case where the wide-field image data of the entire imaging region or the high-resolution image data of the entire region of interest is acquired first, and then the unit image data that is a part of the wide-field image data or the high-resolution image data is acquired and sequentially processed, but the present disclosure is not limited thereto, and all or part of, for example, the wide-field image data or the high-resolution image data may also be executed in a pipeline processing manner. Specifically, for example, regarding the Gram matrix until each unit image data is generated^tThe processing of aqaqaqaqq (e.g., steps S2001 to S2004 in fig. 37 or steps S2103 to S106 in fig. 41) may generate a Gram matrix of each unit image data by using the image data of each field of view output from the measurement system (see fig. 36) as unit image data and performing the above processing in response to the input of the unit image data^tAqAq。

(6.5. Effect)

With regard to the effect expected by the present embodiment, hereinafter, with regard to the procedure up to obtaining a solution of the spectrum S and coefficient C of a ═ sxc by the NMF, the case where the NMF of the matrix a (p, w) has been performed (case 1) and the Gram matrix obtained from the matrix a has been performed will be described by way of example^tNMF of AA (w, w) (case 1).

In a case where the calculation times of the four arithmetic operations are assumed to be almost equal to each other and the overhead is not assumed to be taken into consideration, when the calculation amount of the NMF loop is calculated for each of the

cases

1 and 2, it is estimated that the passing through the Gram matrix in case 2 is compared with the case 1 where the matrix a is subjected to the NMF^tThe AA treatment rate can be increased by about 6000 times.

Further, in the case where calculation of 10 to 100 unit image data with wide-field image data (e.g., WSI) has been considered, the Gram matrix of the wide-field image data is calculated by convolving the Gram matrix of each unit image data as compared with case 1 where the wide-field image data a is subjected to NMF as it is^tIn case 2 of AA, the estimated processing speed may be increased by about 60000 to 600000 times.

Further, in the case where each unit image data is 1024 × 1024 image data and the number (M) of wavelength channels is 100 dotsIn contrast to case 1, where matrix A (p, w) is subjected to NMF, the Gram matrix at matrix A^tIn case 2, where AA (w, w) is subjected to NMF, the maximum amount of memory required to expand the data may be reduced to about 1/10000. In addition, in the case where ten to hundreds of unit image data have been considered, the storage amount may be further reduced, for example, the storage amount may be reduced to 1/100000 to 1/1000000.

<7. hardware configuration example >

Modifications of the present disclosure have been described above. Next, a hardware configuration example of the information processing apparatus 100 according to each embodiment and modification will be described with reference to fig. 42. Fig. 42 is a block diagram showing an example of the hardware configuration of the information processing apparatus 100. Various processes of the information processing apparatus 100 are realized by cooperation between software and hardware described below.

As shown in fig. 42, the information processing apparatus 100 includes a Central Processing Unit (CPU)901, a Read Only Memory (ROM)902, a Random Access Memory (RAM)903, and a host bus 904 a. Further, the information processing apparatus 100 includes a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 911, a communication device 913, and a sensor 915. The information processing apparatus 100 may have a processing circuit such as a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), or the like, instead of the CPU 901 or together with the CPU 901.

The CPU 901 functions as an arithmetic processing device and a control device, and generally controls operations in the information processing apparatus 100 according to various programs. Further, the CPU 901 may be a microprocessor. The ROM 902 stores programs, operation parameters, and the like used by the CPU 901. The RAM 903 temporarily stores programs used in execution of the CPU 901, parameters appropriately changed in execution, and the like. For example, the CPU 901 may embody at least the processing unit 130 and the control unit 150 of the information processing apparatus 100.

The CPU 901, the ROM 902, and the RAM 903 are connected to each other by a host bus 904a including a CPU bus and the like. The host bus 904a is connected to an external bus 904b, for example, a peripheral component interconnect/interface (PCI) bus or the like, through a bridge 904. Note that the host bus 904a, the bridge 904, and the external bus 904b do not necessarily need to be configured separately, and the functions of the host bus 904a, the bridge 904, and the external bus 904b may be mounted on a single bus.

The input device 906 is implemented by, for example, a device such as a mouse, a keyboard, a touch panel, buttons, a microphone, switches, a lever, or the like, to which a doctor inputs information. Further, the input device 906 may be, for example, a remote control device using infrared rays or other electric waves, or may be an external connection device corresponding to the operation of the information processing apparatus 100, such as a mobile phone, a Personal Digital Assistant (PDA), or the like. Further, the input device 906 may include, for example, an input control circuit or the like that generates an input signal based on information input by a physician using the above-described input device and outputs the generated input signal to the CPU 901. The physician may input various data to the information processing apparatus 100 or instruct the information processing apparatus 100 to perform processing operations by operating the input device 906. For example, the input device 906 may contain at least the operation unit 160 of the information processing apparatus 100.

The output device 907 is a device capable of visually or audibly notifying the acquired information to the physician. Such devices include display devices (e.g., Cathode Ray Tube (CRT) display devices, liquid crystal display devices, plasma display devices, Electroluminescence (EL) display devices, lamps, etc.), sound output devices (e.g., speakers, headphones, etc.), printer devices, and the like. For example, the output device 907 may embody at least the display unit 140 of the information processing apparatus 100.

The storage 908 is a device for storing data. The storage device 908 is implemented by, for example, a magnetic storage unit device such as a Hard Disk Drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like. The storage device 908 may include a storage medium, a recording device that records data in the storage medium, a reading device that reads data from the storage medium, a deleting device that deletes data recorded in the storage medium, and the like. The storage device 908 stores programs and various data executed by the CPU 901, various data acquired from the outside, and the like. For example, the storage 908 may embody at least the storage unit 120 of the information processing apparatus 100.

The drive 909 is a reader/writer of a storage medium, and is embedded in or externally mounted on the information processing apparatus 100. The drive 909 reads information recorded in a mounted removable storage medium (e.g., a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc.) and outputs the read information to the RAM 903. In addition, the drive 909 may write information to a removable storage medium.

The connection port 911 is an interface to be connected to an external device, and is a connection port of an external device capable of transmitting data by, for example, a Universal Serial Bus (USB) or the like.

The communication device 913 is, for example, a communication interface including a communication device or the like for connecting to the network 920. The communication device 913 is, for example, a communication card or the like for wired or wireless Local Area Network (LAN), Long Term Evolution (LTE), bluetooth (registered trademark), or wireless usb (wusb). Further, the communication device 913 may be a router for optical communication, a router for Asymmetric Digital Subscriber Line (ADSL), a modem for various communications, or the like. The communication device 913 may transmit or receive signals or the like to or from the internet or another communication device according to a predetermined protocol, e.g., transmission control protocol/internet protocol (TCP/IP) or the like.

In the present embodiment, the sensor 915 includes a sensor (e.g., an imaging element, etc.) capable of acquiring a spectrum, and may include other sensors (e.g., an acceleration sensor, a gyro sensor, a geomagnetic sensor, a pressure sensor, a sound sensor, a distance measurement sensor, etc.). For example, the sensor 915 may embody at least the fluorescent-signal acquisition unit 112 of the information processing apparatus 100.

Note that the network 920 is a wired or wireless transmission path of information transmitted from a device connected to the network 920. For example, the network 920 may include a public network (e.g., the internet, a telephone network, a satellite communication network, etc.), various local area networks including ethernet (registered trademark), a Wide Area Network (WAN), and the like. Further, network 920 may include a private line network, such as an internet protocol-virtual private network (IP-VPN), or the like.

The above has described the hardware configuration example capable of realizing the function of the information processing apparatus 100. Each component described above may be implemented using a general-purpose member, or may be implemented by hardware dedicated to the function of each component. Therefore, in executing the present disclosure, the hardware configuration to be used may be appropriately changed according to the technical level.

Note that a computer program for realizing each function of the information processing apparatus 100 as described above may be created and installed in a Personal Computer (PC) or the like. Further, a computer-readable recording medium storing such a computer program may be provided. The recording medium includes, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, and the like. Further, the above-described computer program may be distributed via, for example, a network without using a recording medium.

<8. conclusion >

As described above, the information processing apparatus 100 according to the first embodiment of the present disclosure irradiates the fluorescently stained sample 30 with the plurality of excitation lights having different wavelengths, acquires the plurality of fluorescence spectra corresponding to each of the plurality of excitation lights, corrects the plurality of fluorescence spectra based on the intensity of the excitation light, and links at least a part of the plurality of fluorescence spectra to each other in the wavelength direction to generate the linked fluorescence spectra. Then, the information processing apparatus 100 extracts the spectrum of each fluorescent substance from a reference spectrum including a linked autofluorescence reference spectrum in which the spectra of the autofluorescent substance are linked to each other in the wavelength direction and a linked fluorescence reference spectrum in which the spectra of the fluorescent substances are linked to each other in the wavelength direction. Then, the information processing apparatus 100 separates the linked fluorescence spectrum of each molecule using the extracted spectrum of each fluorescent substance.

In this way, the information processing apparatus 100 can output a unique spectrum as a separation result (the separation result is not different for each excitation wavelength) by performing fluorescence separation processing using the reference spectrum linked in the wavelength direction. Thus, the correct spectrum can be more easily obtained by the physician. Further, a reference spectrum (linked autofluorescence reference spectrum) regarding autofluorescence used for separation is automatically acquired, and fluorescence separation processing is performed, so that a physician does not need to extract a spectrum corresponding to autofluorescence from an appropriate space of a non-stained section.

Further, the information processing apparatus 100 according to the second embodiment of the present disclosure performs fluorescence separation processing using the linked autofluorescence reference spectrum actually measured for each sample 20. Therefore, the information processing apparatus 100 can realize more accurate fluorescence separation processing.

Further, the information processing apparatus 100 according to the modified example of the present disclosure separates the linked fluorescence spectrum of each fluorescent substance using a reference spectrum including a linked autofluorescence reference spectrum and a linked fluorescence reference spectrum calculated based on the number of fluorescent molecules or the number of antibodies bound to the fluorescent molecules. Therefore, as a result of the fluorescence separation process, the information processing apparatus 100 can calculate the number of fluorescent molecules or the number of antibodies in the fluorescence-stained specimen 30.

Further, the information processing apparatus 100 according to the third embodiment of the present disclosure solves the NMF using the sample image acquired from the stained section as the matrix a. Therefore, the fluorescence spectrum of each fluorescent substance can be directly extracted from the stained section, while significantly reducing the time and work cost required for the fluorescence separation process. Further, in the third embodiment of the present disclosure, the fluorescence spectrum of each fluorescent substance is extracted from the sample image obtained from the same stained section, and therefore, more accurate fluorescence separation results can be obtained as compared with, for example, the case where the autofluorescence spectrum obtained from a non-stained section different from the stained section is used.

In the foregoing, the preferred embodiments of the present disclosure have been described in detail with reference to the drawings, but the technical scope of the present disclosure is not limited to these examples. It is obvious to those skilled in the art of the present disclosure that various modifications or variations can be conceived within the scope of the technical idea described in the claims, and naturally understood that such modifications or variations also fall within the technical scope of the present disclosure.

Further, the effects described in the present specification are merely illustrative or exemplary, and are not restrictive. That is, other effects that are obvious to those skilled in the art from the description of the present specification may be achieved according to the technology of the present disclosure in addition to or instead of the above-described effects.

Note that the following configuration also falls within the technical scope of the present disclosure.

(1) An information processing apparatus comprising:

a fluorescence signal acquisition unit that acquires a plurality of fluorescence spectra corresponding to each of a plurality of excitation lights having different wavelengths and irradiated to a fluorescently-stained sample generated by staining the sample with a fluorescent reagent;

a link unit that generates a linked fluorescence spectrum by linking at least a part of the plurality of fluorescence spectra to each other in a wavelength direction;

a separation unit that separates a linked autofluorescence spectrum, in which spectra of the autofluorescent substances in the sample are linked to each other in a wavelength direction, into a spectrum for each fluorescent substance using a reference spectrum including the linked autofluorescence reference spectrum, in which spectra of the fluorescent substances in the fluorescently-stained sample are linked to each other in the wavelength direction, and the linked fluorescence reference spectrum; and

an extraction unit that updates the linked autofluorescence reference spectrum using the spectrum for each fluorescent substance separated by the separation unit.

(2) The information processing apparatus according to the above (1), wherein,

the extraction unit extracts the linked autofluorescence reference spectrum from linked autofluorescence spectra generated by linking at least a part of a plurality of autofluorescence spectra with each other in a wavelength direction, wherein the plurality of autofluorescence spectra are obtained by irradiating a slice with the plurality of excitation lights, and the slice is the same as or similar to the sample.

(3) The information processing apparatus according to the above (2), wherein,

the extraction unit extracts the linked autofluorescence reference spectrum by performing non-negative matrix factorization using the linked autofluorescence spectrum generated by linking at least a part of a plurality of autofluorescence spectra with each other in a wavelength direction, wherein the plurality of autofluorescence spectra are obtained by irradiating the slice with the plurality of excitation lights, and the slice is the same as or similar to the sample.

(4) The information processing apparatus according to the above (3), wherein,

the extraction unit extracts the linked autofluorescence reference spectra by setting initial values in the non-negative matrix factorization using an autofluorescence spectrum acquired in advance.

(5) The information processing apparatus according to any one of the above (1) to (4),

the separation unit separates the linked fluorescence spectrum into a spectrum for each fluorescent substance using any one of a least square method or a weighted least square method using the reference spectrum.

(6) The information processing apparatus according to the above (5), wherein,

the separation unit separates the concatenated fluorescence spectra into spectra for each fluorescent substance by setting a matrix representing the concatenated fluorescence spectra as Signal, setting a matrix representing the reference spectra as St, setting a matrix representing a color mixing ratio of each reference spectrum in the concatenated fluorescence spectra as a, and calculating a matrix representing the color mixing ratio when a sum of squares of values represented by the following equation (34) becomes minimum:

[ equation 34]

Signal-a*St (34)。

(7) The information processing apparatus according to the above (6), wherein,

in the case of using the weighted least square method, the separation unit sets an upper limit value, for which weighting is not performed, as Offset value, and replaces the matrix St representing the reference spectrum in equation (34) with a matrix St _ represented by the following equation (35):

[ equation 35]

(8) The information processing apparatus according to any one of the above (1) to (7),

the separation unit separates the linked fluorescence spectrum into a spectrum of each fluorescent substance using a reference spectrum including a linked autofluorescence reference spectrum and a linked fluorescence reference spectrum calculated based on the number of fluorescent molecules or the number of antibodies bound to the fluorescent molecules.

(9) The information processing apparatus according to the above (8), wherein,

the separation unit separates the linked fluorescence spectrum into a spectrum of each fluorescent substance using a reference spectrum including the linked autofluorescence reference spectrum and the linked fluorescence reference spectrum of each fluorescent molecule or each antibody.

(10) The information processing apparatus according to any one of the above (1) to (9),

the separation unit separates the linked fluorescence spectrum into a spectrum of each fluorescent substance by performing non-negative matrix factorization on the linked fluorescence spectrum.

(11) The information processing apparatus according to the above (10), wherein,

the separation unit calculates a product-moment correlation coefficient with an initial value employed for the non-negative matrix factorization with respect to the spectrum extracted by the non-negative matrix factorization, specifying a correspondence between the fluorescent substance and the extracted spectrum.

(12) The information processing apparatus according to any one of the above (1) to (11), wherein,

the linking unit corrects the plurality of fluorescence spectra, and links at least a part of the corrected plurality of fluorescence spectra to each other in a wavelength direction.

(13) The information processing apparatus according to the above (12), wherein,

the linking unit corrects intensities of the plurality of fluorescence spectra.

(14) The information processing apparatus according to the above (13), wherein,

the linking unit corrects the intensities of the plurality of fluorescence spectra by dividing the plurality of fluorescence spectra by the excitation power density.

(15) The information processing apparatus according to any one of the above (12) to (14), wherein,

the link unit corrects a wavelength resolution of at least one of the plurality of fluorescence spectra to a wavelength resolution different from that of the other fluorescence spectra.

(16) The information processing apparatus according to any one of the above (1) to (15), wherein,

the link unit extracts a fluorescence spectrum in a wavelength band including an intensity peak from each of the plurality of fluorescence spectra, and generates the linked fluorescence spectrum by linking the extracted fluorescence spectra to each other.

(17) The information processing apparatus according to any one of the above (1) to (16),

the linked fluorescence spectra are discontinuously linked in a wavelength direction in the plurality of fluorescence spectra.

(18) The information processing apparatus according to any one of the above (1) to (17),

the fluorescence signal acquisition unit acquires first image data that is obtained by imaging the fluorescence-stained sample and includes the plurality of fluorescence spectra, and

the separation unit separates the first image data into a spectrum for each fluorescent substance by performing non-negative matrix factorization on a first Gram matrix of the first image data.

(19) The information processing apparatus according to the above (18), wherein,

the separation unit calculates the first Gram matrix by convolving a second Gram matrix for each of a plurality of second image data obtained by dividing the first image data.

(20) The information processing apparatus according to any one of the above (1) to (17),

the fluorescence signal acquisition unit acquires first image data by imaging a sample that is not stained and is irradiated with the excitation light, and

the extraction unit extracts a spectrum of each autofluorescent substance from the first image data by performing non-negative matrix factorization on the first Gram matrix of the first image data, and updates the linked autofluorescent reference spectrum using the extracted spectrum of each autofluorescent substance.

(21) The information processing apparatus according to the above (20), wherein,

the extraction unit calculates a first Gram matrix by convolving a second Gram matrix for each of a plurality of second image data obtained by dividing the first image data.

(22) A microscope system, comprising: a light source that irradiates a fluorescently stained sample with a plurality of excitation lights having different wavelengths, the fluorescently stained sample being produced by staining the sample with a fluorescent reagent; an imaging device that acquires a plurality of fluorescence spectra corresponding to each of the plurality of excitation lights; and software for processing using the plurality of fluorescence spectra, wherein,

the software is executed on an information processing apparatus, and

the realization is as follows:

generating a linked fluorescence spectrum by linking at least a portion of the plurality of fluorescence spectra to each other in a wavelength direction;

separating the linked fluorescence spectrum into spectra of each fluorescent substance using a reference spectrum including a linked autofluorescence reference spectrum in which spectra of the autofluorescent substances in the sample are linked to each other in a wavelength direction and a linked fluorescence reference spectrum in which spectra of the fluorescent substances in the fluorescently-stained sample are linked to each other in a wavelength direction; and is

Updating the linked autofluorescence reference spectrum using the separated spectrum of each fluorescent substance.

List of reference numerals

10 fluorescent reagent

11 reagent identification information

20 samples

21 sample identification information

30 fluorescent staining sample

100 information processing apparatus

110 acquisition unit

111 information acquisition unit

112 fluorescent signal acquisition unit

120 memory cell

121 information storage unit

122 fluorescent signal storage unit

130 processing unit

131 link unit

132 separation processing unit

133 image generation unit

140 display unit

150 control unit

160 operating unit

200 database.

Claims

1. An information processing apparatus comprising:

2. The information processing apparatus according to claim 1,

3. The information processing apparatus according to claim 2,

4. The information processing apparatus according to claim 3,

5. The information processing apparatus according to claim 1,

6. The information processing apparatus according to claim 5,

the separation unit separates the concatenated fluorescence spectra into spectra for each fluorescent substance by setting a matrix representing the concatenated fluorescence spectra as Signal, setting a matrix representing the reference spectra as St, setting a matrix representing a color mixing ratio of each reference spectrum in the concatenated fluorescence spectra as a, and calculating a matrix representing the color mixing ratio when a sum of squares of values represented by the following equation (1) becomes minimum:

[ equation 1]

Signal-a*St (1)。

7. The information processing apparatus according to claim 6,

in the case of using the weighted least square method, the separation unit sets an upper limit value, for which weighting is not performed, as Offset value, and replaces the matrix St representing the reference spectrum in equation (1) with a matrix St _ represented by the following equation (2):

[ equation 2]

8. The information processing apparatus according to claim 1,

the separation unit separates the linked fluorescence spectrum into a spectrum of each fluorescent substance using a reference spectrum including the linked autofluorescence reference spectrum and the linked fluorescence reference spectrum calculated based on the number of fluorescent molecules or the number of antibodies bound to the fluorescent molecules, or using a reference spectrum including the linked autofluorescence reference spectrum and the linked fluorescence reference spectrum of each of the fluorescent molecules or each of the antibodies.

9. The information processing apparatus according to claim 1,

10. The information processing apparatus according to claim 9,

11. The information processing apparatus according to claim 1,

12. The information processing apparatus according to claim 11,

the linking unit corrects intensities of the plurality of fluorescence spectra.

13. The information processing apparatus according to claim 12,

the linking unit corrects the intensities of the plurality of fluorescence spectra by dividing the plurality of fluorescence spectra by excitation power density.

14. The information processing apparatus according to claim 11,

15. The information processing apparatus according to claim 1,

16. The information processing apparatus according to claim 1,

17. The information processing apparatus according to claim 1,

18. The information processing apparatus according to claim 17,

19. The information processing apparatus according to claim 1,

20. A microscope system, comprising: a light source that irradiates a fluorescently stained sample with a plurality of excitation lights having different wavelengths, the fluorescently stained sample being produced by staining the sample with a fluorescent reagent; an imaging device that acquires a plurality of fluorescence spectra corresponding to each of the plurality of excitation lights; and software for processing using the plurality of fluorescence spectra, wherein,

the software is executed on an information processing apparatus, and

the realization is as follows: