JP7470339B2 - Dye image estimator learning device, image processing device, dye image estimator learning method, image processing method, dye image estimator learning program, and image processing program - Google Patents

Description

The present invention relates to a dye image estimator learning device, an image processing device, a dye image estimator learning method, an image processing method, a dye image estimator learning program, and an image processing program.

Cell and tissue specimens used in pathological diagnosis are often colorless and transparent, which can make them difficult to observe. Therefore, to make them easier to observe under a microscope, the cells and tissues are often stained beforehand before observation.

JP 2013-113689 A JP 2011-185843 A JP 2008-51772 A JP 2013-114233 A JP 2012-233784 A JP 2016-161417 A International Publication No. 2012/163211

Suzuki K, Armato SG, 3rd, Li F, Sone S, Doi K: Massive training artificial neural network (MTANN) for reduction of false positives in computerized detection of lung nodules in low-dose computed tomography, Med. Phys., 30(7), 1602-1617, 2003. Suzuki K, Horiba I, Sugie N, Nanki M: Extraction of left ventricular contours from left ventriculograms by means of a neural edge detector, IEEE Trans. Med. Imaging, 23(3), 330-339, 2004. Suzuki K, Horiba I, Sugie N: Neural edge enhancer for supervised edge enhancement from noisy images, IEEE Trans. Pattern Anal. Mach. Intell., 25(12), 1582-1596, 2003. Xu JW, Suzuki K: Massive-training support vector regression and Gaussian process for false-positive reduction in computer-aided detection of polyps in CT colonography, Med. Phys., 38(4), 1888-1902, 2011.

Although staining makes it easier to observe the state of tissues and cells, the problem is that staining takes time. In addition, staining requires processing such as decolorization. Furthermore, to achieve high-quality staining, specific staining equipment and staining techniques are required. Therefore, there is a need to make it easier to observe tissues and cells without staining.

The present invention aims to provide a technique that allows for the observation of tissues and cells without staining them.

To solve the above problems, the following measures will be adopted.

That is, the first aspect is
an image acquisition unit that acquires an autofluorescence image by capturing autofluorescence of a biological material sample due to a predetermined excitation light, and acquires a stained image by capturing the biological material sample stained with a predetermined staining solution;
a stained image estimator learning unit that learns a stained image estimator that estimates an image of the stained biological material sample when the biological material sample in the autofluorescence image is stained with the staining solution, based on the autofluorescence image and the stained image acquired by the image acquisition unit;
The image processing device includes:

The disclosed aspects may be realized by a program being executed by an information processing device. That is, the disclosed configuration may be specified as a program for causing an information processing device to execute the processes executed by each of the means in the above-mentioned aspects, or a computer-readable recording medium on which the program is recorded. The disclosed configuration may also be specified by a method in which an information processing device executes the processes executed by each of the above-mentioned means. The disclosed configuration may also be specified as a system including an information processing device that executes the processes executed by each of the above-mentioned means.

The present invention provides a technology that allows tissues and cells to be observed without staining them.

FIG. 1 is a diagram illustrating an example of the configuration of a system according to the first embodiment. FIG. 2 is a diagram illustrating an example of a functional configuration of the image processing apparatus. FIG. 3 is a diagram illustrating an example of a hardware configuration of an information processing device. FIG. 4 is a diagram showing an example of an operational flow when generating a dye amount estimator in an image processing apparatus. FIG. 5 is a diagram showing an example (1) of an autofluorescence image of a biological material (liver tissue) photographed by a fluorescence microscope. FIG. 6 is a diagram showing an example (2) of an autofluorescence image of a biological material (liver tissue) photographed by a fluorescence microscope. FIG. 7 is a diagram showing an example (3) of an autofluorescence image of a biological material (liver tissue) photographed by a fluorescence microscope. FIG. 8 is a diagram showing an example of a graph of the wavelength dependency of the spectral absorption coefficient of a dye solution or the like. FIG. 9 is a diagram showing an example of an operational flow when generating a simulated stained image in the image processing device. FIG. 10 is a diagram illustrating an example of the configuration of a system according to the second embodiment. FIG. 11 is a diagram showing an example of an operation flow when the stained image estimator learning device learns the stained image estimator. FIG. 12 is a diagram for explaining the learning of the stained image estimator in the stained image estimator learning unit. FIG. 13 is a diagram showing an example of the configuration of a simulated stained image generating system according to the second embodiment. FIG. 14 is a diagram showing an example of an operational flow when generating a simulated stained image in the image processing device. FIG. 15 is a diagram for explaining the generation of a simulated stained image in the simulated stained image generating section. FIG. 16 is a diagram showing the structure of an MTANN as an example of an image-based deep learning machine. FIG. 17 is a diagram showing the learning of the MTANN.

The following describes the embodiments with reference to the drawings. The configurations of the embodiments are merely examples, and the configuration of the invention is not limited to the specific configurations of the disclosed embodiments. When implementing the invention, specific configurations according to the embodiments may be appropriately adopted.

[Embodiment 1]
(Configuration Example 1)
1 is a diagram showing an example of the configuration of a system according to the present embodiment. The system 10 according to the present embodiment includes an image processing device 100, a fluorescence microscope 200, and a multispectral camera 300.

The image processing device 100 acquires an autofluorescence image of the sample from the fluorescence microscope 200, and acquires a stained image of the sample for each band from the multispectral camera 300. The image processing device 100 detects the amount of dye from the stained image. The image processing device 100 generates a dye amount estimator by associating the detected amount of dye with the autofluorescence image. The image processing device 100 uses the generated dye amount estimator to output a stained image based on another autofluorescence image. The sample is, for example, a biological material such as a cell or tissue. The image processing device 100, the fluorescence microscope 200, and the multispectral camera 300 are connected to each other so as to be able to communicate with each other directly or via a network or the like.

The fluorescence microscope 200 is a microscope that observes a sample by irradiating the sample with excitation light and observing the autofluorescence from the sample. The fluorescence microscope 200 can irradiate the sample with multiple types of excitation light (excitation light with different wavelengths) and obtain each autofluorescence image.

The multispectral camera 300 is a camera that can capture images of an object in multiple bands (frequency bands, wavelength bands). The multispectral camera 300 captures images of a stained sample in each band.

Figure 2 is a diagram showing an example of the functional configuration of an image processing device. The image processing device 100 includes an image acquisition unit 102, a color information acquisition unit 104, a pigment amount detection unit 106, a pigment amount estimation unit 108, and an image generation unit 110.

The image acquisition unit 102 acquires an autofluorescence image of the sample from the fluorescence microscope 200 and acquires a stained image of the sample from the multispectral camera 300.

The color information acquisition unit 104 acquires color information from the stained image of the stained sample captured by the multispectral camera 300.

The dye amount detection unit 106 estimates the amount of dye from the color information of the acquired multiband stained image.

The pigment amount estimation unit 108 generates a pigment amount estimator based on the pigment amounts in the autofluorescence image and the stained image.

The image generating unit 110 uses a dye amount estimator to generate a digital stained image (simulated stained image) based on the autofluorescence image of the sample captured by the fluorescence microscope 200.

Fig. 3 is a diagram showing an example of the hardware configuration of an information processing device. The information processing device 90 shown in Fig. 3 has the configuration of a general computer. The image processing device 100 is realized by using the information processing device 90 as shown in Fig. 3. The information processing device 90 in Fig. 3 has a processor 91, a memory 92, a storage unit 93, an input unit 94, an output unit 95, and a communication control unit 96. These are connected to each other by a bus. The memory 92 and the storage unit 93 are computer-readable recording media. The hardware configuration of the information processing device is not limited to the example shown in Fig. 3, and components may be omitted, replaced, or added as appropriate.

The information processing device 90 has a processor 91 that loads a program stored on a recording medium into a working area of a memory 92 and executes the program, and each component is controlled through the execution of the program, thereby realizing functions that meet a specified purpose.

The processor 91 is, for example, a CPU (Central Processing Unit) or a DSP (Digital Signal Processor).

Memory 92 includes, for example, RAM (Random Access Memory) and ROM (Read Only Memory). Memory 92 is also called a main storage device.

The storage unit 93 is, for example, an erasable programmable ROM (EPROM) or a hard disk drive (HDD). The storage unit 93 may also include a removable medium, i.e., a portable recording medium. The removable medium is, for example, a universal serial bus (USB) memory or a disk recording medium such as a compact disc (CD) or a digital versatile disc (DVD). The storage unit 93 is also called a secondary storage device.

The storage unit 93 stores various programs, various data, and various tables used by the information processing device 90 in a readable and writable recording medium. The storage unit 93 stores an operating system (OS), various programs, various tables, etc. The information stored in the storage unit 93 may be stored in the memory 92. Also, the information stored in the memory 92 may be stored in the storage unit 93.

The operating system is software that mediates between software and hardware, manages memory space, manages files, and manages processes and tasks. The operating system includes a communication interface. The communication interface is a program that exchanges data with other external devices connected via the communication control unit 96. External devices include, for example, other information processing devices, external storage devices, etc.

The input unit 94 includes a keyboard, a pointing device, a wireless remote control, a touch panel, etc. The input unit 94 can also include an input device for video or images, such as a camera, and an input device for audio, such as a microphone.

The output unit 95 includes a display device such as an LCD (Liquid Crystal Display), an EL (Electroluminescence) panel, a CRT (Cathode Ray Tube) display, a PDP (Plasma Display Panel), etc., a printer, etc. The output unit 95 may also include an audio output device such as a speaker.

The communication control unit 96 connects to other devices and controls communication between the information processing device 90 and the other devices. The communication control unit 96 is, for example, a LAN (Local Area Network) interface board, a wireless communication circuit for wireless communication, and a communication circuit for wired communication. The LAN interface board and the wireless communication circuit are connected to a network such as the Internet.

In the information processing device 90, the processor deploys a program stored in the auxiliary memory unit in an executable manner in the working area of the main memory unit, and controls peripheral devices, etc. through the execution of the program. This allows the information processing device to realize functions that meet a specified purpose. The main memory unit and the auxiliary memory unit are recording media that can be read by the information processing device.

(Operation example 1)
<Pigment amount estimator generation>
FIG. 4 is a diagram showing an example of an operational flow when generating a dye amount estimator in an image processing apparatus.

In S101, the image acquisition unit 102 of the image processing device 100 acquires an autofluorescence image of a sample captured by the fluorescence microscope 200 from the fluorescence microscope 200. The autofluorescence image is obtained by irradiating the sample with excitation light of a predetermined frequency and capturing the autofluorescence of the sample. In the fluorescence microscope 200, the sample is irradiated with multiple types (multiple wavelengths) of excitation light, and the autofluorescence of the sample for each excitation light is captured. The sample is, for example, a slice specimen of a biological material such as cells or tissue. The samples used in S101 and S102 are slice specimens produced from the same biological material. The acquired autofluorescence image may be obtained using one type of excitation light.

Figures 5, 6, and 7 are examples of autofluorescence images of biological material (liver tissue) taken with a fluorescence microscope. In Figures 5, 6, and 7, whiter colors indicate brighter images. In the autofluorescence image of Figure 5, the excitation light has a wavelength of 470 nm, and a filter that passes light with a wavelength of 495 nm or more is used. In the autofluorescence image of Figure 6, the excitation light has a wavelength of 545 nm, and a filter that passes light with a wavelength of 565 nm or more is used. In the autofluorescence image of Figure 7, the excitation light has a wavelength of 360 nm, and a filter that passes light with a wavelength of 400 nm or more is used. The autofluorescence image does not contain excitation light.

In S102, the image acquisition unit 102 acquires a stained image of the stained sample captured by the multispectral camera 300 from the multispectral camera 300. The stained image is captured by staining the sample with a predetermined staining solution. By adding a predetermined staining solution to the sample, a predetermined portion develops color. The portion and color that develops color depend on the staining solution. The multispectral camera 300 captures stained images in multiple bands (frequency bands). The staining solution is, for example, hematoxylin and eosin. Multiple staining solutions may be used as the staining solution. By using the multispectral camera 300, more detailed information on the transmitted light that has passed through the sample can be obtained.

In S103, the color information acquisition unit 104 obtains the absorbance of each pixel (position) in the image from the stained image of each band. Absorbance is a quantity that indicates the degree to which the intensity of light is weakened when light passes through a medium. The absorbance of each pixel depends on the brightness of each pixel. For each band, the color information acquisition unit 104 obtains the absorbance of each pixel from the brightness of each pixel in the stained image. The absorbance of each band depends on the amount of pigment in the staining solution incorporated into the sample and the extinction coefficient of the staining solution.

これらの式より、推定行列Ｘ^＋を吸光度ベクトルａに乗算することによって、試料に取り込まれた染色液毎のピクセル毎（位置毎）の色素量Ｃを求めることができる。色素量検出部１０６は、ピクセル毎の色素量に基づいて、試料の各位置の色素量を示す色素量画像を生成する。色素量画像は、試料に取り込まれた染色液の分布を示す。色素量画像では、例えば、染色液が多く取り込まれた位置はより明るくなる。 In S104, the dye amount detection unit 106 calculates the amount of dye of the staining solution incorporated into the sample for each pixel based on the absorbance determined in S103 and the known spectral absorption coefficient of the staining solution. The spectral absorption coefficient is an amount that indicates the degree to which a medium absorbs light when the light is incident on the medium. The absorption coefficient of the staining solution depends on the wavelength of the incident light. If the absorbance vector is a, the spectral absorption coefficient matrix is X, and the amount of dye is C, then it can be expressed as follows according to Lambert-Beer's law. The absorbance vector is a vector representation of the absorbance for each band. The spectral absorption coefficient matrix is a matrix representation of the absorption coefficient for each staining solution and each band.

From these equations, the dye amount C for each pixel (each position) for each staining solution taken into the sample can be obtained by multiplying the absorbance vector a by the estimated matrix X ⁺ . The dye amount detection unit 106 generates a dye amount image showing the dye amount at each position on the sample based on the dye amount for each pixel. The dye amount image shows the distribution of the staining solution taken into the sample. In the dye amount image, for example, positions where a large amount of staining solution has been taken in are brighter.

Figure 8 is a diagram showing an example of a graph of the wavelength dependence of the spectral absorption coefficient of a staining solution, etc. The example in Figure 8 shows the spectral absorption coefficients of hematoxylin, eosin, and red blood cells. In the example in Figure 8, for example, the spectral absorption coefficient of eosin peaks at about 550 nm. The spectral absorption coefficient matrix is generated based on this graph.

In S105, the dye amount detection unit 106 analyzes the relationship between the autofluorescence image acquired in S101 and the dye amount image generated in S104 by machine learning or the like. For the analysis, a method using a learning space such as deep learning by a neural network, regression SVM, regression random forest, multiple regression analysis, look up table, or the like can be used. For the analysis, a method for predicting continuous values can be applied. For example, if the dye amount detection unit 106 is a neural network, the input data is an autofluorescence image of a sample excited under a plurality of conditions (excitation light), and the output data is a dye amount image obtained from a multispectral image obtained by staining the sample. The dye amount detection unit 106 optimizes the weighting coefficient between neurons of the neural network by supervised learning in which the input data and the output data are linked to each other, thereby constructing a dye amount estimator. The dye amount estimator is optimized by using a larger number of pairs of autofluorescence images and dye amount images. In addition, the dye amount estimator can be generated for each staining solution. The input data may also include image processing features such as texture information (information indicating the characteristics of tissues or cells, etc.).

In addition, by generating dye amount estimators using stained images of samples stained with various staining solutions, it is possible to generate dye amount estimators for various staining solutions.

<Simulated dye image generation>
9 is a diagram showing an example of an operation flow when generating a simulated stained image in an image processing device. The simulated stained image (digital stained image) is a simulated stained image obtained based on an estimated dye amount image estimated by a dye amount estimator using an autofluorescence image of a sample. The simulated stained image is a simulation of a stained image obtained by actually staining a sample. Here, the image processing device 100 generates the simulated stained image using an autofluorescence image of an unstained sample and the above-mentioned dye amount estimator.

In S201, the image acquisition unit 102 of the image processing device 100 acquires an autofluorescence image of a sample captured by the fluorescence microscope 200 from the fluorescence microscope 200. In the fluorescence microscope 200, the sample is irradiated with multiple types (multiple wavelengths) of excitation light, and the autofluorescence of the sample in response to each excitation light is captured. The sample is, for example, a slice specimen of a biological material such as a cell or tissue. The sample corresponding to the autofluorescence image used here is a different sample from the sample corresponding to the autofluorescence image used when generating the pigment amount estimator.

In S202, the dye amount estimation unit 108 uses the autofluorescence image acquired by the image acquisition unit 102 as input data and generates an estimated dye amount image of the desired dye solution using a dye amount estimator corresponding to the desired staining solution. In the dye amount image, for example, the greater the dye amount, the brighter the color displayed. The dye amount image is an image that indicates the estimated dye amount of the staining solution incorporated in the specimen.

In S203, the image generating unit 110 calculates the absorbance for each pixel for each band from the dye amount image (or dye amount) and the spectral absorption coefficient for each staining solution. The image generating unit 110 calculates the light transmittance for each band from the calculated absorbance for each band, and calculates the pixel value (RGB value) of each pixel by integrating for each RGB (Red Green Blue) component for each pixel. The image generating unit 110 generates a pseudo-stained image (digital stained image) based on the calculated pixel value of each pixel. This allows the image processing device 100 to generate a pseudo-stained image for a desired staining solution without staining the sample.

(Actions and Effects of the First Embodiment)
The image processing device 100 acquires an autofluorescence image of a biological material sample such as a cell or tissue captured by a fluorescence microscope 200. The image processing device 100 acquires a stained image stained with a staining solution captured by a multispectral camera 300. The image processing device 100 calculates the amount of dye in the staining solution incorporated into the sample based on the stained image and the spectral absorption coefficient of the staining solution. The image processing device 100 generates a dye amount estimator based on the autofluorescence image and the calculated dye amount (dye amount image).

The image processing device 100 generates an estimated dye amount image using an autofluorescence image of the sample and a dye amount estimator corresponding to the desired staining solution. The image processing device 100 calculates the absorbance for each band from the dye amount image and the spectral absorption coefficient for each staining solution. The image processing device 100 calculates the transmittance for each band from the absorbance for each band, and integrates the transmittance for each band for each RGB component to calculate the pixel value of each pixel and generate a pseudo-stained image.

The image processing device 100 uses autofluorescence, which has traditionally been considered noise during staining, to estimate the amount of dye when stained and generate a digital stained image (simulated stained image), thereby making it possible to obtain a stained image without staining the specimen (sample). With the image processing device 100, it is possible to obtain a simulated stained image without invasively inspecting even valuable samples. Unstained samples can be easily observed without the need for a special microscope for observing unstained samples, etc. Furthermore, with the image processing device 100, since the sample is not stained, it is possible to generate pseudo-stained images using various staining solutions from a single sample.

[Embodiment 2]
(Configuration Example 2)
10 is a diagram showing an example of the configuration of a system according to this embodiment. A system 40 according to this embodiment includes a stained image estimator learning device 400, an autofluorescence image capturing device 500, and a stained image capturing device 600.

The stained image estimator learning device 400 acquires an autofluorescence image of a sample from the autofluorescence image capturing device 500, and acquires a stained image of the sample stained from the stained image capturing device 600. The stained image estimator learning device 400 associates the autofluorescence image with the stained image to learn the stained image estimator. The sample is, for example, a biological material sample such as a cell or tissue. The stained image estimator learning device 400, the autofluorescence image capturing device 500, and the stained image capturing device 600 are connected to each other so as to be able to communicate with each other directly or via a network or the like.

The stained image estimator learning device 400 includes an image acquisition unit 410, a registration unit 420, and a stained image estimator learning unit 430. The image processing device 400 also includes a stained image estimator (red) 45.
1, a stained image estimator (green) 452, and a stained image estimator (blue) 453. The stained image estimator (red) 451, the stained image estimator (green) 452, and the stained image estimator (blue) 453 are collectively referred to as the stained image estimators of each color.

The image acquisition unit 410 acquires an autofluorescence image of the sample from the autofluorescence image capture device 500, and acquires a stained image of the sample from the stained image capture device 600. The sample used when capturing the autofluorescence image is the sample used when stained with a reagent and capturing the stained image. The sample used when capturing the autofluorescence image and the sample used when stained with a reagent and capturing the stained image may be two samples collected in close proximity with almost no difference between them. These two samples can be considered to be the same sample.

The alignment unit 420 aligns the position of the sample in the autofluorescence image with the position of the sample in the stained image. By aligning the autofluorescence image with the stained image, the accuracy of the stained image estimator is improved.

The stained image estimator learning unit 430 learns the stained image estimator for each color based on a pair of an autofluorescence image and a stained image.

The stain image estimator learning device 400 is realized by using an information processing device 90 as shown in FIG. 3. The stain image estimator (red) 451, the stain image estimator (green) 452, and the stain image estimator (blue) 453 can be stored in a storage means such as the memory 92 or storage unit 93 of the information processing device 90.

The autofluorescence image capturing device 500 includes a fluorescence microscope 510 and an image capturing unit 520. The fluorescence microscope 510 is a microscope that observes a sample by irradiating the sample with excitation light and observing the autofluorescence from the sample. The fluorescence microscope 510 can irradiate the sample with multiple types of excitation light (excitation light with different wavelengths). The image capturing unit 520 is installed in the fluorescence microscope 510 and captures the sample magnified by the fluorescence microscope 510. The image capturing unit 520 can acquire an autofluorescence image for each excitation light of the sample. The image capturing unit 520 is, for example, a grayscale camera, a multispectral camera (Hyper Spectral Imaging (HSI) camera), an RGB (Red Green Blue) camera, etc. The multispectral camera is a camera that can capture the subject (object) for each of multiple bands (frequency bands, wavelength bands). The RGB camera is a camera that can capture the subject for each of the colors red, green, and blue. With an RGB camera, images of red, green, and blue colors can be acquired. A full-color image can be obtained by combining images of red, green, and blue brightness. The image capturing unit 520 can acquire images capturing the autofluorescence of the sample. The image capturing unit 520 may also be provided with a filter that transmits light in a specific frequency band, and the image capturing unit 520 may capture light in the specific frequency band (such as the autofluorescence of the sample).

The stained image capturing device 600 includes an optical microscope 610 and an imaging unit 620. The optical microscope 610 is a microscope that observes a sample stained with a reagent using visible light. The imaging unit 620 is installed in the optical microscope 610 and captures an image of the sample magnified by the optical microscope 610. The imaging unit 620 is an RGB camera. The imaging unit 620 may be another camera such as a multispectral camera.

The autofluorescence image capturing device 500 and the stained image capturing device 600 may be integrated into a single capturing device having a microscope and an image capturing unit. The microscope has the function of a fluorescence microscope 510 and the function of an optical microscope 610. The image capturing unit has the function of an image capturing unit 520 and the function of an image capturing unit 620.

Learning the stain image estimator
FIG. 11 is a diagram showing an example of an operation flow when the stained image estimator learning device learns the stained image estimator.

In S301, the image acquisition unit 410 of the image processing device 400 acquires an autofluorescence image of a sample captured by the autofluorescence image capture device 500 from the autofluorescence image capture device 500. The autofluorescence image is obtained by irradiating the sample with excitation light of a predetermined frequency and capturing the autofluorescence of the sample. The excitation light may include not only visible light but also infrared light and ultraviolet light. In the autofluorescence image capture device 500, the fluorescence microscope 510 irradiates the sample with multiple types (multiple wavelengths) of excitation light, and the image capture unit 520 captures the autofluorescence of the sample for each excitation light. That is, the image capture unit 520 of the autofluorescence image capture device 500 captures the autofluorescence of the sample for each excitation light. The sample is, for example, a slice specimen of a biological material such as cells or tissue. The sample used in S301 and S302 is a slice specimen produced from the same biological material. The acquired autofluorescence image may be obtained by one type of excitation light.

In S302, the image acquisition unit 410 acquires a stained image of the stained sample captured by the stained image capture device 600 from the stained image capture device 600. The stained image is captured by staining the sample with a predetermined staining liquid. By adding a predetermined staining liquid to the sample, a predetermined portion develops color. The portion and color of the colored portion depend on the staining liquid. In the stained image capture device 600, the stained image is captured by the RGB camera of the imaging unit 620. The staining liquid is, for example, hematoxylin and eosin. A plurality of staining liquids may be used as the staining liquid. The image acquisition unit 410 may acquire the stained image as a stained image for each color of red, green, and blue. The image acquisition unit 410 may also acquire the stained image for each staining liquid used.

In S303, the alignment unit 420 aligns the position of the sample in the autofluorescence image with the position of the sample in the stained image. Even if the images are of the same sample, the coordinates of a specific position of the sample in one image may differ from those in the other image due to the influence of the positions of the imaging units, the staining process, the performance of the microscope, and the like. The alignment unit 420 may align the two images by having the user specify multiple identical positions of the sample in the two images, or may automatically perform the alignment using a well-known image recognition technique. For example, the alignment unit 420 calculates the sum of the difference values of the pixel values of each pixel of the two images while shifting the position of the other image relative to one image, and when the sum is minimum, it is determined that the positions of the two images are aligned. The alignment unit 420 may enlarge, reduce, or rotate one image during alignment. After alignment, the alignment unit 420 assigns the same coordinates to the same positions of the sample in the two images. The alignment unit 420 may trim the portions that are common to the two images from each image to generate a new autofluorescence image and a new stained image, respectively. The process of S303 may be omitted.

In S304, the stained image estimator learning unit 430 analyzes the relationship between the autofluorescence image acquired in S301 and the stained image acquired in S302 (the relationship between the autofluorescence image and the stained image aligned in S303) by machine learning or the like. For the analysis, a method of directly learning an image, such as image output type deep learning, or the like, may be used. An example of image output type deep learning is MTANN (massive-training artificial neural network). MTANN is a highly nonlinear deep learning model that can output an image. Here, the stained image estimator learning unit 430 learns an estimator by MTANN as a stained image estimator for each color. The stained image estimator is one of the image output type deep learning devices. In addition, other deep learning may be used for the analysis. Other examples of deep learning include convolutional neural networks (CNN), shift-invariant neural networks, deep belief networks (DBN), deep neural networks (DNN), fully convolutional neural networks (FCN), U-Net, V-Net, multi-resolution massive-training artificial neural networks, multiple expert massive
-training artificial neural networks, SegNet, VGG-16, LeNet, AlexNet, ResNet, Auto encoders and decoders, Generative adversarial networks (GAN), Recurrent Neural Networks (RNN), Recursive Neural Networks, Long Short-Term Memory (L
STM) and the like. The deep learning used here is not limited to these. A method of predicting and outputting an image can be applied to the analysis. Here, the stained image estimator learning unit 430 uses a plurality of autofluorescence images of a sample excited under a plurality of conditions (excitation light) as input data, and uses red, green, and blue stained images (red, green, and blue components of the stained image) of the stained sample obtained by staining the sample as teacher data. The stained image estimator learning unit 430 optimizes the weighting coefficients between the neurons of the neural network by supervised learning in which the input data and the teacher data are linked to construct a stained image estimator. At this time, the stained image estimator learning unit 430 constructs a stained image estimator for each color (red, green, and blue) of the stained image. The stained image estimator for each color is an estimator that estimates a stained image (stained image of each color) that is teacher data from an autofluorescence image that is input data. The output from the estimator is a simulated stained image. The stained image estimator learning unit 430 constructs a stained image estimator for each color by comparing pixel values for each pixel of each image or for each predetermined region (predetermined block). The stained image estimator for each color is optimized by using a larger number of pairs (learning data) of autofluorescence images and stained images. In addition, the stained image estimator for each color can be constructed for each staining solution used. In addition, image processing features such as texture information (information indicating characteristics of tissues and cells, etc.) may be included as input data. The training data may be a full-color image of the stained image.

Figure 12 is a diagram for explaining the learning of the stained image estimator in the stained image estimator learning unit. Images 51, 52, and 53 in Figure 12 show examples of autofluorescence images of a sample captured by the imaging unit 520, which are input data. The sample used here is the same biological sample. The input data is input to the stained image estimator for each color. Images 54, 55, and 56 in Figure 12 show examples of output images of each color (red, green, and blue) output from the stained image estimator for each color, respectively. Images 57, 58, and 59 in Figure 12 show examples of stained images of each color (each color component of the stained image) captured by the RGB camera of the imaging unit 620, which are teaching data. In each image, the whiter the color, the brighter it is (the larger the pixel value).

Image 51 is an autofluorescence image when the sample is irradiated with excitation light of a wavelength of 405 nm. Image 52 is an autofluorescence image when the sample is irradiated with excitation light of a wavelength of 599 nm. Image 53 is an autofluorescence image when the sample is irradiated with excitation light of a wavelength of 436 nm. A specified filter is used when capturing each autofluorescence image, so that the excitation light is not included in the autofluorescence image. Here, autofluorescence images using three types of excitation light are used as input data, but the types of excitation light are not limited to three.

Image 57 is an image of the red component of the stained image obtained by staining the sample with hematoxylin and eosin (HE), image 58 is an image of the green component of the stained image obtained by staining the sample with HE, and image 59 is an image of the blue component of the stained image obtained by staining the sample with HE.

The stained image estimator learning unit 430 links the images (image 51, image 52, image 53) as input data with the images (image 57, image 58, image 59) as teacher data to construct stained image estimator (red) 451, stained image estimator (green) 452, and stained image estimator (blue) 453. For example, the stained image estimator learning unit 430 inputs three autofluorescence images (image 51, image 52, image 53) to the stained image estimator (red) 451. The stained image estimator (red) 451 outputs image 54 as a red stained image (output image). The stained image estimator learning unit 430 compares image 54, which is output data of the stained image estimator (red) 451, with image 57, which is an actual red stained image. The stained image estimator learning unit 430 further learns the stained image estimator (red) 451 based on the comparison result (for example, the difference between pixel values of both images or the mean square error). The stained image estimator learning unit 430 learns the stained image estimator (red) 451 so that the image 54, which is the output image, approaches the image 57, which is the actual stained image. The stained image estimator learning unit 430 repeatedly learns the stained image estimator (red) 451, for example, until the difference (or the mean square error) between each pixel value of the image 54 and each pixel value of the image 57 falls within a predetermined range. The stained image estimator may be trained for each of the regions by dividing each image into a plurality of regions. The stained image estimator may be trained by randomly extracting samples from each image and using the extracted samples. The stained image estimator learning unit 430 trains the stained image estimator (green) 452 and the stained image estimator (blue) 453 in the same manner. In this way, the stained image estimator learning unit 430 constructs a stained image estimator (red) 451, a stained image estimator (green) 452, and a stained image estimator (blue) 453. The set of images used as learning data (a set of an image as input data and an image as training data) is not just one set, but multiple sets of images are used as training data. The accuracy of the stained image estimator is improved by using more sets of training data.

<Simulated dye image generation>
13 is a diagram showing an example of the configuration of a simulated stained image generating system of this embodiment. The simulated stained image generating system 70 includes an image processing device 700 and an autofluorescence image capturing device 500. The autofluorescence image capturing device 500 is similar to the autofluorescence image capturing device 500 in FIG.

The image processing device 700 acquires an autofluorescence image of the sample from the autofluorescence image capturing device 500. The image processing device 700 uses the stained image estimator described above to output a stained image based on the autofluorescence image acquired from the autofluorescence image capturing device 500. The sample is, for example, a biological material sample such as a cell or tissue. The image processing device 700 and the autofluorescence image capturing device 500 are connected to each other so as to be able to communicate with each other directly or via a network or the like.

The image acquisition unit 710 acquires an autofluorescence image of the sample from the autofluorescence image capture device 500.

The simulated stained image generating unit 740 generates a digital stained image (simulated stained image) using a stained image estimator for each color based on the autofluorescence image of the sample captured by the autofluorescence image capturing device 500.

The dyed image estimator (red) 751, the dyed image estimator (green) 752, and the dyed image estimator (blue) 753 are respectively the dyed image estimator (red) 451, the dyed image estimator (green) 452, and the dyed image estimator (blue) 453 constructed by the dyed image estimator learning unit 430 of the dyed image estimator learning device 400.

Figure 14 is a diagram showing an example of an operational flow when generating a simulated stained image in an image processing device. A simulated stained image (digital stained image) is a simulated stained image estimated by a stained image estimator using an autofluorescence image of a sample. A simulated stained image is a simulation of a stained image obtained by actually staining a sample. Here, the image processing device 700 generates a simulated stained image using an autofluorescence image of an unstained sample and the above-mentioned stained image estimator.

In S401, the image acquisition unit 710 of the image processing device 700 acquires from the autofluorescence image capturing device 500 an autofluorescence image of a sample that has been irradiated with excitation light by the fluorescence microscope 510 of the autofluorescence image capturing device 500 and captured by the image capturing unit 520. In the fluorescence microscope 510 of the autofluorescence image capturing device 500, the sample is irradiated with multiple types (multiple wavelengths) of excitation light, and the autofluorescence of the sample for each excitation light is captured. The sample is, for example, a slice specimen of a biological material such as cells or tissue. The sample corresponding to the autofluorescence image used here is a different sample from the sample corresponding to the autofluorescence image used when generating the stained image estimator.

In S402, the simulated stained image generating unit 740 uses the autofluorescence image acquired by the image acquiring unit 710 as input data and generates an estimated stained image using a stained image estimator corresponding to the desired staining liquid. Here, the simulated stained image generating unit 740 generates a red simulated stained image, a green simulated stained image, and a blue simulated stained image by using a stained image estimator for each color (red, green, blue). The simulated stained image generating unit 740 uses the autofluorescence image acquired by the image acquiring unit 710 as input data and generates a red simulated stained image (red component of the simulated stained image) using a red stained image estimator. Similarly, the simulated stained image generating unit 740 generates a green simulated stained image using a green stained image estimator and generates a blue simulated stained image using a blue stained image estimator.

The simulated stained image generating unit 740 generates a full-color simulated stained image by combining the generated red, green, and blue simulated stained images. This generates a simulated stained image of a sample stained with a specified staining solution. The image processing device 700 can generate a simulated stained image for a desired staining solution from an autofluorescence image without staining the sample.

Figure 15 is a diagram for explaining the generation of a simulated stained image in the simulated stained image generation unit. Images 61, 62, and 63 in Figure 15 show examples of autofluorescence images of a sample captured by the image capture unit 520, which are input data. The sample used here is the same biological sample. The input data is input to the stained image estimator for each color. Images 64, 65, and 66 in Figure 15 show examples of output images (simulated stained images for each color) output from the stained image estimator for each color (red, green, and blue), respectively. For example, image 64 is a red simulated stained image (red component of the simulated stained image) output from the stained image estimator (red) 451. In each image, the whiter the color, the brighter it is (the larger the pixel value).

Image 61 is an autofluorescence image when the sample is irradiated with excitation light having a wavelength of 405 nm. Image 62 is an autofluorescence image when the sample is irradiated with excitation light having a wavelength of 599 nm. Image 63 is an autofluorescence image when the sample is irradiated with excitation light having a wavelength of 436 nm. The excitation light used here is the same as the excitation light of the autofluorescence image included in the input data of the learning data.

The simulated stained image generating unit 740 inputs the images (image 61, image 62, image 63) that are input data to the stained image estimator (red) 751, the stained image estimator (green) 752, and the stained image estimator (blue) 753 to obtain a simulated stained image of each color. For example, the simulated stained image generating unit 740 inputs three autofluorescence images (image 61, image 62, image 63) to the stained image estimator (red) 751. The stained image estimator (red) 751 outputs image 64 as a red simulated stained image (output image). In this way, the simulated stained image generating unit 740 obtains a red simulated stained image. In the same manner, the simulated stained image generating unit 740 inputs the three autofluorescence images (image 61, image 62, image 63) to the stained image estimator (green) 752 and the stained image estimator (blue) 753 to obtain a green simulated stained image and a blue simulated stained image. Furthermore, the simulated stained image generating unit 740 synthesizes the red, green, and blue simulated stained images to generate the full-color simulated stained image 67. The red simulated stained image corresponds to the red component of the full-color simulated stained image 67, the green simulated stained image corresponds to the green component of the full-color simulated stained image 67, and the blue simulated stained image corresponds to the blue component of the full-color simulated stained image 67. In this way, the simulated stained image generating unit 740 can obtain the (full-color) simulated stained image 67.

The stain image estimator learning device 400, which trains the stain image estimator, and the image processing device 700, which generates the simulated stain image, may be integrated to operate as a single device.

ただし、ＮＮ（・）は、階層型ＡＮＮ回帰モデルの出力、ｇ（ｘ，ｙ）は入力画像の画素値である。例えば、Ｒ_Ｓが、３ｘ３画素の領域である場合、階層型ＡＮＮ回帰モデルへの入力は、以下のようになる。

従って、この場合の階層型ＡＮＮ回帰モデルの入力層ユニット数は、９個である。ＭＴＡＮＮでは、出力が連続値であるため、階層型ＡＮＮ回帰モデルに、線形出力ＡＮＮモデル（非特許文献２、非特許文献３）を用いる。これは、出力層ユニットの応答関数に、通常使われるシグモイド関数の代わりに、次式の線形関数を用いたＡＮＮのモデルであり、連続値を出力とする応用に大変適している（非特許文献２、非特許文献３）。

従って、入力層、中間層、出力層のユニットの応答関数は、それぞれ、恒等関数、シグモイド関数、線形関数である。階層型ＡＮＮ回帰モデルを、入力画像上で、畳み込み演算のように走査することにより、出力画像を得る。また、畳み込み演算により中間層上に得られる中間画像上で、階層型ＡＮＮ回帰モデルを畳み込み演算のように走査し、これを繰り返すことによって、深い層を形成する。 (About MTANN)
Structure of MTANN (Non-Patent Document 1)
FIG. 16 is a diagram showing the structure of MTANN as an example of an image-based deep learning device. MTANN is a deep learning model that performs image processing using a hierarchical neural network (ANN) regression model (Non-Patent Document 2, Non-Patent Document 3) as the core of calculation. In MTANN, pixel values themselves are input to the hierarchical ANN regression model. In addition, the output of a normal deep learning model is a class to which an object in an image belongs (for example, 1 and 0 are assigned to abnormal shadows and normal shadows, respectively), but the output of the hierarchical ANN regression model of MTANN is the pixel value itself, and the output of MTANN is an image. In MTANN, the input to the hierarchical ANN regression model is the pixel value in a local region (image patch) R _S extracted from the input image as shown in the following formula. Each pixel value is input to each input layer unit (neuron). There is one output layer unit, which outputs a pixel value f(x, y) corresponding to the center position (x, y) of the local region.

where NN(·) is the output of the hierarchical ANN regression model, and g(x,y) is the pixel value of the input image. For example, if _RS is a region of 3x3 pixels, the input to the hierarchical ANN regression model is:

Therefore, the number of input layer units in this case of the hierarchical ANN regression model is 9. In the MTANN, since the output is a continuous value, a linear output ANN model (Non-Patent Document 2, Non-Patent Document 3) is used for the hierarchical ANN regression model. This is an ANN model that uses the following linear function instead of the commonly used sigmoid function as the response function of the output layer unit, and is very suitable for applications in which continuous values are output (Non-Patent Document 2, Non-Patent Document 3).

Therefore, the response functions of the units in the input layer, intermediate layer, and output layer are the identity function, the sigmoid function, and the linear function, respectively. The hierarchical ANN regression model is scanned like a convolution operation on the input image to obtain an output image. Also, the hierarchical ANN regression model is scanned like a convolution operation on the intermediate image obtained on the intermediate layer by the convolution operation, and a deeper layer is formed by repeating this.

The response function of the units in the intermediate layer can be a function other than a sigmoid function, for example, a quasi-linear function such as a rectified linear function, a ramp function (Rectified Linear: ReL), or a truncated power function.

Besides MTANN, other deep learning methods, such as convolutional neural networks (CN
N), shift-invariant neural networks, deep belief networks (DBN), deep neural
Neural networks (DNN), fully convolutional neural networks (FCN), U-Net, V-Net, SegNet, VGG-16, LeNet, AlexNet, ResNet, Auto encoders and decoders, Generative adversarial networks (GAN), Recurrent Neural Networks (RNN), Recursive Neural Networks, Long short-term memory (LSTM) can also be used. The hierarchical ANN regression model used as the core of the operation can be replaced with other machine learning models, for example, a support vector regression model, a nonlinear Gaussian process regression model (Non-Patent Document 4).

ただし、Ｔ（ｘ，ｙ）は教師画像、ｆ（ｘ，ｙ）は出力画像、Ｒ_Ｔは学習領域である。学習には、誤差逆伝播学習法を線形出力ＡＮＮモデル用に修正した学習法を用いる。線形出力ＡＮＮモデルでは、線形関数を出力層ユニットに用いているため、通常の階層型ＡＮＮ回帰モデルの修正量を決める偏微分が、次式となる。

（３）式の線形関数を微分すると、次式となる。

従って、中間層・出力層間の重みの修正量は、次式となる。

ここで、

とおくことにより、入力層・中間層間の重みの修正量も、誤差逆伝播学習法の導出と同様に、求めることができる。ＭＴＡＮＮでは、対象に応じて入力画像と教師画像を所望のものに変え、学習を行うことにより、様々な対象に対する画像処理を実現可能である。 <MTANN Study>
Fig. 17 is a diagram showing the learning of MTANN. MTANN acquires the desired image processing by learning by providing an input image and an ideal teacher image for it. The error to be minimized by learning is defined by the following equation.

where T(x, y) is the teacher image, f(x, y) is the output image, and R _T is the learning region. A learning method modified from the backpropagation learning method for a linear output ANN model is used for learning. In a linear output ANN model, a linear function is used in the output layer unit, so the partial differential that determines the amount of correction for a normal hierarchical ANN regression model is expressed by the following equation.

Differentiating the linear function of equation (3) gives the following equation:

Therefore, the correction amount of the weight between the intermediate layer and the output layer is expressed by the following formula.

here,

By setting the weights between the input layer and the hidden layer, the weights can be modified in the same way as in the backpropagation learning method. In the MTANN, the input image and the teacher image are changed according to the target, and learning is performed, making it possible to realize image processing for various targets.

(Actions and Effects of the Second Embodiment)
The stained image estimator learning device 400 acquires an autofluorescence image of a biological material sample such as a cell or tissue captured by an autofluorescence image capturing device 500. The stained image estimator learning device 400 acquires a stained image stained with a staining solution captured by a stained image capturing device 600. The stained image estimator learning device 400 constructs a stained image estimator for each color based on the autofluorescence image and the stained image using MTANN or the like, which is an image output type deep learning.

The image processing device 700 generates an estimated simulated stained image using an autofluorescence image of the sample and a stained image estimator for each color corresponding to the desired staining solution. The image processing device 700 can generate a simulated stained image without estimating the amount of pigment due to staining by constructing a stained image estimator for each color based on the autofluorescence image and the stained image for each color.

The image processing device 700 can generate a simulated stained image (digital stained image) using autofluorescence, which has traditionally been considered noise during staining, and can obtain a stained image without staining the specimen (sample). The image processing device 700 can obtain a simulated stained image without invasively inspecting even valuable samples. The image processing device 700 can easily observe unstained samples without the need for a special microscope for observing unstained samples, etc. Furthermore, since the image processing device 700 does not stain the sample, it can generate simulated stained images using various staining solutions with a single sample.

The method for generating a simulated stained image in each embodiment can be applied to any tissue that exhibits autofluorescence.

Computer-readable recording medium
A program for causing a computer or other machine or device (hereinafter, referred to as a computer, etc.) to realize any of the above functions can be recorded on a recording medium readable by the computer, etc. Then, the computer, etc. can provide the function by reading and executing the program from the recording medium.

Here, a computer-readable recording medium refers to a recording medium that stores information such as data and programs through electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer. Such a recording medium may be provided with elements that constitute a computer, such as a CPU and memory, and the CPU may be made to execute a program.

In addition, among such recording media, examples that can be removed from a computer include flexible disks, optical magnetic disks, CD-ROMs, CD-R/Ws, DVDs, DATs, 8mm tapes, memory cards, etc.

In addition, examples of recording media fixed to computers include hard disks and ROMs.

(others)
Although the embodiments of the present invention have been described above, these are merely examples and the present invention is not limited to these. Various modifications based on the knowledge of those skilled in the art, such as combinations of each component, are possible without departing from the spirit of the claims.

10: System 100: Image processing device 102: Image acquisition section 104: Color information acquisition section 106: Dye amount detection section 108: Dye amount estimation section 110: Image generation section 200: Fluorescence microscope 300: Multispectral camera 40: System 400: Stained image estimator learning device 410: Image acquisition section 420: Positioning section 430: Stained image estimator learning section 500: Autofluorescence image capturing device 510: Fluorescence microscope 520: Imaging section 600: Stained image capturing device 610: Optical microscope 620: Imaging section 70: System 700: Image processing device 710: Image acquisition section 740: Simulated stained image generating section 90: Information processing device 91: Processor 92: Memory 93: Storage section 94: Input section 95: Output section 96: Communication control section

Claims (8)

an image acquisition unit that acquires an autofluorescence image by capturing autofluorescence of a biological material sample due to a predetermined excitation light, and acquires stained images by capturing the biological material sample stained with a predetermined staining solution in a plurality of frequency bands using a multi-wavelength spectroscopic camera;
a stained image estimator learning unit that learns a plurality of stained image estimators that estimate images of respective frequency band components of the stained image captured in the plurality of frequency bands based on the autofluorescence image and the stained image acquired by the image acquisition unit;
A stain image estimator learning device comprising: The stain image estimator learning device according to claim 1, wherein the stain image estimator learning unit learns the stain image estimator by performing machine learning using the autofluorescence image as input data and the stain image as training data for the input data. an image acquisition unit for acquiring an autofluorescence image obtained by capturing autofluorescence of a biological material sample by using a predetermined excitation light;
a plurality of stained image estimators that estimate simulated stained images simulating images of frequency band components obtained by photographing the biological material sample, which is obtained by staining the biological material sample with a staining solution, in a plurality of frequency bands using a multi-wavelength spectroscopic camera, based on the autofluorescence image acquired by the image acquisition unit;
a simulated stained image generating unit that synthesizes a plurality of simulated stained images estimated by the plurality of stained image estimators ,
Image processing device. the stained image estimator is trained by performing machine learning using as input data an autofluorescence image obtained by capturing autofluorescence of another biological material sample using the excitation light, the autofluorescence image being different from the autofluorescence image, and using as training data for the input data a stained image obtained by capturing the other biological material sample stained with the staining solution.
The image processing device according to claim 3 . A stain image estimator learning device
Acquiring an autofluorescence image by capturing autofluorescence of a biological material sample using a predetermined excitation light;
The biological material sample is stained with a predetermined staining solution, and a stained image is acquired by photographing the biological material sample in a plurality of frequency bands using a multi-wavelength spectroscopic camera;
training a plurality of stained image estimators that estimate images of respective frequency band components of the stained images captured in the plurality of frequency bands based on the acquired autofluorescence image and the stained image;
A stain image estimator learning method that performs the above. The image processing device
Acquiring an autofluorescence image by capturing autofluorescence of a biological material sample using a predetermined excitation light;
using a plurality of stained image estimators to estimate simulated stained images simulating images of frequency band components obtained by photographing the biological material sample in a plurality of frequency bands by a multi-wavelength spectroscopic camera, the biological material sample being stained with a staining solution, based on the acquired autofluorescence image ;
synthesizing the plurality of simulated stained images estimated by the plurality of stained image estimators ;
An image processing method that performs the above. A stain image estimator learning device
Acquiring an autofluorescence image by capturing autofluorescence of a biological material sample using a predetermined excitation light;
The biological material sample is stained with a predetermined staining solution, and a stained image is acquired by photographing the biological material sample in a plurality of frequency bands using a multi-wavelength spectroscopic camera;
training a plurality of stained image estimators that estimate images of respective frequency band components of the stained images captured in the plurality of frequency bands based on the acquired autofluorescence image and the stained image;
A stain image estimator learning program to carry out this. The image processing device
Acquiring an autofluorescence image by capturing autofluorescence of a biological material sample using a predetermined excitation light;
a plurality of simulated stained images simulating images of frequency band components obtained by photographing the biological material sample in a plurality of frequency bands by a multi-wavelength spectroscopic camera, the simulated stained images being obtained based on the acquired autofluorescence image , the simulated stained images being estimated using a plurality of stained image estimators , and the estimated plurality of simulated stained images being synthesized ;
An image processing program to carry out this task.

Images