JP6901072B2 - Image processing device, image processing method, image processing program - Google Patents

Description

The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

Specimens of cells and tissues used for pathological diagnosis and the like are often colorless and transparent, and may be difficult to observe. Therefore, in order to facilitate observation with a microscope, cells and tissues are stained in advance and then observed.

Japanese Unexamined Patent Publication No. 2013-113689 Japanese Unexamined Patent Publication No. 2011-185843 Japanese Unexamined Patent Publication No. 2008-51772 Japanese Unexamined Patent Publication No. 2013-114233 Japanese Unexamined Patent Publication No. 2012-233784

Staining makes it easier to observe the state of tissues and cells, but the problem is that staining takes time. Further, when dyed, it is required to perform a treatment such as decolorization. Therefore, it is required to make it easier to observe tissues and cells without staining.

An object of the present invention is to provide a technique for observing tissues and cells without staining.

The following means are adopted to solve the above problems.

That is, the first aspect is
An image acquisition unit that acquires an autofluorescent image obtained by photographing the autofluorescence of the biological material sample with a predetermined excitation light source, and acquires a stained image obtained by photographing the biological material sample stained with a predetermined staining solution in a plurality of bands.
A color information acquisition unit that acquires color information at each position from the dyed images of a plurality of bands,
Based on the color information of the stained image and the spectral absorption coefficient of the dyeing solution, a dye amount image showing the distribution of the dyeing amount of the dyeing solution incorporated into the biological material sample is generated, and the autofluorescent image and the autofluorescent image and the above. A dye amount detection unit for generating a dye amount estimator that analyzes the relationship with the dye amount image and estimates the dye amount when the biological substance sample of the autofluorescent image is stained with the staining solution is provided.
The image acquisition unit acquires another autofluorescent image different from the autofluorescent image, and obtains the autofluorescent image.
The dye amount detection unit estimates the amount of dye when the biological substance sample of the other autofluorescent image is stained with the staining solution by using the other autofluorescent image and the dye amount estimator.
It is an image processing device.

The aspect of disclosure may be realized by executing the program by an information processing device. That is, the structure of the disclosure can be specified as a program for causing the information processing apparatus to execute the process executed by each means in the above-described embodiment, or as a computer-readable recording medium on which the program is recorded. Further, the structure of the disclosure may be specified by a method in which the information processing apparatus executes the processing executed by each of the above-mentioned means. The configuration of the disclosure may be specified as a system including an information processing device that performs processing executed by each of the above means.

According to the present invention, it is possible to provide a technique for observing tissues and cells without staining.

FIG. 1 is a diagram showing a configuration example of the system of the embodiment. FIG. 2 is a diagram showing a functional configuration example of the image processing device. FIG. 3 is a diagram showing a hardware configuration example of the information processing device. FIG. 4 is a diagram showing an example of an operation flow when generating a dye amount estimator in an image processing apparatus. FIG. 5 is a diagram showing an example (1) of an autofluorescent image of a biological substance (liver tissue) taken with a fluorescence microscope. FIG. 6 is a diagram showing an example (2) of an autofluorescent image of a biological substance (liver tissue) taken with a fluorescence microscope. FIG. 7 is a diagram showing an example (3) of an autofluorescent image of a biological substance (liver tissue) taken with a fluorescence microscope. FIG. 8 is a diagram showing an example of a graph showing the wavelength dependence of the spectral absorption coefficient of a dyeing solution or the like. FIG. 9 is a diagram showing an example of an operation flow when generating a simulated dyed image in an image processing apparatus.

Hereinafter, embodiments will be described with reference to the drawings. The configuration of the embodiment is an example, and the configuration of the invention is not limited to the specific configuration of the disclosed embodiment. In carrying out the invention, a specific configuration according to the embodiment may be appropriately adopted.

[Embodiment]
(Configuration example)
FIG. 1 is a diagram showing a configuration example of the system of the present embodiment. The system of this embodiment includes an image processing device 100, a fluorescence microscope 200, and a multispectral camera 300.

The image processing apparatus 100 acquires an autofluorescent image of the sample from the fluorescence microscope 200, and acquires a stained image of the sample for each band from the multispectral camera 300. The image processing device 100 detects the amount of dye from the stained image. The image processing apparatus 100 creates a dye amount estimator by associating the detected dye amount with the autofluorescent image. The image processing apparatus 100 uses the generated dye amount estimator to output a stained image based on another autofluorescent image. The sample is, for example, a biological substance such as a cell or a tissue. The image processing apparatus 100, the fluorescence microscope 200, and the multispectral camera 300 are communicably connected to each other directly or via a network or the like.

The fluorescence microscope 200 is a microscope for observing a sample by irradiating the sample with excitation light and observing autofluorescence from the sample. The fluorescence microscope 200 can irradiate a sample with a plurality of types of excitation light (excitation light having different wavelengths) and acquire an autofluorescent image of each.

The multispectral camera 300 is a camera capable of shooting a shooting target for each band (frequency band, wavelength band) divided into a plurality of bands. The multispectral camera 300 captures the stained sample band by band.

FIG. 2 is a diagram showing a functional configuration example of the image processing device. The image processing device 100 includes an image acquisition unit 102, a color information acquisition unit 104, a dye amount detection unit 106, a dye amount estimation unit 108, and an image generation unit 110.

The image acquisition unit 102 acquires an autofluorescent image of the sample from the fluorescence microscope 200, and acquires a stained image of the sample from the multispectral camera 300.

The color information acquisition unit 104 acquires color information from the stained image of the stained sample taken by the multispectral camera 300.

The dye amount detection unit 106 estimates the dye amount from the color information of the acquired multi-band stained image.

The dye amount estimation unit 108 generates a dye amount estimator based on the amount of dye in the autofluorescent image and the dyed image.

The image generation unit 110 generates a digitally stained image (simulated stained image) using a dye amount estimator based on the autofluorescent image of the sample taken by the fluorescence microscope 200.

FIG. 3 is a diagram showing a hardware configuration example of the information processing device. The information processing device 90 shown in FIG. 3 has a general computer configuration. The image processing device 100 is realized by using the information processing device 90 as shown in FIG. The information processing device 90 of FIG. 3 includes a processor 91, a memory 92, a storage unit 93, an input unit 94, an output unit 95, and a communication control unit 96. These are connected to each other by a bus. The memory 92 and the storage unit 93 are computer-readable recording media. The hardware configuration of the information processing device is not limited to the example shown in FIG. 3, and components may be omitted, replaced, or added as appropriate.

The information processing device 90 meets a predetermined purpose by having the processor 91 load the program stored in the recording medium into the work area of the memory 92 and execute the program, and each component or the like is controlled through the execution of the program. The function can be realized.

The processor 91 is, for example, a CPU (Central Processing Unit) or a DSP (Digital Signal Processor).

The memory 92 includes, for example, a RAM (Random Access Memory) and a ROM (Read Only Memory). The memory 92 is also called a main storage device.

The storage unit 93 is, for example, an EPROM (Erasable Programmable ROM) or a hard disk drive (HDD, Hard Disk Drive). Further, the storage unit 93 can include a removable medium, that is, a portable recording medium. The removable media is, for example, a USB (Universal Serial Bus) memory or a disc recording medium such as a CD (Compact Disc) or a DVD (Digital Versatile Disc). The storage unit 93 is also called a secondary storage device.

The storage unit 93 stores various programs, various data, and various tables used in the information processing device 90 in a readable and writable recording medium. The storage unit 93 stores an operating system (OS), various programs, various tables, and the like. The information stored in the storage unit 93 may be stored in the memory 92. Further, the information stored in the memory 92 may be stored in the storage unit 93.

The operating system is software that mediates between software and hardware, manages memory space, manages files, manages processes and tasks, and so on. The operating system includes a communication interface. The communication interface is a program that exchanges data with other external devices and the like connected via the communication control unit 96. External devices and the like include, for example, other information processing devices, external storage devices, and the like.

The input unit 94 includes a keyboard, a pointing device, a wireless remote controller, a touch panel, and the like. Further, the input unit 94 can include a video or image input device such as a camera or an audio input device such as a microphone.

The output unit 95 includes a display device such as an LCD (Liquid Crystal Display), an EL (Electroluminescence) panel, a CRT (Cathode Ray Tube) display, a PDP (Plasma Display Panel), and an output device such as a printer. Further, the output unit 95 can include an audio output device such as a speaker.

The communication control unit 96 connects to another device and controls communication between the information processing device 90 and the other device. The communication control unit 96 is, for example, a LAN (Local Area Network) interface board, a wireless communication circuit for wireless communication, and a communication circuit for wired communication. The LAN interface board and wireless communication circuit are connected to a network such as the Internet.

The information processing device 90 deploys a program stored in the auxiliary storage unit in an executable manner in a work area of the main storage unit, and controls peripheral devices and the like through the execution of the program. As a result, the information processing apparatus can realize a function that meets a predetermined purpose. The main storage unit and the auxiliary storage unit are recording media that can be read by the information processing apparatus.

(Operation example)
<Dye amount estimator generation>
FIG. 4 is a diagram showing an example of an operation flow when generating a dye amount estimator in an image processing apparatus.

In S101, the image acquisition unit 102 of the image processing apparatus 100 acquires an autofluorescent image of the sample taken by the fluorescence microscope 200 from the fluorescence microscope 200. The autofluorescence image is obtained by irradiating the sample with excitation light having a predetermined frequency and photographing the autofluorescence of the sample. In the fluorescence microscope 200, the sample is irradiated with excitation light of a plurality of types (a plurality of wavelengths), and the autofluorescence of the sample with respect to each excitation light is photographed. The sample is, for example, a section sample of a biological substance such as a cell or a tissue. The samples used in S101 and S102 are section specimens produced from the same biological material. The acquired autofluorescent image may be due to one type of excitation light.

5, FIG. 6 and FIG. 7 are diagrams showing an example of an autofluorescent image of a biological substance (liver tissue) taken with a fluorescence microscope. In FIGS. 5, 6 and 7, the whiter the color, the brighter it is. In the autofluorescent image of FIG. 5, the wavelength of the excitation light is 470 nm, and a filter that passes light having a wavelength of 495 nm or more is used. In the autofluorescent image of FIG. 6, the wavelength of the excitation light is 545 nm, and a filter that passes light having a wavelength of 565 nm or more is used. In the autofluorescent image of FIG. 7, the wavelength of the excitation light is 360 nm, and a filter that passes light having a wavelength of 400 nm or more is used. The autofluorescent image does not contain excitation light.

In S102, the image acquisition unit 102 acquires a stained image of the stained sample taken by the multispectral camera 300 from the multispectral camera 300. The stained image was taken by staining the sample with a predetermined staining solution. By adding a predetermined staining solution to the sample, a predetermined portion is colored. The part to be colored and the color depend on the dyeing solution. The multispectral camera 300 captures stained images in a plurality of bands (frequency bands). The stains are, for example, Haematoxylin and Eosin. As the stain solution, a plurality of stain solutions may be used. By using the multispectral camera 300, more detailed information on the transmitted light transmitted through the sample can be obtained.

In S103, the color information acquisition unit 104 obtains the absorbance of each pixel (position) in the image from the stained image of each band. Absorbance is a quantity that indicates how much the intensity of light weakens when it passes through a medium. The absorbance of each pixel depends on the brightness of each pixel. The color information acquisition unit 104 obtains the absorbance of each pixel from the brightness of each pixel in the dyed image for each band. The absorbance of each band depends on the amount of dye in the stain and the extinction coefficient of the stain taken into the sample.

これらの式より、推定行列Ｘ^＋を吸光度ベクトルａに乗算することによって、試料に取り込まれた染色液毎のピクセル毎（位置毎）の色素量Ｃを求めることができる。色素量検出部１０６は、ピクセル毎の色素量に基づいて、試料の各位置の色素量を示す色素量画像を生成する。色素量画像は、試料に取り込まれた染色液の分布を示す。色素量画像では、例えば、染色液が多く取り込まれた位置はより明るくなる。 In S104, the dye amount detection unit 106 calculates the dye amount of the dye solution taken into the sample for each pixel based on the absorbance obtained in S103 and the known spectral absorption coefficient of the stain solution. The spectral absorption coefficient is a quantity that indicates how much light is absorbed by the medium when it is incident on the medium. The absorption coefficient of the stain depends on the wavelength of the incident light. Assuming that the absorbance vector is a, the spectroscopic absorption coefficient matrix is X, and the amount of dye is C, it is expressed as follows according to Lambertvale's law. The absorbance vector is a vector representation of the absorbance for each band. The spectral absorption coefficient matrix is a matrix display of the absorption coefficients for each staining solution and each band.

From these equations, the dye amount C for each pixel (for each position) of each stain taken into the sample can be obtained by multiplying the ^{estimation matrix X + by the absorbance vector a.} The dye amount detection unit 106 generates a dye amount image showing the dye amount at each position of the sample based on the dye amount for each pixel. The dye amount image shows the distribution of the stain solution taken into the sample. In the dye amount image, for example, the position where a large amount of the staining liquid is taken in becomes brighter.

FIG. 8 is a diagram showing an example of a graph showing the wavelength dependence of the spectral absorption coefficient of a dyeing solution or the like. In the example of FIG. 8, the spectral absorption coefficients of hematoxylin, eosin, and erythrocytes are shown. In the example of FIG. 8, for example, the spectral absorption coefficient of eosin peaks near 550 nm. The spectral absorption coefficient matrix is generated based on the graph.

In S105, the dye amount detection unit 106 analyzes the relationship between the autofluorescent image acquired in S101 and the dye amount image generated in S104 by machine learning or the like. For the analysis, a method using a learning space such as deep learning by a neural network, Regression SVM, Regression Random Forest, multiple regression analysis, Look Up Table, etc. can be used. Any method that predicts continuous values can be applied to the analysis. For example, in the case of a neural network, the autofluorescent image of a sample excited under a plurality of conditions (excitation light) is used as input data, and the dye amount image obtained from the multispectral image obtained by staining the sample is output data. And. The dye amount detection unit 106 constructs a dye amount estimator by optimizing the weighting coefficient between neurons of the neural network by supervised learning in which the input data and the output data are linked. The dye amount estimator is optimized by using more pairs of autofluorescent images and dye amount images. In addition, the dye amount estimator can be generated for each staining solution. Further, as the input data, an image processing feature amount such as texture information (information indicating the features of tissues and cells, etc.) may be included.

Further, by generating a dye amount estimator using a dyed image obtained by dyeing a sample with various dyeing solutions, it is possible to generate a dye amount estimator for various dyeing solutions.

<Simulated stained image generation>
FIG. 9 is a diagram showing an example of an operation flow when generating a simulated dyed image in an image processing apparatus. The simulated stained image (digital stained image) is a simulated stained image obtained based on an estimated dye amount image estimated by a dye amount estimator using an autofluorescent image of a sample. The simulated stained image is a simulation of a stained image obtained by actually staining a sample. Here, the image processing apparatus 100 generates a simulated stained image by using the autofluorescent image of the unstained sample and the above-mentioned dye amount estimator.

In S201, the image acquisition unit 102 of the image processing apparatus 100 acquires an autofluorescent image of the sample taken by the fluorescence microscope 200 from the fluorescence microscope 200. In the fluorescence microscope 200, the sample is irradiated with excitation light of a plurality of types (a plurality of wavelengths), and the autofluorescence of the sample with respect to each excitation light is photographed. The sample is, for example, a section sample of a biological substance such as a cell or a tissue. The sample corresponding to the autofluorescent image used here is a sample different from the sample corresponding to the autofluorescent image used in the generation of the dye amount estimator.

In S202, the dye amount estimation unit 108 uses the autofluorescent image acquired by the image acquisition unit 102 as input data, and uses a dye amount estimator corresponding to the desired dye solution to estimate the dye of the desired dye solution. Generate a quantity image. In the pigment amount image, for example, the larger the pigment amount, the brighter the color is displayed. The dye amount image is an image showing the dye amount of the staining solution taken into the estimated sample.

In S203, the image generation unit 110 calculates the absorbance for each pixel for each band from the dye amount image (or dye amount) and the spectral absorption coefficient for each dyeing solution. The image generation unit 110 calculates the light transmittance for each band from the calculated absorbance for each band, integrates each pixel for each RGB (Red Green Blue) component, and the pixel value (RGB value) of each pixel. ) Is calculated. The image generation unit 110 generates a simulated dyed image (digital dyed image) based on the calculated pixel value of each pixel. As a result, the image processing apparatus 100 can generate a pseudo-stained image of the desired dyeing solution without staining the sample.

(Action and effect of the embodiment)
The image processing apparatus 100 acquires an autofluorescent image of a sample of a biological substance such as a cell or tissue taken by a fluorescence microscope 200. The image processing device 100 acquires a stained image dyed with a dyeing solution taken by the multispectral camera 300. The image processing apparatus 100 calculates the amount of dye in the staining solution taken into the sample based on the stained image and the spectral absorption coefficient of the staining solution. The image processing apparatus 100 generates a dye amount estimator based on the autofluorescent image and the calculated dye amount (dye amount image).

The image processing apparatus 100 uses an autofluorescent image of the sample and a dye amount estimator corresponding to a desired staining solution to generate an estimated dye amount image. The image processing apparatus 100 calculates the absorbance for each band from the dye amount image and the spectral absorption coefficient for each dye solution. The image processing device 100 calculates the transmittance for each band from the absorbance for each band, integrates the transmittance for each band for each RGB component, calculates the pixel value of each pixel, and generates a pseudo-stained image. To do.

The image processing apparatus 100 uses autofluorescence, which has conventionally been regarded as noise during dyeing, to estimate the amount of dye at the time of dyeing and generate a digitally dyed image (simulated dyed image) to generate a sample (simulated dyed image). A stained image can be obtained without staining the sample). According to the image processing apparatus 100, even a valuable sample can obtain a simulated stained image without invading the sample. Unstained specimens can be easily observed without preparing a special microscope for observing unstained specimens and the like. Further, according to the image processing apparatus 100, since the sample is not stained, it is possible to generate a pseudo-stained image with various staining solutions with one sample.

The method of generating a simulated stained image of the present embodiment is applicable to any autofluorescent tissue.

<Computer readable recording medium>
A program that enables a computer or other machine or device (hereinafter, computer or the like) to realize any of the above functions can be recorded on a recording medium that can be read by the computer or the like. Then, the function can be provided by causing a computer or the like to read and execute the program of this recording medium.

Here, a recording medium that can be read by a computer or the like is a recording medium that can store information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read from the computer or the like. To say. In such a recording medium, elements constituting a computer such as a CPU and a memory may be provided, and the CPU may execute a program.

Among such recording media, those that can be removed from a computer or the like include, for example, flexible discs, magneto-optical discs, CD-ROMs, CD-R / Ws, DVDs, DATs, 8 mm tapes, memory cards, and the like.

In addition, there are hard disks, ROMs, and the like as recording media fixed to computers and the like.

(Other)
Although the embodiments of the present invention have been described above, these are merely examples, and the present invention is not limited thereto, and as long as the gist of the claims is not deviated, combinations of each configuration, etc., etc. Various changes can be made based on the knowledge of those skilled in the art.

100 Image processing device 102 Image acquisition unit 104 Color information acquisition unit 106 Dye amount detection unit 108 Dye amount estimation unit 110 Image generation unit 200 Fluorescence microscope 300 Multispectral camera
90 Information processing equipment
91 processor
92 memory
93 Memory
94 Input section
95 Output section
96 Communication control unit

Claims (4)

An image acquisition unit that acquires an autofluorescent image obtained by photographing the autofluorescence of the biological material sample with a predetermined excitation light source, and acquires a stained image obtained by photographing the biological material sample stained with a predetermined staining solution in a plurality of bands.
A color information acquisition unit that acquires color information at each position from the dyed images of a plurality of bands,
Based on the color information of the stained image and the spectral absorption coefficient of the dyeing solution, a dye amount image showing the distribution of the dyeing amount of the dyeing solution incorporated into the biological material sample is generated, and the autofluorescent image and the autofluorescent image and the above. A dye amount detection unit for generating a dye amount estimator that analyzes the relationship with the dye amount image and estimates the dye amount when the biological substance sample of the autofluorescent image is stained with the staining solution is provided.
The image acquisition unit acquires another autofluorescent image different from the autofluorescent image, and obtains the autofluorescent image.
The dye amount detection unit estimates the amount of dye when the biological substance sample of the other autofluorescent image is stained with the staining solution by using the other autofluorescent image and the dye amount estimator.
Image processing device. An image generation unit is provided that generates a simulated stained image in which the biological material sample of the other autofluorescent image simulates staining with the staining solution based on the amount of dye in the other autofluorescent image.
The image processing apparatus according to claim 1. The image processing device
An autofluorescent image obtained by photographing the autofluorescence of the biological material sample with a predetermined excitation light source was acquired, and a stained image obtained by photographing the biological material sample stained with a predetermined staining solution in a plurality of bands was acquired.
The color information of each position is acquired from the dyed images of a plurality of bands, and the color information is obtained.
Based on the color information of the stained image and the spectral absorption coefficient of the dyeing solution, a dye amount image showing the distribution of the dyeing amount of the dyeing solution incorporated into the biological material sample is generated, and the autofluorescent image and the autofluorescent image and the above. By analyzing the relationship with the dye amount image, a dye amount estimator for estimating the dye amount when the biological substance sample of the autofluorescent image was stained with the staining solution was generated.
Obtain another autofluorescent image different from the autofluorescent image,
Using the other autofluorescent image and the dye amount estimator, the amount of dye when the biological material sample of the other autofluorescent image is stained with the staining solution is estimated.
Image processing method to do that. The image processing device
An autofluorescent image obtained by photographing the autofluorescence of the biological material sample with a predetermined excitation light source was acquired, and a stained image obtained by photographing the biological material sample stained with a predetermined staining solution in a plurality of bands was acquired.
The color information of each position is acquired from the dyed images of a plurality of bands, and the color information is obtained.
Based on the color information of the stained image and the spectral absorption coefficient of the dyeing solution, a dye amount image showing the distribution of the dyeing amount of the dyeing solution incorporated into the biological material sample is generated, and the autofluorescent image and the autofluorescent image and the above. By analyzing the relationship with the dye amount image, a dye amount estimator for estimating the dye amount when the biological substance sample of the autofluorescent image was stained with the staining solution was generated.
Obtain another autofluorescent image different from the autofluorescent image,
Using the other autofluorescent image and the dye amount estimator, the amount of dye when the biological material sample of the other autofluorescent image is stained with the staining solution is estimated.
An image processing program to do that.

Images