WO2021070371A1

WO2021070371A1 - Cell image analysis method and cell analysis device

Info

Publication number: WO2021070371A1
Application number: PCT/JP2019/040275
Authority: WO
Inventors: 秋絵外口
Original assignee: 株式会社島津製作所
Priority date: 2019-10-11
Filing date: 2019-10-11
Publication date: 2021-04-15
Also published as: JPWO2021070371A1; JP7248139B2

Abstract

One aspect of a cell analysis device according to the present invention is provided with: a holographic microscope (10); an image creation unit (23, 24) for creating a phase image of a cell on the basis of hologram data; a first learning model storage unit (251) for storing a nuclear region learning model that is created by performing machine learning through setting the phase image of the cell as an input image and using learning data in which a pseudo nuclear region image corresponding to the input image and based on a stain image obtained by staining a cell nucleus is defined as a correct image; a second learning model storage unit (261) for storing a cell region learning model that is created by performing machine learning through setting the phase image of the cell as an input image and using learning data in which a pseudo cell region image corresponding to the input image and based on the stain image obtained by staining cytoskeletons is defined as a correct image; a nuclear region inference unit (25) for obtaining, as an output image, a nuclear region inference image indicating the region of the cell nucleus by using the nuclear region learning model and setting, as an input image, the phase image created for the cell being analyzed; a cell region inference unit (26) for obtaining, as an output image, a cell region inference image indicating a cell region by using the cell region learning model and setting, as an input image, the phase image for the cell being analyzed; and a nuclear region extraction unit (27) for extracting a cell nucleus present in a range inferred to be the cell region by using the nuclear region inference image and the cell region inference image.

Description

Cell image analysis method and cell analysis device

The present invention relates to a cell image analysis method for analyzing and processing an observation image obtained for a cell, and a cell analysis device using the method.

The cell nucleus (hereinafter, sometimes simply referred to as the "nucleus" according to convention) contains DNA that controls the genetic information of the cell, and is one of the most important organs in the cell, which is responsible for cell division and gene expression control. Therefore, observation of nuclei in microscopic images of cells is very important for various cell-related research and product development.

It is generally considered difficult to observe a nucleus as a living cell. For example, Patent Document 1 discloses a technique for observing a morphology after staining a nucleus with a simple operation. .. Observation of cell nuclei is also important in the field of pathological image analysis. For example, in Non-Patent Document 1, cell nuclei are extracted from images (pathological images) obtained of cells stained with cell nuclei and proteins per cell are quantified. The technology to be used is disclosed. Further, Patent Documents 2 and 3 disclose a technique for detecting a region of a cell nucleus by using a machine learning learning model for an image obtained by staining a cell or a cell nucleus with a microscope. Furthermore, Patent Document 4 discloses a technique for measuring the number of cell nuclei using a cell image (pathological image) obtained by staining cell nuclei.

However, as described above, the method of staining cells and cell nuclei is an invasive method, and the cells used for observation are continuously cultured, or the cells used for observation are used as they are for another purpose, for example. It cannot be administered to patients as a product for regenerative medicine. In research and development of regenerative medicine using pluripotent stem cells such as iPS cells and ES cells, it is necessary to culture a large amount of undifferentiated cells in a state of maintaining pluripotency. For that purpose, it is necessary to frequently check the state of cells in culture, but in such a field, the invasive observation method as described above cannot be used. Therefore, there is a strong demand for the establishment of a non-invasive and non-destructive method for observing cell nuclei.

On the other hand, Patent Document 5 discloses a technique for determining whether a cell is alive or dead in a non-staining manner, that is, non-invasively, using a microscope using Raman scattered light. However, although Raman imaging can obtain spectrum information on the corresponding pixels in the image, it requires complicated processing such as the need to specify the spectrum that contributes to the determination performed in the living cell. In addition, although cell nuclei can be observed without staining with a microscope using Raman scattered light, not only complicated preliminary studies such as spectrum identification are required, but also observations that can be taken in a short time are possible. There is a restriction that the area is small. Therefore, it is not suitable for observing the nuclei of living cells existing in the entire well in which the cells are cultured. It is also impossible to measure the total number of cell nuclei in the well in which the cells are cultured.

International Patent Publication No. 2012/095935 Japanese Unexamined Patent Publication No. 2019-95853 Japanese Unexamined Patent Publication No. 2019-91307 Japanese Patent No. 5924406 International Patent Publication No. 2014/162744 International Patent Publication No. 2016/162945

The present invention has been made to solve the above problems, and its main purpose is to observe and analyze the nuclei of cells existing in a wide range in a container such as a well in a non-invasive manner. It is to provide a cell image analysis method and a cell analysis apparatus which can be performed.

One aspect of the cell image analysis method according to the present invention, which has been made to solve the above problems, is
The phase image of the cell created based on the hologram data acquired by the holographic microscope is used as the input image, and the corresponding pseudo nuclear region image based on the stained image obtained by staining the cell nucleus is the correct image. The first learning model creation step of creating a nuclear region learning model by performing machine learning using the training data of
By performing machine learning using the learning data in which the phase image of the cells is used as the input image and the corresponding pseudo cell region image based on the stained image obtained by staining the cytoskeleton is used as the correct image. The second learning model creation step to create the cell region learning model,
Using the nuclear region learning model, a nuclear region estimation step of acquiring a phase image of the cell to be analyzed as an input image and a nuclear region estimation image showing the region of the cell nucleus as an output image,
Using the cell region learning model, a cell region estimation step of acquiring a phase image of the cell to be analyzed as an input image and a cell region estimation image showing the cell region as an output image, and
Using the nuclear region estimation image and the cell region estimation image, a nuclear region extraction step of extracting cell nuclei existing in a range estimated to be a cell region, and a nuclear region extraction step.
It has.

Further, one aspect of the cell analyzer according to the present invention made to solve the above problems is
With a holographic microscope
An image creation unit that creates a phase image of the cells based on the hologram data obtained by observing the cells with the holographic microscope.
Using the learning data in which the phase image of the cell created based on the hologram data is used as the input image and the corresponding pseudo-nucleus region image based on the stained image obtained by staining the cell nucleus is used as the correct image. The first learning model storage unit that stores the nuclear region learning model created by performing machine learning,
Created by performing machine learning using learning data in which the phase image of the cells is used as an input image and the pseudo cell region image based on the stained image obtained by staining the corresponding cell skeleton is used as the correct image. A second learning model storage unit that stores the stored cell region learning model,
Using the nuclear region learning model stored in the first learning model storage unit, the phase image created by the image creation unit for the cell to be analyzed is used as an input image, and a nuclear region estimation image showing the region of the cell nucleus is output. The nuclear region estimation unit acquired as an image and
Using the cell region learning model stored in the second learning model storage unit, the phase image of the cell to be analyzed is used as an input image, and the cell region estimation image showing the cell region is acquired as an output image. Department and
Using the nuclear region estimation image and the cell region estimation image, a nuclear region extraction unit that extracts cell nuclei existing in a range estimated to be a cell region, and a nuclear region extraction unit.
Is provided.

The holographic microscope is typically a digital holographic microscope, and the phase image is, for example, an image reconstructed based on phase information obtained by calculation processing based on hologram data obtained by the digital holographic microscope. is there.

In the cell image analysis method and cell analyzer according to the above aspect of the present invention, a phase image of an appropriate cell and a nuclear stained image obtained by staining the cell nucleus of the same cell and observing it with a fluorescence microscope or the like are used. Then, a nuclear region learning model for identifying the region of the cell nucleus is created by machine learning such as deep learning. However, due to the principle of measurement, interference fringes may appear in the phase image reconstructed based on the hologram data, and when the nuclear region is detected using the above nuclear region learning model, the interference fringes on the phase image may appear. Due to the influence of, a part that is not originally a cell nucleus may be mistakenly detected as a cell nucleus. Further, even if a foreign substance such as dust is present on the medium in the culture vessel, it may be erroneously detected as a cell nucleus.

On the other hand, in the cell image analysis method and the cell analysis apparatus according to the above aspect of the present invention, a region in which the cytoskeleton is distributed is specified by using a stained image obtained by staining the cytoskeleton such as actin filament, in addition to the nuclear region learning model. Create a cell region learning model for this by machine learning. The cytoskeleton is a structural element that exists in a fibrous or reticular form throughout the cytoplasm and determines the shape and morphology of cells. Therefore, the distribution region of the cytoskeleton can be regarded as roughly indicating the cell shape. That is, the cell region estimation image obtained by using the cell region learning model in the cell region estimation step shows the division between the cell region and the extracellular region (background region). Naturally, the cell nucleus exists in the cell region, so compare the nuclear region estimation image and the cell region estimation image for the same imaging range, and if there is a cell nucleus existing in the background region other than the cell, it is a false detection of the nucleus. It can be judged that there is. By making such a judgment, the nuclear region extraction step can eliminate falsely detected cell nuclei and extract only more accurate cell nuclei.

In this way, according to the cell image analysis method and the cell analysis apparatus according to the above aspect of the present invention, the cell nucleus is accurately extracted from the phase image without performing invasive treatment such as staining of the cell to be analyzed. be able to. In particular, even when interference fringes appear in the phase image or an image due to a foreign substance appears, the influence thereof can be eliminated or reduced, and the cell nucleus can be observed accurately. Then, for example, by counting the number of the cell nuclei, information on the number of cells can be obtained. In addition, the true shape and morphology of the cell nucleus, or the state inside the cell nucleus can be observed. In addition, since the cells to be analyzed can be photographed (measured) non-invasively, the cells after imaging can be continuously cultured or used for analysis or observation for another purpose.

The schematic block diagram of one Embodiment of the cell analysis apparatus using the cell image analysis method which concerns on this invention. The conceptual diagram of the structure of the full-thickness convolutional neural network used in the cell analysis apparatus of this embodiment. The flowchart which shows the flow of processing at the time of making a learning model in the cell analysis apparatus of this embodiment. The flowchart which shows the flow of the process from the imaging of the cell to be analyzed to the output of the cell count result in the cell analysis apparatus of this embodiment. It is a figure which shows the processing process of the correct answer image used at the time of making a learning model in the cell analysis apparatus of this embodiment, the original image (a) of a nuclear-stained fluorescence image, the image (b) after background removal processing, and binarization. Image (c) after processing. The figure which shows an example of the IHM phase image obtained by the cell analyzer of this embodiment. FIG. 6 is a diagram showing an example of a nuclear region estimation image output by estimation using a learning model using the IHM phase image shown in FIG. 6 as an input image. The figure which shows the correct answer image of the nuclear region estimation image shown in FIG. 7. The figure which shows the image which superposed the nuclear position candidate point on the IHM phase image shown in FIG. The figure which shows the image which superposed the correct answer point of a nuclear region on the IHM phase image shown in FIG. The figure which shows an example of the actin filament stained image and the bright field image. FIG. 6 is a diagram showing a cell region estimation image (binary image) output by estimation using a learning model using the IHM phase image shown in FIG. 6 as an input image. The figure which shows the nuclear region estimation image after performing the mask processing using the cell region estimation image shown in FIG. The figure which shows the image which superposed the core position candidate point after mask processing on the IHM phase image shown in FIG.

Hereinafter, an embodiment of the cell image analysis method and the cell analysis apparatus according to the present invention will be described with reference to the accompanying drawings.
FIG. 1 is a block configuration diagram of a main part of a cell analysis apparatus according to an embodiment for carrying out the cell image analysis method according to the present invention.

The cell analysis device 1 of the present embodiment includes a microscopic observation unit 10, a control / processing unit 20, an input unit 30 and a display unit 40 which are user interfaces. Further, the cell analysis device 1 is provided with an FCN model creation unit 50.

The microscopic observation unit 10 is an in-line Holographic Microscopy (IHM), includes a light source unit 11 including a laser diode and an image sensor 12, and is located between the light source unit 11 and the image sensor 12. A culture plate 13 containing the cells 14 is placed.

The control / processing unit 20 controls the operation of the microscopic observation unit 10 and processes the data acquired by the microscopic observation unit 10, and includes the imaging control unit 21, the hologram data storage unit 22, and the phase information calculation. A functional block includes a unit 23, an image creation unit 24, a nuclear region estimation unit 25, a cell region estimation unit 26, a mask processing unit 27, a cell counting unit 28, and a display processing unit 29. The nuclear region estimation unit 25 includes a nuclear region learning model storage unit 251, and the cell region estimation unit 26 includes a cell region learning model storage unit 261.

The FCN model creation unit 50 includes a learning image data input unit 51, an image alignment processing unit 52, a stained image preprocessing unit 53, a stained image binarization unit 54, a learning execution unit 55, and a model construction unit 56. And are included as functional blocks. The learned learning model created by the FCN model creation unit 50 is stored in the storage unit of the control / processing unit 20, and functions as the nuclear region learning model storage unit 251 and the cell region learning model storage unit 261.

Usually, the entity of the control / processing unit 20 is a personal computer in which predetermined software is installed, a higher-performance workstation, or a computer system including a high-performance computer connected to such a computer via a communication line. is there. That is, the function of each block included in the control / processing unit 20 is stored in the computer or computer system, which is executed by executing software installed in a computer system including a single computer or a plurality of computers. It can be embodied by processing using various types of data.

Also, the entity of the FCN model creation unit 50 is a personal computer on which predetermined software is installed or a workstation with higher performance. Normally, this computer is a computer different from the control / processing unit 20, but it may be the same computer. That is, the control / processing unit 20 can have the function of the FCN model creation unit 50.

The work and processing up to the creation of the IHM phase image, which is an observation image of cells, will be described in the cell analysis device 1 of the present embodiment.
When the operator sets the culture plate 13 containing the cells 14 at a predetermined position and performs a predetermined operation on the input unit 30, the imaging control unit 21 controls the microscopic observation unit 10 and acquires hologram data by the following procedure. To do.

That is, the light source unit 11 irradiates a predetermined area of the culture plate 13 with coherent light having an angle spread of about 10 °. The coherent light (object light 16) transmitted through the culture plate 13 and the cells 14 reaches the image sensor 12 while interfering with the light (reference light 15) transmitted through the region close to the cells 14 on the culture plate 13. The object light 16 is light whose phase changes when it passes through the cell 14, while the reference light 15 is light that does not pass through the cell 14 and therefore does not undergo the phase change caused by the cell 14. Therefore, on the detection surface (image surface) of the image sensor 12, an image, that is, a hologram is formed by interference fringes between the object light 16 whose phase has been changed by the cells 14 and the reference light 15 whose phase has not changed.

The light source unit 11 and the image sensor 12 are sequentially moved in the X-axis direction and the Y-axis direction in conjunction with each other by a movement mechanism (not shown). As a result, the irradiation region (observation region) of the coherent light emitted from the light source unit 11 is moved on the culture plate 13, and the hologram data (2 of the hologram formed on the detection surface of the image sensor 12) covers a wide two-dimensional region. Dimensional light intensity distribution data) can be obtained.

As described above, the hologram data obtained by the microscopic observation unit 10 is sequentially sent to the control / processing unit 20 and stored in the hologram data storage unit 22. In the control / processing unit 20, the phase information calculation unit 23 reads the hologram data from the hologram data storage unit 22 and executes a predetermined arithmetic process for phase recovery to obtain the phase information of the entire observation area (photographing area). calculate. The image creation unit 24 creates an IHM phase image based on the calculated phase information. A well-known algorithm disclosed in Patent Document 6 and the like can be used when calculating such phase information and creating an IHM phase image.

FIG. 6 is an example of an IHM phase image. It can be seen that transparent cells are difficult to see with a general optical microscope, but individual cells can be observed fairly clearly in the IHM phase image. However, it is difficult to visually recognize the nucleus of each cell from this IHM phase image. Therefore, in the cell analysis device 1 of the present embodiment, a region in which a nucleus is presumed to exist in each cell is determined by using a full-layer convolutional neural network (FCN), which is one of the machine learning methods. Obtain the shown nuclear region estimation image.

FIG. 2 is a conceptual diagram of the structure of the FCN. The details of the structure and processing of FCN are explained in detail in many documents. It can also be implemented using commercially available or free software such as "MATLAB" provided by MathWorks in the United States. Therefore, a schematic description will be given here.

As shown in FIG. 2, the FCN includes, for example, a multi-layer network 60 in which repetitions of a convolutional layer and a pooling layer are multi-layered, and a convolutional layer 61 corresponding to a fully connected layer in a convolutional neural network. In this case, in the multi-layer network 60, the convolution process using a filter (kernel) of a predetermined size and the pooling process of reducing the convolution result two-dimensionally and extracting an effective value are repeated. However, the multilayer network 60 may be composed of only a convolutional layer without a pooling layer. Further, in the final stage convolution layer 61, local convolution and deconvolution are performed while sliding a filter of a predetermined size in the input image. In this FCN, by performing semantic segmentation on the input image 63 such as the IHM phase image, it is possible to output the segmentation image 64 in which the region of the cell nucleus is labeled on a pixel-by-pixel basis.

Here, the multilayer network 60 and the convolutional layer 61 are designed so as to label the input IHM phase image on a pixel-by-pixel basis. That is, the smallest unit of one region labeled in the segmentation image 64, which is the output image, is one pixel on the IHM phase image. Therefore, even if the size of the cell nucleus is about one pixel on the IHM phase image, the cell nucleus is detected as one region in the segmentation image 64.

In order to perform the above-mentioned semantic segmentation by FCN, the coefficients (weights) of the filters in each of the plurality of convolution layers included in the multilayer network 60 and the final convolution layer 61 are used in advance by using a large number of image data for learning. It is necessary to train and build a learning model.

Next, the operation of the learning process in the FCN model creation unit 50 will be described according to the flowchart shown in FIG. When learning, for example, learning can be performed using the stochastic gradient descent method that is often used in machine learning (particularly deep learning).

In the FCN model creation unit 50, the learning image data input unit 51 is a learning data (also called teacher data or training data) in which the IHM phase image created by the image creation unit 24 and the corresponding correct image are a set, but here. Then, a large number of sets (referred to as learning data) are read in advance (step S11). The IHM phase image is created based on the data obtained by actually photographing the cells with the cell analysis device 1 as described above, but the IHM phase image is not necessarily limited to a specific cell analysis device, and other cell analysis having the same configuration is performed. It may be the one obtained by the device. On the other hand, the correct image is a fluorescence image (nuclear-stained fluorescence image) obtained by staining only the nucleus of the cell when the IHM phase image was created and taking the image with an appropriate microscope. The staining method is not particularly limited as long as the cell nucleus can be stained, and for example, DAPI (4', 6-diamidino-2-phenylindole), propidium iodide, SYSTEMX (registered trademark), TO-PRO (registered trademark)- 3 and the like can be used.

It is desirable that the position, direction, and size of cells are exactly the same between the IHM phase image and the nuclear-stained fluorescence image, which are a set, but in general, the stained fluorescence image is taken in parallel with the imaging with a digital holographic microscope. It is inevitable that the position, direction, size, etc. of the cells will differ between the obtained IHM phase image and the nuclear-stained fluorescence image. Therefore, the image alignment processing unit 52 aligns both images by performing image processing such as translation, rotation, enlargement / reduction, etc. on one of the images (step S12). In general, it is advisable to perform image processing so as to align the position of the nuclear-stained fluorescence image with reference to the IHM phase image in which the cells are more clearly visible. This alignment work may be performed manually by the operator with reference to, for example, the edge of the well or the mark added to the culture plate, or may be automatically performed by a predetermined algorithm.

Next, the stained image preprocessing unit 53 performs noise removing processing and background removing processing so that the cell nucleus region becomes clearer in the nuclear stained fluorescent image (steps S13 and S14). The noise removal process aims to remove various types of noise, and for example, various types of filters such as a linear filter and a median filter can be used. Further, the background removing process is mainly intended to remove intensity unevenness in a background portion other than the cell nucleus, and a method using an average value filter or the like is known as a background subtraction process. As the noise removal processing and the background removal processing method itself, various methods used in the conventional image processing can be used. Further, since the noise condition depends on the characteristics of the microscope and the characteristics of the target sample, the noise removal process can be omitted in some cases.

The stained image binarization unit 54 binarizes the preprocessed image as described above to create a binary image in which the nuclear region and the other regions are clarified (step S15). Further, the stained image binarization unit 54 performs a closing process that combines an expansion process and a reduction process as a morphology conversion process on the binarized image (step S16).

5 (a) is an original image of a nuclear-stained fluorescent image, FIG. 5 (b) is an image of the image shown in FIG. 5 (a) after background removal processing, and FIG. 5 (c) is FIG. 5 (c). This is an example of a binary image after binarizing the image shown in a). The same applies to the images described below, but here, mesenchymal stem cells (MSCs) were targeted, and DAPI was used for nuclear staining.

In general, in a nuclear-stained fluorescent image, the intensity unevenness in the background portion other than the nuclear region is large, so that the binarization capable of extracting the nuclear region cannot be performed unless the background removal treatment is performed. On the other hand, by performing the background removal process in advance, as shown in FIG. 5C, it is possible to obtain a binary image in which the nuclear region is accurately extracted. This binary image is an image in which the cell nucleus region and the other regions are semantically segmented for each pixel on the corresponding IHM phase image. Further, as described above, by performing the closing process after binarization, it is possible to remove mainly small noise on the fluorescence image such as bright spot noise. However, the closing process may be omitted depending on the noise situation and the performance of the noise removal process in step S13.

By performing the processes of steps S12 to S15 for all the training data, if a large number of training data including a binary image of the IHM phase image and the nuclear-stained fluorescent image are prepared, the learning execution unit 55 can use the training execution unit 55. FCN learning is executed using a large number of training data (step S17). That is, the filter coefficients in a plurality of convolution layers in the FCN network are learned so that the result of semantic segmentation by FCN is as close as possible to the correct image. The model building unit 56 builds a model in the process of repeating the learning, and when the predetermined learning is completed, the model building unit 56 saves the learning model based on the learning result (step S18).
The data forming the learning model indicating the nuclear region created in this way is stored in the nuclear region learning model storage unit 251.

However, due to the principle of measurement by a holographic microscope, interference fringes appear not a little in the hologram data obtained by the microscopic observation unit 10, and this may also appear in the IHM phase image. In the IHM phase image shown in FIG. 6, a substantially concentric image derived from the interference fringes appears fairly clearly. When the interference fringes become clear on the IHM phase image, a part of the interference fringes may be erroneously detected as a cell nucleus when estimating the nuclear region using the nuclear region learning model.

FIG. 7 is a diagram showing an example of a nuclear region estimation image obtained by estimation using a nuclear region learning model using the IHM phase image shown in FIG. 6 as an input image. FIG. 8 is a diagram showing a correct image of the nuclear region estimation image shown in FIG. 7, that is, an accurate nuclear region. FIG. 9 is a diagram showing an image in which a point (nuclear position candidate point) which is a nuclear region estimation result is superimposed on the IHM phase image shown in FIG. FIG. 10 is a diagram showing an image in which the correct points of the nuclear region are superimposed on the IHM phase image shown in FIG. The points indicating the nuclear positions in FIGS. 9 and 10 are rectangular, and they can be obtained by performing a maximum value region extraction process and a binarization process, which will be described later, on the nuclear region represented by the gray scale. It was done.

Comparing FIGS. 9 and 10, it can be seen that in FIG. 9, a large number of false nuclear regions are detected on the interference fringes where the cell nuclei should not exist. This means that simply estimating the nuclear region using the nuclear position learning model will result in a large number of false positives.
Therefore, in the cell analysis apparatus of the present embodiment, in order to perform the process of excluding the falsely detected false nuclear region, a learning model for recognizing the cell region is created separately from the learning model for recognizing the nuclear region. To do.

Here, in order to obtain a correct image showing the cell region, the cytoskeleton existing in the fiber shape or the network shape throughout the inside of the cell is used. 11 (a) and 11 (b) are a fluorescence-stained image of actin filaments in the same observation region and a bright-field image obtained by a normal microscope. Actin filaments are a type of cytoskeleton and are present in the form of fibers throughout the inside of cells. As shown in FIG. 11 (b), it is difficult to visually recognize the cell region in the bright field image, but as shown in FIG. 11 (a), the actin filaments are present in almost the entire cell, so that the actin filaments are present. The distributed range can be considered to be the cellular region. Therefore, in the cell analyzer of the present embodiment, a cell region learning model for extracting a cell region is created by using a fluorescent image stained with a cytoskeleton (here, actin filament) as an accurate image.

The cell region learning model can be created by the same procedure as the nuclear region learning model, that is, the procedure shown in FIG.

The cytoskeleton is fibrously stretched inside the cell, but it does not necessarily exist evenly, and even within the cell, if the cytoskeleton does not exist or its density is low, it is binarized. Some pixels are partially blackened. On the other hand, by performing the closing process after binarization, the pixels whose surroundings are white are converted to white even if they are black. As a result, it is possible to obtain an image showing not only the part where the cytoskeleton actually exists but also the range where the cytoskeleton is distributed, that is, the entire cell region. That is, the image after the closing process is an image in which the cell region and the other regions are separated for each pixel on the corresponding IHM phase image.

When a large number of learning data including the IHM phase image and the cytoskeleton-stained fluorescent image after the closing process are prepared, the learning execution unit 55 executes FCN learning using the large number of learning data. The model building unit 56 builds a model in the process of repeating the learning, and when a predetermined learning is completed, the model building unit 56 saves the cell region learning model based on the learning result. The data constituting the cell region learning model created in this way is stored in the cell region learning model storage unit 261 in the cell analysis device 1.

Next, the process of creating a nuclear region estimation image for the cell to be analyzed and the cell counting process based on the process performed by the cell analysis device 1 will be described with reference to the flowchart shown in FIG.

The operator sets the culture plate 13 containing the cells 14 to be analyzed at a predetermined position of the microscopic observation unit 10, and performs a predetermined operation on the input unit 30. As a result, under the control of the imaging control unit 21, the microscopic observation unit 10 photographs the sample (cells 14 in the culture plate 13) (step S21). The phase information calculation unit 23 and the image creation unit 24 perform phase calculation based on the hologram data obtained by the photographing, and form an IHM phase image (step S22).

The nuclear region estimation unit 25 reads the IHM phase image obtained in step S22 as an input image, performs processing by FCN using the nuclear region learning model stored in the nuclear region learning model storage unit 251 and performs processing by FCN, and inputs the input image. The segmentation image corresponding to is acquired as an output image (step S23). The segmentation image at this time is a nuclear region estimation image showing the region of the cell nucleus and the region other than the cell nucleus separately in the same observation range as the IHM phase image which is the input image.

As is well known, in FCN processing using a learning model constructed by performing machine learning as described above, a numerical value indicating the probability of certainty of semantic segmentation can be obtained for each pixel. That is, here, the probability of the certainty of estimation that it is the region of the nucleus is obtained for each pixel. Therefore, the nuclear region estimation unit 25 outputs a grayscale image in which the gradation from white to black is changed according to the probability value of each pixel. That is, in the nuclear region estimation image, the portion estimated with high accuracy as the nuclear region is displayed with white or a gradation close to it, and the portion with low estimation accuracy as the nuclear region is relatively close to black. It is displayed in gradation.

As described above, the nuclear region estimation image shown in FIG. 7 can be obtained for the IHM phase image shown in FIG. As described above, when the interference fringes appear to some extent clearly on the input IHM phase image or when the image of a foreign substance appears, a false nuclear region appears at a site where the cell nucleus does not actually exist. ..

Next, the cell region estimation unit 26 reads the IHM phase image obtained in step S22 as an input image, and performs processing by FCN using the cell region learning model stored in the cell region learning model storage unit 261. , The segmentation image corresponding to the input image is acquired as an output image (step S24). The segmentation image at this time is a cell region estimation image showing the cell region and other regions separately for the same observation range as the IHM phase image which is the input image. FIG. 12 is an example of a cell region estimation image obtained by using the IHM phase image shown in FIG. 6 as an input image.

Similar to the nuclear region estimation image, at this time as well, a numerical value indicating the probability of the certainty of the segmentation obtained for each pixel by the FCN processing can be obtained. Therefore, the cell region estimation unit 26 compares the probability value of each pixel with a predetermined threshold value, and outputs a binary segmentation image in which the pixel having the probability value equal to or higher than the threshold value is white and the other pixels are black.

Next, the mask processing unit 27 performs mask processing on the nuclear region estimation image obtained in step S23 using the binarized cell region estimation image obtained in step S24 (step S25). That is, only the white part of the binary image shown in FIG. 12, that is, the nuclear region existing in the cell region is extracted, and the other nuclear regions are excluded. The nuclear region excluded at this time is a nuclear region erroneously detected due to the influence of interference fringes or an image of a foreign substance. FIG. 13 is a nuclear region estimation image after mask processing, and as can be seen by comparison with FIG. 7, the number of nuclear regions is significantly reduced.

The core region estimation image after this mask processing is a grayscale image, and the peak value of the signal differs depending on the core. Therefore, even if the determination process using a single threshold value is performed, not all nuclei can be detected. Further, since a plurality of adjacent nuclei may be seen as one region, it is preferable to perform a process of separating the plurality of adjacent nuclei. Therefore, the mask processing unit 27 performs the maximum value region extraction processing on the nuclear region estimation image after the mask processing.

Specifically, in the nuclear region estimation image which is a grayscale image, the region other than the black region (background region) is spatially expanded. Next, in the expanded image, the brightness is generally lowered by subtracting the offset value determined in advance in consideration of the noise tolerance value from the signal value (luminance value) of each pixel. Then, the luminance value is subtracted for each pixel between the image before the expansion process and the image after the luminance decrease. When such subtraction processing is performed, the luminance value becomes non-zero in a narrow range near the peak of the luminance value in the original image, regardless of the peak value, and the luminance value becomes zero in other regions. Become. That is, by this processing, it is possible to extract a region having a maximum value in which the brightness value is higher than that around the original image.

Although this algorithm for maximal value region extraction processing is common in image processing, it is not limited to this algorithm, and an appropriate known method can be used for maximum value region extraction processing. For example, after binarizing the nuclear region estimation image, the position of the center of gravity may be obtained for each cell nucleus to obtain the nuclear position candidate point.

The mask processing unit 27 creates an image obtained by extracting the maximum value region from the masked nuclear region estimation image, and then binarizes the image to obtain a nuclear position candidate point image showing a candidate point of the nuclear region. (Step S26). FIG. 14 is a diagram in which a nuclear position candidate point is superimposed on the IHM image shown in FIG. Comparing FIG. 14 and FIG. 10, it can be seen that the nuclear position candidate points substantially coincide with the correct nuclear position. That is, by performing mask processing using the cell region estimation image, it is possible to accurately eliminate the false nuclear region and extract the region that is truly the cell nucleus.

The cell counting unit 28 counts the nuclear position candidate points on the nuclear position candidate point image obtained in step S26 (step S27). Except in special cases, one cell has one cell nucleus. Therefore, it can be considered that the number of cell nuclei represents the number of cells, and the display processing unit 29 displays the counting result as the number of cells on the display unit 40 (step S28). Of course, one or a plurality of IHM phase images, kernel region estimation images, kernel position candidate point images, cell region estimation images, and the like may be displayed together.

As described above, the cell analyzer of the present embodiment can accurately calculate the number of cells existing in the observation range and provide it to the user.
Further, in the cell analysis apparatus of the present embodiment, if an image in which the nuclear position candidate points are superimposed on the IHM phase image as shown in FIG. 14 is displayed on the display unit 40, the user can easily position the cell nucleus in the cell. Can be grasped. Further, if the nuclear region estimation image as shown in FIG. 13 is displayed on the display unit 40, the user can easily grasp the shape and morphology of the cell nucleus.
Since the counting of living cells and the observation of cell nuclei as described above are performed non-invasively, the cells used for the analysis and observation can be continuously cultured or used for another purpose.

In the cell analyzer of the above embodiment, the grayscale nuclear region estimation image is masked, and after the masking, the maximum value region extraction process and the binarization process are performed to obtain the nuclear position candidate points for each nuclear region. However, after obtaining the nuclear position candidate point image from the grayscale nuclear region estimation image, mask processing using the cell region estimation image may be performed. In addition, as a masking process, after binarizing a grayscale nuclear region estimation image, for each nuclear region, when one nuclear region and a part or all of the cell region on the cell region estimation image do not overlap. , The nuclear region may be regarded as a false positive and excluded. As described above, various methods can be adopted as the mask processing method.

Further, in the above embodiment, FCN is used as a machine learning method for semantic segmentation of cell nuclei, but it is clear that a normal convolutional neural network (CNN) may be used. Further, it is effective to apply the present invention not only to the machine learning method using a neural network but also to any machine learning method capable of semantic segmentation of an image. Such machine learning methods include, for example, Support Vector Machine (SVM), Random Forest, and AdaBoost. As described above, FCN can output the estimated probability of segmentation when outputting a segmentation image for the input image, but other methods such as CNN and SVM also use patch images (the entire image is finely divided). A small area image) can be scanned and the probability of cell nucleus-likeness at the center point of each patch image can be output.

Further, in the cell analysis apparatus of the above embodiment, an in-line holographic microscope is used as the microscopic observation unit 10, but if it is a microscope that can obtain a hologram, an off-axis type, a phase shift type, or the like can be used. It is natural that it can be replaced with a holographic microscope of the above method.

Further, the above-described embodiment and various modifications are merely examples of the present invention, and it is clear that even if modifications, modifications, and additions are made as appropriate within the scope of the present invention, they are included in the claims of the present application. is there.

[Various aspects]
It will be apparent to those skilled in the art that the exemplary embodiments described above are specific examples of the following embodiments.

(Item 1) One aspect of the cell image analysis method according to the present invention is
The phase image of the cell created based on the hologram data acquired by the holographic microscope is used as the input image, and the corresponding pseudo nuclear region image based on the stained image obtained by staining the cell nucleus is the correct image. The first learning model creation step of creating a nuclear region learning model by performing machine learning using the training data of
By performing machine learning using the learning data in which the phase image of the cells is used as the input image and the corresponding pseudo cell region image based on the stained image obtained by staining the cytoskeleton is used as the correct image. The second learning model creation step to create the cell region learning model,
Using the nuclear region learning model, a nuclear region estimation step of acquiring a phase image of the cell to be analyzed as an input image and a nuclear region estimation image showing the region of the cell nucleus as an output image,
Using the cell region learning model, a cell region estimation step of acquiring a phase image of the cell to be analyzed as an input image and a cell region estimation image showing the cell region as an output image, and
Using the nuclear region estimation image and the cell region estimation image, a nuclear region extraction step of extracting cell nuclei existing in a range estimated to be a cell region, and a nuclear region extraction step.
It has.

(Section 7) Further, one aspect of the cell analyzer according to the present invention is
With a holographic microscope
An image creation unit that creates a phase image of the cells based on the hologram data obtained by observing the cells with the holographic microscope.
Using the learning data in which the phase image of the cell created based on the hologram data is used as the input image and the corresponding pseudo-nucleus region image based on the stained image obtained by staining the cell nucleus is used as the correct image. The first learning model storage unit that stores the nuclear region learning model created by performing machine learning,
Created by performing machine learning using learning data in which the phase image of the cells is used as an input image and the pseudo cell region image based on the stained image obtained by staining the corresponding cell skeleton is used as the correct image. A second learning model storage unit that stores the stored cell region learning model,
Using the nuclear region learning model stored in the first learning model storage unit, the phase image created by the image creation unit for the cell to be analyzed is used as an input image, and a nuclear region estimation image showing the region of the cell nucleus is output. The nuclear region estimation unit acquired as an image and
Using the cell region learning model stored in the second learning model storage unit, the phase image of the cell to be analyzed is used as an input image, and the cell region estimation image showing the cell region is acquired as an output image. Department and
Using the nuclear region estimation image and the cell region estimation image, a nuclear region extraction unit that extracts cell nuclei existing in a range estimated to be a cell region, and a nuclear region extraction unit.
Is provided.

According to the cell image analysis method according to the first item and the cell analysis device according to the seventh item, the cells to be analyzed are created based on hologram data without performing invasive processing such as staining. The cell nucleus can be accurately extracted from the phase image of the cell. In particular, even when interference fringes appear in the phase image or an image due to a foreign substance appears, the influence thereof can be eliminated or reduced, and the cell nucleus can be observed accurately. Then, for example, by counting the number of the cell nuclei, information on the number of cells can be obtained. In addition, the true shape and morphology of the cell nucleus, or the state inside the cell nucleus can be observed. In addition, since the cells to be analyzed can be photographed (measured) non-invasively, the cells after imaging can be continuously cultured or used for analysis or observation for another purpose.

(2) The cell image analysis method according to paragraph 1 may further include a cell counting step for counting the number of cell nuclei extracted in the nuclear region extraction step.

(Item 8) Further, the cell analyzer according to item 7 can further include a cell counting unit that counts the number of cell nuclei extracted by the nuclear region extraction unit.

Except for special cases, one cell has one cell nucleus, so the number of cell nuclei can be considered to represent the number of cells. According to the cell image analysis method according to the second item and the cell analysis device according to the eighth item, the number of cell nuclei extracted accurately can be counted, so that the number of cells existing in the observation range can be determined. It can be calculated accurately.

(Clause 3) The cell image analysis method according to the first or second paragraph further includes a display step of displaying a nuclear region estimation image in which the cell nuclei extracted in the nuclear region extraction step are displayed. can do.

(Section 9) Further, the cell analysis apparatus according to the seventh or eighth paragraph further includes a display processing unit that displays a nuclear region estimation image in which the cell nuclei extracted by the nuclear region extraction unit are displayed. Can be.

According to the cell image analysis method according to the third item and the cell analysis device according to the ninth item, the user can observe the shape and morphology of the accurately extracted cell nucleus based on the displayed image. it can.

(Clause 4) In the cell image analysis method according to any one of paragraphs 1 to 3, the cell region estimation image is a binary image in which a cell region and other regions are separated. Can be.

(Item 10) In the cell analyzer according to any one of paragraphs 7 to 9, the cell region estimation image is a binary image in which a cell region and other regions are separated. can do.

According to the cell image analysis method described in the fourth item and the cell analysis device described in the tenth item, the cell region and the other regions are clarified, so that the cell nucleus is likely to be used. It can be extracted by accurate and simple processing.

(Section 5) In the cell image analysis method according to any one of paragraphs 1 to 4, the machine learning can be performed using a convolutional neural network.

(Section 6) In the cell image analysis method according to paragraph 5, the convolutional neural network can be a fully convolutional neural network.

(Item 11) In the cell analysis apparatus according to any one of items 7 to 10, the machine learning can be performed using a convolutional neural network.

(Item 12) Further, in the cell analysis apparatus according to the item 11, the convolutional neural network can be a fully convolutional neural network.

According to the cell image analysis method according to the fifth and sixth paragraphs and the cell analysis apparatus according to the tenth and eleventh paragraphs, there is a high possibility that the region or cell is likely to be a cell nucleus. Each region can be estimated accurately.

(Section 13) In the cell analyzer according to any one of paragraphs 7 to 12,
A learning model creation unit that creates a learning model by performing machine learning using learning data that uses a cell phase image as an input image and a pseudo region image based on the corresponding stained image as a correct image. Prepare,
The nuclear region learning model and the cell region learning model can be created by using the learning model creation unit.

Although it is not essential that the cell analysis apparatus according to the seventh item has a function of creating a learning model used for obtaining a cell region estimation image from a phase image of a cell, the cell analysis according to the thirteenth item. The device has that function. Therefore, according to the cell analyzer according to the thirteenth paragraph, for example, the nuclear region estimation image obtained for the phase image of the cell to be analyzed in the nuclear region estimation unit is added to the training data to re-learn the learning model. It is possible to easily improve the learning model such as building.

1 ... Cell analysis device 10 ... Microscopic observation unit 11 ... Light source unit 12 ... Image sensor 13 ... Culture plate 14 ... Cell 15 ... Reference light 16 ... Object light 20 ... Control / processing unit 21 ... Imaging control unit 22 ... Hologram data storage unit 23 ... Phase information calculation unit 24 ... Image creation unit 25 ... Nuclear region estimation unit 251 ... Nuclear region learning model storage unit 26 ... Cell region estimation unit 261 ... Cell region learning model storage unit 27 ... Mask processing unit 28 ... Cell counting unit 29 ... Display processing unit 30 ... Input unit 40 ... Display unit 50 ... FCN model creation unit 51 ... Learning image data input unit 52 ... Image alignment processing unit 53 ... Stained image preprocessing unit 54 ... Stained image binarization unit 55 ... Learning Execution unit 56 ... Model construction unit

Claims

The phase image of the cell created based on the hologram data acquired by the holographic microscope is used as the input image, and the corresponding pseudo nuclear region image based on the stained image obtained by staining the cell nucleus is the correct image. The first learning model creation step of creating a nuclear region learning model by performing machine learning using the training data of
By performing machine learning using the learning data in which the phase image of the cells is used as the input image and the corresponding pseudo cell region image based on the stained image obtained by staining the cytoskeleton is used as the correct image. The second learning model creation step to create the cell region learning model,
Using the nuclear region learning model, a nuclear region estimation step of acquiring a phase image of the cell to be analyzed as an input image and a nuclear region estimation image showing the region of the cell nucleus as an output image,
Using the cell region learning model, a cell region estimation step of acquiring a phase image of the cell to be analyzed as an input image and a cell region estimation image showing the cell region as an output image, and
Using the nuclear region estimation image and the cell region estimation image, a nuclear region extraction step of extracting cell nuclei existing in a range estimated to be a cell region, and a nuclear region extraction step.
A cell image analysis method.
The cell image analysis method according to claim 1, further comprising a cell counting step of counting the number of cell nuclei extracted in the nuclear region extraction step.
The cell image analysis method according to claim 1, further comprising a display step of displaying a nuclear region estimation image in which the cell nuclei extracted in the nuclear region extraction step are displayed.
The cell image analysis method according to claim 1, wherein the cell region estimation image is a binary image in which a cell region and other regions are separated.
The cell image analysis method according to claim 1, wherein the machine learning uses a convolutional neural network.
The cell image analysis method according to claim 5, wherein the convolutional neural network is a fully convolutional neural network.
With a holographic microscope
An image creation unit that creates a phase image of the cells based on the hologram data obtained by observing the cells with the holographic microscope.
Using the learning data in which the phase image of the cell created based on the hologram data is used as the input image and the corresponding pseudo-nucleus region image based on the stained image obtained by staining the cell nucleus is used as the correct image. The first learning model storage unit that stores the nuclear region learning model created by performing machine learning,
Created by performing machine learning using learning data in which the phase image of the cells is used as an input image and the pseudo cell region image based on the stained image obtained by staining the corresponding cell skeleton is used as the correct image. A second learning model storage unit that stores the stored cell region learning model,
Using the nuclear region learning model stored in the first learning model storage unit, the phase image created by the image creation unit for the cell to be analyzed is used as an input image, and a nuclear region estimation image showing the region of the cell nucleus is output. The nuclear region estimation unit acquired as an image and
Using the cell region learning model stored in the second learning model storage unit, the phase image of the cell to be analyzed is used as an input image, and the cell region estimation image showing the cell region is acquired as an output image. Department and
Using the nuclear region estimation image and the cell region estimation image, a nuclear region extraction unit that extracts cell nuclei existing in a range estimated to be a cell region, and a nuclear region extraction unit.
A cell analyzer equipped with.
The cell analysis apparatus according to claim 7, further comprising a cell counting unit that counts the number of cell nuclei extracted by the nuclear region extraction unit.
The cell analysis apparatus according to claim 7, further comprising a display processing unit that displays a nuclear region estimation image in which the cell nuclei extracted by the nuclear region extraction unit are displayed.
The cell analysis apparatus according to claim 7, wherein the cell region estimation image is a binary image in which a cell region and other regions are separated.
The cell analysis device according to claim 7, wherein the machine learning uses a convolutional neural network.
The cell analysis apparatus according to claim 11, wherein the convolutional neural network is a fully convolutional neural network.
A learning model creation unit that creates a learning model by performing machine learning using learning data that uses a cell phase image as an input image and a pseudo region image based on the corresponding stained image as a correct image. The cell analysis apparatus according to claim 7, further comprising, using the learning model creating unit to create the nuclear region learning model and the cell region learning model.