JP7135303B2 - Computer program, detection device, imaging device, server, detector, detection method and provision method - Google Patents

Description

TECHNICAL FIELD The present disclosure relates to computer programs, detection devices, imaging devices, servers , detectors , detection methods, and providing methods.

There is a strong demand for an image analysis technique for automating the observation of minute specimens such as cells and bacteria for the industrialization and efficiency of research in the field of regenerative medicine. Among the image analysis technologies, the technology to detect cells is the most basic one. , various technological developments are being carried out.

In Patent Document 1, edges are extracted by applying a secondary differential filter to a phase contrast image of cells acquired using a phase contrast microscope, and the circle formed by the extracted edges with the largest area is floated. Techniques for extracting cell shapes are disclosed.

In addition, Patent Document 2 discloses a technique for measuring the number of specific bacteria by irradiating excitation light on microorganisms that have been fluorescently stained using a compound that specifically reacts with specific bacteria and analyzing the resulting fluorescence. disclosed.

JP 2008-212017 A Japanese Patent Application Laid-Open No. 2001-340072

However, in phase-contrast images such as those disclosed in Patent Document 1, since cells are often colorless and transparent in the first place, the brightness difference between the cells and the background is small. is very difficult in practice. Therefore, although the presence of cells can be recognized in the phase-contrast image, it is difficult to recognize individual cells.

In addition, as in Patent Document 2, when cells are fluorescently stained and details are observed, the cells are altered or killed by the staining, making it impossible to recognize the original behavior of the cells. In addition, work and costs are incurred for fluorescent dyeing.

The present disclosure has been made in view of such circumstances, and includes a computer program, a detection device, an imaging device, a server , a detector , and a detection method capable of recognizing minute specimens such as individual cells and bacteria. and how to provide it.

The present application includes a plurality of means for solving the above problems. To give an example, the computer program is a computer program for causing a computer to detect an analyte, and the computer is provided with the method obtained by the first method. Further, a process of acquiring first image data related to a first image including a learning specimen, and a second method different from the first method, which enables identification of a region of interest in the learning specimen. A process of acquiring second image data related to a second image including the region of interest of the learning specimen, a process of generating learning data based on the first image data and the second image data, and the first method and a process of learning the detector based on the generated learning data so that the detector can detect the specimen based on the obtained first image data related to the first image including the specimen.

According to the present disclosure, minute specimens such as individual cells and bacteria can be recognized.

1 is a block diagram showing an example of the configuration of a detection device according to an embodiment; FIG. FIG. 4 is an explanatory diagram showing an example of a phase contrast image and a stained nuclear image; FIG. 4 is an explanatory diagram schematically showing a first example of the configuration of a detection model unit according to the embodiment; FIG. 4 is an explanatory diagram showing a first example of a method of generating learning data for learning a detection model unit according to the embodiment; FIG. 4 is an explanatory diagram showing a first example of learning by a learning processing unit according to the embodiment; 4 is a flow chart showing a first example of a procedure of learning processing of a processing unit according to the present embodiment; FIG. 4 is an explanatory diagram showing a first example of a method of generating input data by an input data generation unit according to the embodiment; FIG. 4 is an explanatory diagram showing a first example of detection processing by a detection model unit according to the embodiment; 4 is a flow chart showing a first example of a procedure of detection processing by a processing unit according to the present embodiment; FIG. 4 is an explanatory diagram showing an example of an estimated stained nuclear image generated by a processing unit according to the present embodiment; FIG. 10 is an explanatory diagram showing an example of a combined image obtained by combining a phase contrast image and an estimated nuclear staining image; FIG. 10 is an explanatory diagram showing another example of an estimated stained nuclear image and a combined image; FIG. 10 is an explanatory diagram schematically showing a second example of the configuration of the detection model unit according to the embodiment; FIG. 10 is an explanatory diagram showing a second example of a method of generating learning data for learning the detection model unit according to the embodiment; FIG. 9 is an explanatory diagram showing a second example of learning by the learning processing section of the embodiment; 9 is a flow chart showing a second example of the procedure of learning processing of the processing unit according to the embodiment; FIG. 9 is an explanatory diagram showing a second example of detection processing by the detection model unit of the embodiment; 9 is a flow chart showing a second example of a procedure of detection processing by the processing unit of the present embodiment; FIG. 4 is an explanatory diagram showing an example of a nuclear staining image after binarization obtained by binarizing a nuclear staining image with different threshold values; 7 is a flow chart showing an example of a pre-processing procedure for binarization processing by a processing unit according to the present embodiment; FIG. 10 is an explanatory diagram showing an example of an image in pre-processing of binarization processing by the processing unit of the present embodiment; FIG. 4 is a block diagram showing another example of the configuration of the detection device according to this embodiment; 1 is a block diagram showing an example of a configuration of a microscope according to an embodiment; FIG. 1 is a schematic diagram showing an example of a configuration of a detection system according to an embodiment; FIG. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a schematic diagram showing an example of a method for providing mouse epidermal cells according to the present embodiment. 1 is a flow chart showing an example of a procedure of a method for providing mouse epidermal cells according to the present embodiment.

Embodiments of the present disclosure will be described below with reference to the drawings. FIG. 1 is a block diagram showing an example of the configuration of a detection device 50 of this embodiment. The detection device of this embodiment is a device that detects a specimen. Specimens include all objects that can be identified by labeling with the second method described below. Specimens include, for example, cells in various parts of a living body (animal) (for example, epidermal cells of mice), microscopic objects to be identified such as bacteria. A second method is a method that enables identification of the region of interest of the specimen. By staining the cell nucleus or cell membrane by the second method, the cell nucleus can be identified, and as a result, the cell can be detected as a specimen. In this case, the region of interest is the cell nucleus. The second method also includes a method of intentionally staining the cytoplasm to make the cell nucleus stand out in order to emphasize the cell nucleus so that it can be distinguished. Also, if only the nuclei of living cells are stained by the second method, the living cells can be identified, and as a result, the living cells can be detected as specimens. In this case, the region of interest is living cells. Moreover, the specimen is not limited to a tissue of a living organism such as a cell or fungus, but can also include a powder that is an aggregate of solid particles containing specific molecules or proteins. The specimen includes cases in which live cells and dead cells are not distinguished, cases in which either one or both are distinguished. Almost all cells have a cell nucleus (simply called "nucleus"). The nucleus is one of the organelles that make up eukaryotic cells, and genetic information is stored in the nucleus. there is The cytoplasm is the part of the cytoplasm, which is the part of the cell surrounded by the cell membrane, other than the cell nucleus. The detection apparatus of the present embodiment detects each cell recognizably in a state in which a large number of cells are densely packed. The position, number of cells, etc. can be detected accurately. The specimen can include bacteria, solid particles, gel particles, and the like. Hereinafter, the specimen will be described as a cell in this specification.

The detection device 50 includes a processing unit 51, a first image data acquisition unit 52, a second image data acquisition unit 53, a memory 54, an output unit 55, a display unit 56, and the like. The processing unit 51 includes a detection model unit 511, an input data generation unit 512, an estimated image generation unit 513, a learning processing unit 514, a learning data generation unit 515, and the like. The detection device 50 can also be connected to a display device 60 capable of displaying images. Note that the detection device 50 can be incorporated in, for example, an information processing device such as a personal computer used by a user, a server that provides various services and data, or a microscope that observes a specimen.

The first image data acquisition unit 52 has functions as a first acquisition unit and an acquisition unit, and acquires cells obtained by the first method (both the learning cells of the learning processing unit 514 and the detection cells). obtain first image data relating to a first image including The first method is a method in which it is difficult to clearly and separately recognize regions of interest (for example, nuclei) of cells unless individual processing is applied to the image. The first image obtained by the first method includes, for example, at least one of a phase-contrast image and a bright-field image obtained with a microscope.

A phase-contrast image is an image obtained by a phase-contrast microscope, and is an image in which colorless and transparent cells are visualized by light-dark contrast by utilizing two properties of light diffraction and interference. When light enters a cell, it undergoes a phase change (diffracted light), while light that has not been affected by the cell does not (direct light). Information can be obtained as contrast between light and dark.

A bright-field image is an image obtained by a transmission electron microscope, and is an image formed only by electron beams (transmission waves) that have passed through the electron beams incident on a sample containing many cells. Phase-contrast images and bright-field images do not require cells to be stained, so they can detect living cells such as cell differentiation. In this specification, the first image obtained by the first method will be described as a phase contrast image.

The second image data acquisition unit 53 has a function as a second acquisition unit, and acquires second image data related to a second image including learning cells, which is obtained by a second method different from the first method. . The second method is different from the first method in that it is possible to separately and clearly recognize regions of interest (eg, nuclei) of cells. In other words, according to the first method, it is difficult to individually identify each region of interest of the cell unless the image is individually processed. It can be said that each one can be identified individually. In this specification, the second image obtained by the second method will be described as a stained image.

A nuclear stain image is an image obtained by a fluorescence microscope. Cells are pre-stained with nuclear staining (also known as fluorescent staining), the nuclear-stained cells are irradiated with excitation light, and among the fluorescence emitted from the nuclear-stained cells, a filter that transmits only the fluorescence of the required wavelength is applied. A nuclear staining image can be obtained by extracting only the fluorescence.

The memory 54 is an image memory, and stores the image data (first image data) of the phase contrast image acquired by the first image data acquisition unit 52 and the image data (first image data) of the nuclear staining image acquired by the second image data acquisition unit 53 . 2 image data) are stored.

FIG. 2 is an explanatory diagram showing an example of a phase contrast image and a stained nuclear image. FIG. 2A is a phase contrast image, and FIG. 2B is a nuclear staining image. The phase-contrast image and nuclear staining image shown in FIG. 2 were obtained by imaging the same observation region at the same time. The resolution of the phase contrast image and the nuclear staining image can be 800×600, but is not limited to this. In a phase-contrast image, the presence of cells or the outline of the shape of cells can be recognized. It is difficult to identifiably observe individual cells for all cells in the area. In the nuclear-stained image, the cell nuclei appear as brighter than the background, and the positions and numbers of the cell nuclei can be observed more accurately than in the phase-contrast image. Note that the brightness emitted by each cell nucleus is not uniform and varies depending on the cell nucleus.

The processing unit 51 can be configured by hardware or software, or a combination of both.

The detection model unit 511 has a function as a detector, and is configured by, for example, a plurality of processors (CPU: Central Processing Unit), a plurality of graphics processing units (GPU: Graphics Processing Unit), or a combination of CPU and GPU. be able to. A detection model for detecting cells is created using a required programming language, and the training data is used to tune the configuration such as the parameters of the detection model so that the detection accuracy of the created detection model is high. Learning of the model unit 511 can be performed.

FIG. 3 is an explanatory diagram schematically showing a first example of the configuration of the detection model unit 511 of this embodiment. FIG. 3 shows the detection model unit 511 configured with a neural network model, but the configuration of the detection model unit 511 is not limited to a multi-layered neural network (deep learning) as in the example of FIG. , other machine learning algorithms (eg, SVM: Support Vector Machine, Random Forest: Random Forest) can also be used.

As shown in FIG. 3, the detection model unit 511 is composed of an input layer, an output layer and a plurality of intermediate layers. Although two intermediate layers are shown in FIG. 3 for convenience, the number of intermediate layers is not limited to two, and may be three or more. The number of nodes in the input layer is N (x1, x2, . . . , xN), and the number of nodes in the output layer is 1 (y1).

One or more nodes (neurons) exist in the input layer, the output layer, and the intermediate layer, and the nodes in each layer are unidirectionally connected with the nodes in the preceding and succeeding layers with desired weights. A vector having the same number of components as the number N of nodes in the input layer is given as input data to the learned detection model unit 511 .

When the data given to each node of the input layer is given as input to the first hidden layer, the output of the hidden layer is calculated using the weight and activation function, and the calculated value is given to the next hidden layer. and is transmitted to subsequent layers (lower layers) in the same manner until the output of the output layer is obtained. Note that all weights for connecting nodes are calculated by the learning algorithm of the learning processing unit 514 .

The output layer outputs the luminance value of one pixel based on the input data (x1, x2, . . . xN). Assuming that the luminance range of pixels is 0-255, the output value can take any value from 0-255 (not limited to integers). A luminance value of 0 is black (dark), and a luminance value of 255 is white (bright).

Next, a learning procedure of the detection model unit 511 will be described.

The learning data generation unit 515 obtains the image data of the phase contrast image containing the learning cell obtained by imaging, and the image data of the nuclear staining image containing the region of interest of the learning cell obtained by imaging. Generate training data based on

FIG. 4 is an explanatory diagram showing a first example of a method of generating learning data for learning the detection model unit 511 of this embodiment. The learning data generation method of FIG. 4 corresponds to the first example of the detection model unit 511 shown in FIG.

As shown in FIG. 4A , the learning data generator 515 acquires from the memory 54 image data of a stained nuclear image (for example, the resolution is 800×600) including the region of interest of the learning cell captured in the observation region.

As shown in FIG. 4B, the learning data generation unit 515 identifies a plurality of pixel blocks (for example, g×g, g=28, etc.) composed of a predetermined number of pixels from the stained nuclear image. In the example of FIG. 4B, four pixel blocks G1, G2, G3, G4 are shown for convenience. In practice, a required number of pixel blocks can be identified from a single stained nuclear image. The required number may be, for example, a number equal to the total number of pixels in one stained nuclear image, or may be a number smaller than the total number of pixels in order to reduce the image processing load. The pixel blocks can be specified by randomly selecting positions on the stained nuclear image, and not only pixel blocks containing cell nuclei but also pixel blocks not containing cell nuclei can be specified. By also specifying pixel blocks that do not contain cell nuclei, it is possible to prevent erroneous detection, such as the presence of cell nuclei in spite of the fact that cell nuclei do not exist, and improve the cell detection accuracy. Moreover, it is not necessary to distinguish whether the cells are positive cells or dead cells. Pixel blocks G1 and G3 contain cell nuclei (color development sites), and pixel blocks G2 and G4 do not contain cell nuclei. Also, let the feature values of the pixel blocks G1, G2, G3, and G4 (in the example of FIG. 4B, the luminance value, which is one of the pixel values) be 159, 82, 255, and 90, respectively. Here, the feature amount of the pixels in the pixel block can be, for example, the luminance value, which is one of the pixel values of the pixel existing in the center of the pixel block, but is not limited to this. In addition, a representative value representing the pixel block, such as an average value of pixel values in the pixel block, a median value of the pixel values, etc., can be included. That is, the representative value is also one of the pixel values.

The learning data generator 515 acquires from the memory 54 image data of a phase-contrast image (with a resolution of, for example, 800×600) including learning cells captured in the observation region. Note that the phase-contrast image and the stained nuclear image are images of the same observation region containing the learning cells at the same time.

The learning data generation unit 515 generates learning data by associating the pixel blocks of the phase contrast image corresponding to the pixel blocks specified on the stained nuclear image with the pixel feature amounts of the pixel blocks specified on the stained nuclear image. More specifically, the learning data generation unit 515 assigns one pixel value of a predetermined pixel (central pixel in the example of FIG. 4) in the pixel block specified on the stained nuclear image to the pixel block of the phase contrast image. is associated with the luminance value to generate learning data. Since pixel values are used as learning data (teaching data), for example, compared to the case where feature vectors are calculated and used as teaching data, there is no need to design appropriate feature vectors. In contrast, the pixel values can be used as they are, which simplifies the processing and enables highly accurate cell detection.

As shown in FIG. 4C, the learning data generator 515 cuts out regions X1, X2, X3, and X4 (cropped images) of the phase contrast image corresponding to the pixel blocks G1, G2, G3, and G4 on the stained nuclear image. The regions X1 to X4 are associated with the luminance values of the pixel blocks G1 to G4 as teacher labels. Since each of the regions X1 to X4 is composed of g×g pixels, for the region X1, a vector (x11, x12, , x1N) is generated, and the teacher label is 159.

For the region X2, a vector (x21, x22, . For the region X3, a vector (x31, x32, . For the region X4, a vector (x41, x42, .

FIG. 5 is an explanatory diagram showing a first example of learning by the learning processing unit 514 of this embodiment. The learning processing unit 514 causes the detection model unit 511 to learn based on the learning data generated by the learning data generation unit 515 . Specifically, the learning processing unit 514 inputs learning input data (x11, x12, . Learning is performed so as to approach 159 which is teacher data.

Further, the learning processing unit 514 inputs learning input data (x21, x22, . Learning is performed so as to approach a certain 82. Further, the learning processing unit 514 inputs learning input data (x31, x32, . Learning is performed so as to approach a certain 255. Also, the learning processing unit 514 inputs learning input data (x41, x42, . Learning is performed so as to approach a certain 90.

FIG. 6 is a flow chart showing a first example of the procedure of the learning process of the processing section 51 of this embodiment. In the following description, for the sake of convenience, the processing unit 51 is assumed to be the subject of processing. The processing unit 51 acquires image data of a phase-contrast image including a learning cell in which an observation region is imaged (S11), and acquires nuclei including a region of interest of the learning cell captured in the same observation region at the same time. Image data of the stained image is obtained (S12).

The processing unit 51 randomly identifies a plurality of pixel blocks from the stained nuclear image (S13), and records the luminance value of the center pixel of the identified pixel blocks (S14). The processing unit 51 extracts a clipped image at a position corresponding to the pixel block from the phase difference image (S15).

The processing unit 51 uses the luminance value of each pixel of the extracted clipped image as input data for learning, and generates learning data using the luminance value of the center pixel of the pixel block corresponding to the clipped image as a teacher label (S16). The processing unit 51 causes the detection model unit 511 to learn based on the generated learning data (S17).

The processing unit 51 determines whether the accuracy of the output value of the detection model unit 511 is within the allowable range (S18), and if it is not within the allowable range (NO in S18), continues the processing from step S11. If the accuracy of the output value is within the allowable range (YES in S18), the processing unit 51 terminates the process.

When generating learning data, the number of phase-contrast images and nuclear-stained images is not limited to one, and a plurality of phase-contrast images and nuclear-stained images can be used. For example, a phase-contrast image and a nuclear-stained image can be captured each time a predetermined time has elapsed in order to learn the state according to cell activity. For example, one phase-contrast image and nuclear-stained image may be taken every 15 minutes, and finally about 10 phase-contrast images and nuclear-stained images may be acquired.

Next, a procedure for detecting cells using the learned detection model unit 511 will be described.

The first image data acquisition unit 52 acquires a phase contrast image including cells for detection (not for learning), and stores the acquired phase contrast image in the memory 54 .

The input data generator 512 acquires the phase contrast image from the memory 54 and identifies a plurality of pixel blocks each having a predetermined number of pixels from the acquired phase contrast image. The input data generator 512 generates input data based on the luminance value of each pixel in the specified pixel block.

FIG. 7 is an explanatory diagram showing a first example of a method of generating input data by the input data generation unit 512 of this embodiment. Assume that the resolution of the phase difference image is m×n (eg, m=800, n=600, etc.). For example, the input data generation unit 512 scans the phase difference image in the horizontal direction from the upper left, and when the horizontal scanning is completed, the scanning position is shifted in the vertical direction and the scanning is repeated from left to right again. By scanning to the lower right and specifying pixel blocks (g×g, g=28) corresponding to the number of pixels of all pixels of the phase difference image, M (M=m×n) cutout images A1 and A2 are obtained. , . . . , AM is cut out. Each clipped image is composed of N (N=g×g) pixels. Thus, the input data consists of M vectors each having N components. Note that the number of cutout images is not limited to the total number of pixels in the phase contrast image, and may be the number of pixels when a specific region of the phase contrast image is scanned. You may thin out.

FIG. 8 is an explanatory diagram showing a first example of detection processing by the detection model unit 511 of this embodiment. When the input data (a11, a12, a13, . . . , a1N) for the clipped image A1 is input to each input node of the input layer, the output value b1 is output to the output node of the output layer. Also, when the input data (a21, a22, a23, . . . a2N) for the clipped image A2 is input, the output value b2 is output to the output node of the output layer. Similarly, when the input data (aM1, aM2, aM3, . . . aMN) for the clipped image AM are input, the output value bM is output to the output node of the output layer.

The estimated image generating unit 513 generates an estimated image of the stained nuclear image based on the output values b1, b2, . As shown in FIG. 7, when the resolution of the phase difference image is m×n, the estimated image generation unit 513 associates M (=m×n) output values with m×n pixels. , m×n putative nuclear staining images.

The output unit 55 can output the output values b1, b2, . . . , bM output from the output nodes of the detection model unit 511 to an external device.

FIG. 9 is a flow chart showing a first example of the procedure of detection processing by the processing unit 51 of this embodiment. The processing unit 51 acquires image data of a phase contrast image including cells to be detected (S31), scans the acquired phase contrast image, and extracts a clipped image (S32). The processing unit 51 generates input data based on the luminance value of each pixel of the extracted clipped image (S33).

The processing unit 51 inputs the generated input data to the input node of the detection model unit 511 (S34), and uses the output value output by the detection model unit 511 for each clipped image as the luminance value of each pixel of the estimated nuclear staining image ( S35), and generate an estimated nuclear staining image (S36).

The processing unit 51 displays the phase contrast image and the estimated nuclear staining image in comparison (S37), and ends the process.

FIG. 10 is an explanatory diagram showing an example of an estimated stained nuclear image generated by the processing unit 51 of this embodiment. 10A is a phase contrast image including cells to be detected, FIG. 10B is an estimated image (estimated nuclear staining image) of the nuclear staining image generated by the processing unit 51, and FIG. 10C is a phase contrast image. It is a corresponding nuclear stained image, which is obtained by imaging the same observation region as the phase-contrast image at the same time. As shown in FIG. 10, according to the present embodiment, an estimated nuclear-stained image can be accurately obtained only by acquiring a phase-contrast image without performing nuclear staining on cells to acquire a nuclear-stained image. be able to.

The image data related to the estimated stained nuclear image shown in FIG. 10 is image data in which a cell (cell nucleus) is detected, and is estimated image data related to the estimated image of the stained image including the region of interest of the cell obtained by imaging. and the estimated image is input data based on first image data related to at least one of a phase-contrast image or a bright-field image containing cells obtained by imaging a detection model that detects cells It is generated based on the output value input to the unit 511 and output by the detection model unit 511, and the detection model unit 511 is obtained by imaging. Learning is performed using the learning data generated based on the first image data and the second image data related to the stained image including the region of interest of the learning cell obtained by imaging.

FIG. 11 is an explanatory diagram showing an example of a synthesized image obtained by synthesizing a phase contrast image and an estimated nuclear staining image. The display unit 56 can display the estimated nuclear staining image generated by the estimated image generating unit 513 and the acquired phase contrast image on the display screen of the display device 60 in comparison. Comparing the phase contrast image (FIG. 11A) and the estimated nuclear staining image (FIG. 11B) means not only displaying the phase contrast image and the estimated nuclear staining image side by side, but also displaying the phase contrast image and the estimated nuclear staining image (FIG. 11C). superimposing the phase contrast image and the estimated nuclear staining image such that the upper left position and the lower right position of the phase contrast image match the upper left position and the lower right position of the estimated nuclear staining image, respectively; and editing the phase-contrast image by a predetermined operation based on the information of the estimated nuclear staining image. As a result, the position of the cell recognized on the phase contrast image is determined by the position of the cell nucleus recognized on the estimated nuclear staining image. And it becomes possible to observe the phase-contrast image while accurately recognizing the number of cells. In particular, when only a phase-contrast image is used to recognize a single cell, it is possible to grasp and observe that, for example, two cells are in close contact with each other.

According to the present embodiment, it is possible to obtain an estimated nuclear staining image in which cell nuclei are clearly captured without actually obtaining a nuclear staining image, so that an image suitable for image analysis can be obtained. In addition, the operation and cost for nuclear stain imaging are not required, and the original behavior of cells can be observed without invading the cells.

According to the present embodiment, since teacher data (learning data) is automatically generated based on nuclear staining images, for example, a mask image (teacher data) of a region (cell region) to be detected is manually created. There is no room for the subjectivity of the creator of the teacher data, and it is possible to generate learning data free from variations or subjective blurring.

According to the present embodiment, once learning is completed, an image equivalent to a nuclear staining image can be obtained without performing nuclear staining of cells. disappears.

According to the present embodiment, cells are detected by inputting the image data of the phase contrast image into the detection model unit trained by the learning data based on the nuclear staining image. It eliminates the need to design diverse and appropriate filter models.

According to this embodiment, it is not necessary to estimate whether each pixel is a cell or not based on the optical system model, so there is no need to determine many unknown parameters of the optical system model by trial and error.

According to the present embodiment, when identifying a plurality of pixel blocks from a stained nuclear image, not only pixel blocks containing cell nuclei but also pixel blocks not containing cell nuclei are identified to generate learning data. It is possible to prevent erroneous detection such that a cell nucleus is erroneously present even though it does not exist, thereby increasing the accuracy of cell detection.

FIG. 12 is an explanatory diagram showing another example of the estimated stained nuclear image and the combined image. Note that FIG. 12 shows only one cell (cell nucleus) for convenience. The first image data acquisition unit 52 acquires a plurality of phase-contrast images including detection target cells captured in time series. For example, one phase-contrast image of the observed region can be obtained every 10 minutes or 20 minutes. The input data generation unit 512 extracts (specifies) a plurality of clipped images each having a predetermined number of pixels (eg, g×g) from each of the phase contrast images. The input data generation unit 512 generates a plurality of sets of input data for each of the plurality of phase contrast images, based on the luminance value of each pixel of the plurality of clipped images. The multiple sets of input data correspond to multiple phase contrast images.

Since the estimated image generation unit 513 generates one estimated nuclear stained image based on one phase contrast image, the same number of time-series estimated nuclear staining images as the number of phase contrast images are generated for a plurality of phase contrast images. A stained image can be generated.

The display unit 56 can display an estimated movement trajectory of a cell (cell nucleus) based on output values output from the detection model unit 511 to which multiple sets of input data are input. That is, in the estimated stained nuclear image of FIG. 12A, the estimated movement trajectory from the past estimated position of the cell nucleus to the current (most recent) estimated position is represented. Also, in the composite image of FIG. 12B, the estimated movement trajectory from the past estimated positions of both the cell and the cell nucleus in the cell to the current (most recent) estimated position is represented. This makes it possible to accurately grasp the dynamic behavior of cells.

In the above example, the detection device 50 is configured to include one detection model unit 511, but is not limited to this, and may be configured to include a plurality of detection model units. In this case, for example, according to the type of the region of interest of the cell to be detected, a plurality of detection model units having different detection models for each type can be provided. As a result, it is possible to use the optimum detection model part according to the type of the region of interest of the cell, and it is possible to further improve the detection accuracy of the cell.

Also, the detection model unit 511 can be configured to be relearnable. The learning processing unit 514 updates a plurality of parameters (weight between the input layer and the intermediate layer, weight between the intermediate layer and the output layer, weight between the intermediate layers, etc.) of the detection model unit 511, and detects The model portion 511 can be relearnable. Thereby, cell detection accuracy can be maintained.

Next, another configuration of the detection model unit 511 will be described.

FIG. 13 is an explanatory diagram schematically showing a second example of the configuration of detection model section 511 of the present embodiment. The difference from the first example illustrated in FIG. 3 is that the number of nodes in the output layer is 2 (y1, y2). The number of input nodes in the input layer and the structure of the intermediate layer are the same as in the first example. The output layer, based on the input data (x1, x2, . to output The output value can take any value between 0 and 1 (not limited to integers).

FIG. 14 is an explanatory diagram showing a second example of a method of generating learning data for learning the detection model unit 511 of this embodiment. The learning data generating method of FIG. 14 corresponds to the second example of the detection model unit 511 shown in FIG.

As shown in FIG. 14A , the learning data generation unit 515 acquires from the memory 54 image data of a nuclear staining image (for example, the resolution is 800×600) including the region of interest of the learning cell captured in the observation region.

As shown in FIG. 14B, the learning data generation unit 515 performs binarization processing on the stained nuclear image to generate a binarized image. By binarizing, for example, the coloring position can be made the foreground. In addition, binarization makes it possible to clearly distinguish between the region of interest of the cell and the region other than the region of interest. For example, binarization makes it possible to clearly distinguish between a portion corresponding to the cell nucleus and a portion not corresponding to the cell nucleus, thereby facilitating the learning of the detection model unit 511 .

As shown in FIG. 14C, the learning data generation unit 515 identifies a plurality of pixel blocks (for example, g×g, g=28, etc.) composed of a predetermined number of pixels from the stained nuclear image. In the example of FIG. 14C, four pixel blocks G6, G7, G8 and G9 are shown for convenience. In practice, a required number of pixel blocks can be identified from a single stained nuclear image. The required number may be, for example, a number equal to the total number of pixels in one stained nuclear image, or may be a number smaller than the total number of pixels in order to reduce the image processing load. The pixel blocks can be specified by randomly selecting positions on the stained nuclear image, and not only pixel blocks containing cell nuclei but also pixel blocks not containing cell nuclei can be specified. By also specifying pixel blocks that do not contain cell nuclei, it is possible to prevent erroneous detection, such as the presence of cell nuclei in spite of the fact that cell nuclei do not exist, and improve the cell detection accuracy. In addition, it is not necessary to distinguish whether the cells are positive cells or dead cells. Pixel blocks G6 and G8 contain cell nuclei (colored areas), and pixel blocks G7 and G9 do not contain cell nuclei. Also, the feature amounts of the pixel blocks G6, G7, G8, and G9 are assumed to be binary, 1, 0, 1, and 0, respectively. Binary is the pixel value of the pixel. The binary value can be, for example, but is not limited to, the binary value of the pixel located in the center of the pixel block.

The learning data generator 515 acquires from the memory 54 image data of a phase-contrast image (with a resolution of, for example, 800×600) including learning cells captured in the observation region. Note that the phase-contrast image and the stained nuclear image are obtained by imaging the same observation region of the learning cell at the same time.

The learning data generation unit 515 adds pixel values of binarized predetermined pixels of the pixel block specified on the stained nuclear image to the pixel block of the phase contrast image corresponding to the pixel block specified on the stained nuclear image. Learning data is generated by associating values (0 or 1). More specifically, the learning data generation unit 515 generates learning data by associating the pixel block of the phase contrast image with the binary pixel value of the central pixel in the pixel block specified on the stained nuclear image. Since binary pixel values are used as training data (teaching data), it is not necessary to design appropriate feature vectors compared to the case where feature vectors are calculated and used as teaching data. Since the pixel values can be used as they are compared to the case of calculation, the processing is simplified and cell detection can be performed with high accuracy.

As shown in FIG. 14D, the learning data generator 515 cuts out regions X6, X7, X8, and X9 (cropped images) of the phase contrast image corresponding to the pixel blocks G6, G7, G8, and G9 on the stained nuclear image. The regions X6 to X9 are associated with the binary values of the pixel blocks G6 to G9 as teacher labels. Since the regions X6 to X9 are each composed of g×g pixels, for the region X6, a vector (x61, x62, , x6N) are generated, and the teacher label is 1.

For the region X7, a vector (x71, x72, . For the region X8, a vector (x81, x82, . For the region X9, a vector (x91, x92, .

FIG. 15 is an explanatory diagram showing a second example of learning by the learning processing unit 514 of this embodiment. The learning processing unit 514 causes the detection model unit 511 to learn based on the learning data generated by the learning data generation unit 515 . Specifically, the learning processing unit 514 inputs learning input data (x61, x62, . Learning is performed so that the output value y2 of 1 approaches 1 and the output value y1 of the output node 0 approaches 0.

Also, although not shown, learning input data (x71, x72, . Learning is performed so that the value y1 approaches one and the output value y2 of the output node 1 approaches zero. The same applies to the learning input data of samples X8 and X9.

That is, when input data of teacher label 1 is input, learning is performed so that node 0 approaches 0 and node 1 approaches 1, and when input data of teacher label 0 is input, node 0 becomes Learning is performed so that it approaches 1 and node 1 approaches 0.

FIG. 16 is a flow chart showing a second example of the procedure of the learning process of the processing section 51 of this embodiment. In the following description, for the sake of convenience, the processing unit 51 is assumed to be the subject of processing. The processing unit 51 acquires image data of a phase-contrast image including a learning cell in which an observation region is imaged (S51), and acquires nuclei including a region of interest of the learning cell captured in the same observation region at the same time. Image data of the stained image is acquired (S52).

The processing unit 51 binarizes the stained nuclear image (S53), randomly identifies a plurality of pixel blocks from the binarized stained nuclear image (S54), and records the binary values of the central pixels of the identified pixel blocks ( S55). The processing unit 51 extracts a clipped image at a position corresponding to the pixel block from the phase difference image (S56).

The processing unit 51 uses the luminance value of each pixel of the extracted clipped image as learning input data, and generates learning data using the binary value of the center pixel of the pixel block corresponding to the clipped image as a teacher label (S57). The processing unit 51 causes the detection model unit 511 to learn based on the generated learning data (S58).

The processing unit 51 determines whether the accuracy of the output value of the detection model unit 511 is within the allowable range (S59), and if it is not within the allowable range (NO in S59), continues the processing from step S51. If the accuracy of the output value is within the allowable range (YES in S59), the processing unit 51 terminates the process.

The method of generating input data in the second example is the same as in the first example illustrated in FIG. 7, and M (M=m×n) clipped images A1, A2, . .

FIG. 17 is an explanatory diagram showing a second example of detection processing by the detection model unit 511 of this embodiment. When the input data (a11, a12, a13, . . . , a1N) for the clipped image A1 is input to each input node of the input layer, the output data B1 is output to the output node of the output layer. Output data B1 has a probability of output node 0 of 0.4 and a probability of output node 1 of 0.1. If the probability of output node 0 is greater than or equal to the probability of output node 1, the output value is set to 0. If the probability of output node 0 is less than the probability of output node 1, the output value is set to 1. For the output data B1, the output value b1 is 0 because the probability of the output node 0 is greater than or equal to the probability of the output node 1.

When the input data (a21, a22, a23, . . . a2N) for the clipped image A2 is input, the output data B2 is output to the output node of the output layer. Output data B2 has a probability of output node 0 of 0.3 and a probability of output node 1 of 0.8. For the output data B2, the output value b2 is 1 because the probability of the output node 0 is less than the probability of the output node 1.

Similarly, when the input data (aM1, aM2, aM3, . . . aMN) for the clipped image AM are input, the output data BM are output to the output nodes of the output layer. The output data BM has a probability of output node 0 of 0.5 and a probability of output node 1 of 0.2. As for the output data BM, the output value bM is 0 because the probability of the output node 0 is equal to or greater than the probability of the output node 1 .

The estimated image generating unit 513 generates an estimated image of the stained nuclear image based on the output values b1, b2, . As shown in FIG. 7, when the resolution of the phase difference image is m×n, the estimated image generation unit 513 associates M (=m×n) output values with m×n pixels. , m×n putative nuclear staining images.

FIG. 18 is a flow chart showing a second example of the procedure of detection processing by the processing unit 51 of this embodiment. The processing unit 51 acquires image data of a phase contrast image including cells to be detected (S71), scans the acquired phase contrast image, and extracts a clipped image (S72). The processing unit 51 generates input data based on the luminance value of each pixel of the extracted clipped image (S73).

The processing unit 51 inputs the generated input data to the input node of the detection model unit 511 (S74), and the value (probability) of the node 0 output by the detection model unit 511≧the value (probability) of the node 1 for each clipped image. , the output value is set to 0 (black), and if the value (probability) of node 0 < the value (probability) of node 1, the output value is set to 1 (white), and the pixel value (brightness value) of the estimated nuclear staining image is is determined (S75).

The processing unit 51 generates an estimated nuclear staining image (S76), displays the phase contrast image and the estimated nuclear staining image in comparison (S77), and ends the process.

Note that the processing unit 51 and the display unit 56 of the present embodiment can be configured by a CPU, GPU, ROM, RAM, recording medium reading unit, and the like. A computer program recorded on a recording medium can be read by a recording medium reading unit and stored in a RAM. The processing performed by the processing unit 51 and the display unit 56 can be executed by causing the CPU and GPU to execute computer programs stored in the RAM. Note that the CPU is a general term for processors, processor devices, processing units, etc., and can be composed of one or more. Also, the computer program can be downloaded via a network such as the Internet instead of being read by the recording medium reading unit.

In the second example shown in FIGS. 14, 16, etc., the configuration is such that binarization processing is performed on the stained nuclear image. In the following, preprocessing performed before binarization processing will be described.

FIG. 19 is an explanatory diagram showing an example of a stained nuclear image after binarization obtained by binarizing a stained nuclear image with different threshold values. FIG. 19A shows a stained nuclear image (original image before binarization) obtained by imaging. When a cell is subjected to nuclear staining, the region of interest of the cell (for example, the cell nucleus) does not necessarily emit light with the same intensity. Therefore, as shown in FIG. exists.

As shown in FIG. 19B, when binarization is performed with a threshold of 140, pixels with relatively high luminance in non-cell regions in the image appear as the foreground as coloring positions, and non-cell regions are erroneously treated as cells. will be taken.

In addition, as shown in FIG. 19C, binarization with a threshold value of 150 can somewhat suppress areas that are not cells in the image from being erroneously treated as cells, but the color develops at a weak intensity. In some cases, the cell nucleus is removed and the cell nucleus cannot be detected.

Further, as shown in FIG. 19D, binarization with a threshold value of 160 can prevent areas that are not cells in the image from being erroneously treated as cells, but the color develops with a slightly strong intensity. Even cell nuclei that are present may be removed. Thus, it may not be possible to accurately detect all cell nuclei no matter what threshold is used.

FIG. 20 is a flow chart showing an example of a pre-processing procedure for binarization processing by the processing unit 51 of the present embodiment. The processing shown in FIG. 20 includes processing added between steps S52 and S53 shown in FIG.

The processing unit 51 acquires image data of a stained nuclear image (S101). The processing unit 51 generates an intermediate image using the number of pixels in which the luminance value of the peripheral pixels of the pixel of interest in the stained nuclear image is smaller than the luminance value of the pixel of interest as the value of the pixel of interest (S102). The pixel of interest is scanned in the horizontal direction from the upper left of the stained nuclear image, and after the horizontal scanning is completed, the scanning position is shifted vertically and the scanning is repeated again from left to right, and finally scanned to the lower right. . The peripheral pixels of the target pixel can be, for example, 21×21 pixels including the target pixel, but the number of peripheral pixels is not limited to this.

In the generated intermediate image, pixels with large pixel values represent low brightness around the pixel of interest in the stained nuclear image, and pixels with small pixel values represent high brightness around the pixel of interest in the stained nuclear image. represents. That is, it can be said that pixels with small pixel values in the intermediate image include pixels with high relative luminance values and correspond to cell nuclei with relatively higher emission intensity than surroundings.

The processing unit 51 Fourier-transforms the intermediate image to calculate a frequency spectrum (S103), and removes high-frequency components using a low-pass filter (S104). As a result, it is possible to remove areas with fine patterns or edge areas where the luminance value changes abruptly. can.

The processing unit 51 generates a preprocessed stained nuclear image by inverse Fourier transform (S105), binarizes the stained nuclear image (S106), and ends the process. Strictly speaking, the nuclear-stained images referred to in steps S105 and S106 are intermediate images corresponding to the nuclear-stained images.

FIG. 21 is an explanatory diagram showing an example of an image in preprocessing for binarization processing by the processing unit 51 of this embodiment. FIG. 21A shows a nuclear stained image, which is the original image. FIG. 21B shows an intermediate image in which the value of the pixel of interest is the number of pixels whose luminance value is lower than that of the pixel of interest within the range of pixels surrounding the pixel of interest (for example, 21×21 pixels). FIG. 21C shows the frequency spectrum after Fourier transform. In FIG. 21C, the vicinity of the center is the low frequency component, and the farther from the center, the higher the frequency component.

FIG. 21D shows a frequency spectrum with high frequency components removed by a low-pass filter. In the example of FIG. 21D, areas other than the circular shape near the center are removed as high frequency components. FIG. 21E shows the intermediate image after the inverse Fourier transform. FIG. 21F shows the image after binarization. Comparing FIG. 21A and FIG. 21F, it can be seen that in FIG. 21A, cell nuclei with different emission intensities (different luminance values) appear as the foreground.

As described above, the processing unit 51 acquires the second image. The processing unit 51 compares the luminance value of the pixel of interest with the luminance value of the pixels surrounding the pixel of interest (for example, 21×21 pixels including the pixel of interest) in the acquired second image, and determines the pixels whose luminance values are smaller than that of the pixel of interest. is calculated, and the calculated number of pixels is used as the pixel value of the pixel of interest to generate an intermediate image. The processing unit 51 Fourier-transforms the intermediate image and removes high-frequency components in the frequency spectrum after the Fourier-transformation. The processing unit 51 generates an intermediate image (nuclear stained image) by inverse Fourier transforming the frequency spectrum from which the high frequency component has been removed. The processing unit 51 binarizes the generated intermediate image.

As another method for obtaining a binarized image, the learning data generation unit 515 may use, for example, a background subtraction method with a background image with no cells and only brightness unevenness (as a method for obtaining a background image, there are no cells). It is conceivable that the state is photographed, an image is obtained by taking the median value in the time-series direction from the time-lapse image, etc.), or a binarized image can be generated using adaptive binarization processing or the like. The adaptive binarization process is a method of locally setting a threshold for each region composed of a plurality of pixels without fixing the threshold for binarization. For example, an average value of pixel values of pixels in a certain area is set as the threshold value of the area.

FIG. 22 is a block diagram showing another example of the configuration of the detection device 50 of this embodiment. The difference from the configuration illustrated in FIG. 1 is that the learning processing unit 514, the learning data generation unit 515, and the second image data acquisition unit 53 are not provided in the example of FIG. That is, in the detection device 50 shown in FIG. 22, the detection model unit 511 has already been trained, and if further learning is not required, the learning processing unit 514, the learning data generation unit 515, and the second image data acquisition unit 514 The part 53 is not an essential component. In addition, in the detection device 50 illustrated in FIG. 22, other parts are the same as those in the configuration of FIG. 1, so description thereof will be omitted.

FIG. 23 is a block diagram showing an example of the configuration of the microscope 100 of this embodiment. A microscope 100 will be described below as an example of an imaging apparatus according to the present embodiment. The microscope 100 comprises the detection device 50 illustrated in FIG. 1 or 22 . Note that the configuration of the detection device 50 is not limited to being incorporated in the microscope 100 .

The microscope 100 includes, in addition to the detection device 50, a white light source 11 that emits a white color when capturing a phase contrast image, and a well plate containing cells C to which a nuclear stain that generates fluorescence by excitation light is applied. 10, an objective lens 13, a fluorescence excitation light source 12, a fluorescence mirror unit 14 arranged below the well plate 10, an imaging device 15, an AD converter (A/ D) 16, CPU 20, memory 23, operation unit 22 for the user to operate the microscope 100, fluorescence excitation light source 12, objective lens 13, fluorescence mirror unit 14, imaging element 15, etc. are horizontally arranged with respect to the well plate 10. It includes a drive control unit 21 and the like for moving in the direction.

When capturing a phase-contrast image, the white light from the white light source 11 is irradiated from above to the bottom of the well plate 10, and the light that passes through the cells C (diffracted light) and the light that does not pass through the cells C (direct The shape information of cells can be obtained by the imaging device 15 as a contrast between light and dark due to the phase difference with light. Note that the microscope 100 can also capture a bright-field image instead of the phase-contrast image, or together with the phase-contrast image.

When capturing a stained nuclear image, the excitation light emitted from the fluorescence excitation light source 12 is reflected toward the well plate 10 by the fluorescence mirror unit 14 and enters the cells C via the objective lens 13 . Fluorescence is emitted from the cells C by the incident excitation light. Fluorescence emitted from the cells C is transmitted through the objective lens 13 and the fluorescence mirror unit 14 and is incident on the imaging device 15 to obtain a stained nuclear image.

Under the control of the CPU 20, switching between the imaging of the phase contrast image and the imaging of the stained nuclear image can be performed at the same point (observation region) at the same time. Further, according to the operation from the operation unit 22, for example, it is possible to set the shooting location and the shooting timing and automatically shoot.

FIG. 24 is a schematic diagram showing an example of the configuration of the detection system of this embodiment. The detection system may comprise, for example, a microscope 100, a terminal device 200 for use by a user, a server 300 for providing applications, databases, or the like. A display device 60 capable of displaying an image is connected to the terminal device 200 . Note that the display device 60 can also be incorporated in the terminal device 200 . In the detection system shown in FIG. 24, the detection device 50 of this embodiment can be incorporated into any one of the microscope 100, the terminal device 200 and the server 300. FIG. In the example of FIG. 23 described above, an example in which the detection device 50 is incorporated in the microscope 100 has been described.

The microscope 100 and the terminal device 200 can be connected via a communication line, for example. Also, the terminal device 200 and the server 300 can be connected via the communication network 1 such as the Internet.

When the detection device 50 is incorporated in the server 300, the server 300 includes a detection model unit 511 that detects cells, an acquisition unit that acquires image data related to at least one of a phase contrast image or a bright field image containing cells, and an acquired image. An output unit for inputting data to the detection model unit 511 and outputting estimated image data related to an estimated nuclear staining image including the region of interest of the cell without staining the cell is provided.

In the above embodiment, the stained nuclear image has been described as an example of the second image obtained by the second method. The nuclear staining image includes not only the staining of the cell nucleus, but also an image that allows recognition of intracellular organelles. In addition, the second image is not limited to fluorescent imaging such as a nuclear staining image using a staining agent, for example, RI (radioisotope) imaging using a contrast agent. Images are also included.

Next, a method for providing mouse epidermal cells will be described. As the cells to be provided, mouse epidermal cells are described below, but the cells to be provided are not limited to mouse epidermal cells.

FIG. 25 is a schematic diagram showing an example of a method for providing mouse epidermal cells according to the present embodiment. As shown in FIG. 25, a distribution server 310, a provision server 320, a terminal device 201 used by a user, a tablet terminal 202, and the like are connected via a communication network. Note that the tablet terminal 202 may be a smart phone.

Transmission unit 311 of distribution server 310 distributes the computer program of the present embodiment to terminal device 201 of a specific user. The distributed computer program can be installed on the terminal device 201, whereby the terminal device 201 has a configuration in which the detecting device 50 of the present embodiment is incorporated. The specific user is, for example, a user of a company, research institution, medical institution, or the like that performs cell analysis using the computer program of the present embodiment, and can be a pre-registered user.

The user requests the provision server 320 to provide mouse epidermal cells from the tablet terminal 202 . The request can include an ID, password, etc. for user authentication.

When the reception unit 321 of the providing server 320 receives the request from the user, the authentication unit 322 performs authentication processing to determine whether the user is a legitimate user. As a result of authentication processing, if the request is from a legitimate user, the information generation unit 323 generates provision information for providing mouse epidermal cells to the user. Provided information for providing mouse epidermal cells includes, for example, destination information for transporting mouse epidermal cells.

As a result, mouse epidermal cells can be provided without fail to specific users to whom the computer program of the present embodiment has been distributed. In addition, by providing cells cultured in the same culture environment as the cells used for learning together with a computer program, the accuracy of cell detection by the computer program can be ensured.

Note that the distribution server 310 and the provision server 320 can be integrated into one server (for example, the provision server 320).

FIG. 26 is a flow chart showing an example of the procedure of the method for providing mouse epidermal cells according to the present embodiment. For the sake of convenience, the following description assumes that the main body of processing is the providing server 320 . The providing server 320 distributes the program (computer program) of the present embodiment to a specific user (S91). The provision server 320 determines whether or not there is a request for provision of mouse epidermal cells from the user (S92), and if there is no request (NO in S92), continues the processing of step S92.

If there is a request (YES in S92), the providing server 320 performs user authentication processing (S93) and determines whether the user is authorized (S94). If the user is a legitimate user (YES in S94), the provision server 320 generates provision information for providing mouse epidermal cells to the user (S95), and ends the process. If the user is not an authorized user (NO in S94), the providing server 320 terminates the process.

A computer program according to the present embodiment is a computer program for causing a computer to detect a specimen, wherein the computer is provided with first image data relating to a first image containing a learning specimen obtained by a first method. and a second image including the region of interest of the learning specimen obtained by a second method different from the first method, which enables identification of the region of interest in the learning specimen. A process of acquiring image data, a process of generating learning data based on the first image data and the second image data, and the first image data related to the first image containing the specimen obtained by the first method. a process of training the detector based on the generated learning data so that the detector can detect the specimen based on the.

A computer program according to the present embodiment provides a computer with a process of identifying a plurality of pixel blocks each having a predetermined number of pixels from the second image, and identifying a pixel block of the first image corresponding to the identified pixel block. and generating the learning data by associating the feature values of the pixels of the specified pixel block.

A computer program according to the present embodiment causes a computer to execute a process of generating the learning data by associating the pixel block of the first image with the pixel value of the predetermined pixel in the specified pixel block.

A computer program according to the present embodiment provides a computer with a process for binarizing the second image and a predetermined either a process of specifying a plurality of pixel blocks composed of the number of pixels, or pixel values of binarized predetermined pixels of the specified pixel blocks in the pixel blocks of the first image corresponding to the specified pixel blocks. are associated with each other to generate the learning data.

A computer program according to the present embodiment causes a computer to execute a process of displaying an estimated position of a sample based on a result of learning the detector.

A computer program according to the present embodiment provides a computer with a process of acquiring first image data related to a first image containing a specimen, obtained by the first method, and a predetermined number of pixels from the first image. A process of specifying a plurality of pixel blocks to be configured, a process of generating input data based on the pixel values of each pixel in the specified pixel blocks, and an output value output from the detector to which the generated input data is input. and generating an estimated image of the second image including the region of interest of the specimen obtained by the second method without relying on the second method.

A computer program according to the present embodiment causes a computer to execute processing for displaying the first image and the estimated image in comparison.

A computer program according to the present embodiment provides a computer with a process of identifying a plurality of pixel blocks each composed of a predetermined number of pixels from each of a plurality of first images containing a specimen obtained in time series by the first method. a process of generating a plurality of sets of input data for each of the plurality of first images based on the pixel value of each pixel of the specified pixel block; and the detector to which the plurality of sets of input data are input. and a process of displaying the estimated movement trajectory of the specimen based on the output value output by the.

The computer program according to the present embodiment provides a computer with a process of acquiring the first image data related to at least one of a phase contrast image or a bright field image containing cells, and a staining image containing the region of interest of the cells. and a process of acquiring the second image data.

A computer program according to the present embodiment is a computer program for causing a computer to detect a specimen, wherein the computer acquires first image data relating to a first image containing the specimen obtained by a first method. The second method, without relying on a second method different from the first method, that enables identification of a region of interest in the specimen based on the acquired first image data and the trained detector. and generating an estimated image of the second image including the region of interest of the specimen obtained by.

A detection device according to the present embodiment is a detection device that detects a sample, and includes a first acquisition unit that acquires first image data related to a first image that includes a learning sample and is obtained by a first method; , obtaining second image data relating to a second image including the region of interest of the learning specimen obtained by a second method different from the first method, which enables identification of the region of interest in the learning specimen a second acquisition unit, a learning data generation unit that generates learning data based on the first image data and the second image data, and a first image related to the first image containing the specimen obtained by the first method a learning processing unit that makes the detector learn based on the generated learning data so that the detector can detect the specimen based on the data.

The detection device according to the present embodiment includes an acquisition unit that acquires first image data related to a first image containing a specimen obtained by the first method, and a predetermined number of pixels from the first image. an specifying unit for specifying a plurality of pixel blocks to be specified, an input data generating unit for generating input data based on pixel values of pixels in the pixel blocks specified by the specifying unit, and input data generated by the input data generating unit. An image for generating an estimated image of a second image including the region of interest of the specimen obtained by the second method without relying on the second method, based on the input output values output by the detector. and a generator.

The detection apparatus according to the present embodiment includes a plurality of detectors for each type of region of interest of the specimen.

In the detection device according to this embodiment, the detector has a plurality of parameters for converting input data into an output value, and the learning processing unit updates the plurality of parameters of the detector to detect the Makes the device relearnable.

The imaging device according to the present embodiment includes a first output unit that outputs at least one of a phase-contrast image and a bright-field image containing cells obtained by imaging, and a region of interest of the cell obtained by imaging. A second output unit that outputs a stained image containing a and a second acquisition unit that acquires second image data related to the stained image including the region of interest of the learning cell output by the second output unit, and based on the first image data and the second image data Based on the first image data related to at least one of the phase-contrast image and the bright-field image containing the cell, output by the learning data generation unit that generates learning data and the first output unit so that the detector can detect the cell a learning processing unit for learning the detector based on the generated learning data.

The imaging device according to the present embodiment includes a specifying unit that specifies a plurality of pixel blocks each having a predetermined number of pixels from at least one of a phase-contrast image and a bright-field image containing cells, and a pixel block specified by the specifying unit. an input data generation unit that generates input data based on the pixel value of each pixel of the input data generation unit; and an image data generation unit that generates estimated image data related to an estimated image of a stained image that includes the region of interest of the cell.

The server according to the present embodiment includes a detector that detects cells, an acquisition unit that acquires image data related to at least one of a phase contrast image or a bright field image containing cells, and image data acquired by the acquisition unit. and an output unit for inputting to the detector and outputting estimated image data relating to an estimated image of a stained image including the region of interest of the cell without staining the cell.

The image data according to the present embodiment is image data in which cells are detected, and has a structure including estimated image data related to an estimated image of a stained image including a region of interest of a cell obtained by imaging, The estimated image is obtained by inputting input data based on first image data related to at least one of a phase contrast image or a bright field image containing cells to a detector that detects cells, and the detector is generated based on the output value output by the detector, the first image data related to at least one of the phase contrast image or the bright field image containing the learning cell obtained by imaging, and the first image data obtained by imaging learning is performed by learning data generated based on the second image data related to the stained image including the region of interest of the learning cell.

The detector according to the present embodiment is a detector that detects cells, and is obtained by imaging, first image data related to at least one of a phase contrast image or a bright field image containing learning cells, and Learning is performed by learning data generated based on the second image data related to the stained image including the region of interest of the learning cell obtained by imaging.

The detector according to the present embodiment identifies a plurality of pixel blocks composed of a predetermined number of pixels from the stained image, and at least one pixel block of the phase contrast image or the bright field image corresponding to the identified pixel block. is learned by learning data generated by associating the feature values of the pixels of the specified pixel block.

The detector according to the present embodiment includes an input unit that receives as input data image data related to at least one of a phase contrast image and a bright field image containing cells obtained by imaging, and an input received by the input unit an output unit configured to output estimated image data related to an estimated image of a stained image including a region of interest of the cell obtained by imaging the cell without staining the cell, based on the data.

The detection method according to the present embodiment is a detection method using a detector that detects cells, and the detector is obtained by imaging and relates to at least one of a phase contrast image or a bright field image containing cells Receiving image data as input data, and outputting estimated image data related to an estimated image of a stained image including a region of interest of the cell obtained by imaging the cell without staining the cell based on the received input data. .

The provision method according to the present embodiment is a method for providing cells cultured under a predetermined environment, wherein at least one of a phase contrast image or a bright field image containing the cells obtained by imaging a computer is and outputting estimated image data related to an estimated image of a stained image including a region of interest of the cell obtained by imaging the cell without staining the cell based on the acquired image data. When a computer program for executing the processing to be performed is distributed, and a request from the user to whom the computer program is distributed is received, information for providing the user with cells cultured under a predetermined environment is generated.

50 detection device 51 processing unit 511 detection model unit 512 input data generation unit 513 estimated image generation unit 514 learning processing unit 515 learning data generation unit 52 first image data acquisition unit 53 second image data acquisition unit 54 memory 55 output unit 56 display Section 60 Display Device 100 Microscope 200, 201 Terminal Device 300 Server 310 Distribution Server 320 Provision Server

Claims (18)

A computer program for causing a computer to detect an analyte,
to the computer,
A process of acquiring first image data related to a first image containing a learning sample obtained by the first method;
A process of acquiring second image data relating to a second image including the region of interest of the learning specimen, which is obtained by a second method different from the first method that enables identification of the region of interest in the learning specimen. When,
a process of generating learning data based on the first image data and the second image data;
A process of learning the detector based on the generated learning data so that the detector can detect the specimen based on the first image data related to the first image containing the specimen obtained by the first method;
a process of binarizing the second image;
a process of identifying a plurality of pixel blocks each composed of a predetermined number of pixels corresponding to each of the binary pixel values of the pixels obtained by the binarization process;
a computer that causes the pixel block of the first image corresponding to the specified pixel block to be associated with any of the pixel values of the binarized predetermined pixels of the specified pixel block to generate the learning data. program. to the computer,
2. The computer program according to claim 1, which executes a process of displaying the estimated position of the specimen based on the result of learning the detector. to the computer,
a process of acquiring first image data relating to a first image containing a specimen, obtained by the first method;
a process of identifying a plurality of pixel blocks each composed of a predetermined number of pixels from the first image;
a process of generating input data based on the pixel value of each pixel in the identified pixel block;
Estimation of a second image including the region of interest of the specimen obtained by the second method without relying on the second method, based on the output value output by the detector to which the generated input data is input 3. The computer program according to claim 1 or 2, causing the execution of a process of generating an image. to the computer,
4. The computer program according to claim 3, causing execution of processing for displaying the first image and the estimated image in comparison. to the computer,
A process of identifying a plurality of pixel blocks composed of a predetermined number of pixels from each of a plurality of first images containing the specimen obtained in time series by the first method;
generating a plurality of sets of input data for each of the plurality of first images based on the pixel value of each pixel in the identified pixel block;
5. The process of displaying an estimated movement trajectory of the specimen based on output values output by the detector to which the plurality of sets of input data are input, and executing: computer program. to the computer,
A process of acquiring the first image data related to at least one of a phase contrast image or a bright field image containing cells;
6. The computer program according to any one of claims 1 to 5, causing execution of a process of acquiring the second image data related to the stained image including the region of interest of the cell. A computer program for causing a computer to detect an analyte,
to the computer,
a process of acquiring first image data relating to a first image containing the specimen obtained by the first method;
The obtained by the second method without relying on a second method different from the first method, which enables identification of a region of interest in the specimen based on the acquired first image data and the trained detector. a process of generating an estimated image of the second image containing the region of interest of the specimen;
a process of binarizing the second image;
a process of identifying a plurality of pixel blocks each composed of a predetermined number of pixels corresponding to each of the binary pixel values of the pixels obtained by the binarization process;
Learning data for learning the learned detector is generated by associating a pixel block of the first image corresponding to the specified pixel block with any of pixel values of binarized predetermined pixels of the specified pixel block. A computer program that causes a process to be performed. A detection device for detecting a specimen,
a first acquisition unit that acquires first image data related to a first image containing a learning sample, which is obtained by the first method;
Acquiring second image data related to a second image including the region of interest of the learning specimen obtained by a second method different from the first method, which enables identification of the region of interest in the learning specimen 2 an acquisition unit;
a learning data generation unit that generates learning data based on the first image data and the second image data;
A learning processing unit for learning the detector based on the generated learning data so that the detector can detect the specimen based on the first image data related to the first image containing the specimen obtained by the first method. and
The learning data generation unit
binarizing the second image;
identifying a plurality of pixel blocks each composed of a predetermined number of pixels corresponding to each binary value, which is the pixel value of the pixel obtained by the binarization;
A detection device that generates the learning data by associating a pixel block of the first image corresponding to the specified pixel block with any of pixel values of binarized predetermined pixels of the specified pixel block. an acquisition unit that acquires first image data related to a first image containing the specimen, which is obtained by the first method;
a specifying unit that specifies a plurality of pixel blocks each having a predetermined number of pixels from the first image;
an input data generation unit that generates input data based on the pixel value of each pixel in the pixel block identified by the identification unit;
including the region of interest of the specimen obtained by the second method without relying on the second method, based on the output value output by the detector to which the input data generated by the input data generation unit is input 9. The detection device according to claim 8, comprising an image generator for generating an estimated image of the second image. 10. The detection apparatus according to claim 8, comprising a plurality of said detectors for each type of region of interest in the specimen. The detector is
has multiple parameters that transform input data into output values,
The learning processing unit
11. A detection apparatus according to any one of claims 8 to 10, wherein the plurality of parameters of the detector are updated to allow the detector to be retrained. a first output unit that outputs at least one of a phase-contrast image or a bright-field image containing cells obtained by imaging;
a second output unit that outputs a stained image containing a region of interest of a cell obtained by imaging;
a first acquisition unit that acquires first image data related to at least one of a phase contrast image or a bright field image containing learning cells, which is output by the first output unit;
a second acquisition unit for acquiring second image data relating to a stained image including the region of interest of the learning cell, output by the second output unit;
a learning data generation unit that generates learning data based on the first image data and the second image data;
The detection based on the generated learning data so that the detector can detect the cell based on the first image data related to at least one of the phase contrast image or the bright field image containing the cell, which the first output unit outputs and a learning processing unit for learning a device,
The learning data generation unit
binarizing the stained image;
identifying a plurality of pixel blocks each composed of a predetermined number of pixels corresponding to each binary value, which is the pixel value of the pixel obtained by the binarization;
The learning data by associating a pixel block of at least one of the phase-contrast image and the bright-field image corresponding to the specified pixel block with the binarized pixel value of the specified pixel of the specified pixel block. An imaging device that produces an identifying unit that identifies a plurality of pixel blocks each composed of a predetermined number of pixels from at least one of a phase-contrast image and a bright-field image containing cells;
an input data generation unit that generates input data based on the pixel value of each pixel in the pixel block identified by the identification unit;
Based on the output value output by the detector to which the input data generated by the input data generation unit is input, estimated image data related to an estimated image of a stained image including the region of interest of the cell without staining the cell. 13. The imaging apparatus according to claim 12, further comprising: an image data generation unit that generates a . a detector for detecting cells;
an acquisition unit that acquires image data related to at least one of a phase-contrast image and a bright-field image containing cells;
an output unit that inputs image data acquired by the acquisition unit to the detector and outputs estimated image data related to an estimated image of a stained image that includes a region of interest of the cell without staining the cell;
a learning data generation unit that generates learning data for learning the detector,
The learning data generation unit
binarizing the stained image;
identifying a plurality of pixel blocks each composed of a predetermined number of pixels corresponding to each binary value, which is the pixel value of the pixel obtained by the binarization;
The learning data by associating a pixel block of at least one of the phase-contrast image and the bright-field image corresponding to the specified pixel block with the binarized pixel value of the specified pixel of the specified pixel block. server that generates the . A detector for detecting cells,
First image data related to at least one of a phase-contrast image or a bright-field image containing the learning cell obtained by imaging, and a stained image containing the region of interest of the learning cell obtained by imaging Learned by learning data generated based on the second image data,
Further, binarize the stained image,
identifying a plurality of pixel blocks each composed of a predetermined number of pixels corresponding to each binary value, which is the pixel value of the pixel obtained by the binarization;
generated by associating a pixel block of at least one of the phase-contrast image and the bright-field image corresponding to the specified pixel block with one of the binarized pixel values of the specified pixels of the specified pixel block A detector that has been trained on the training data. an input unit that receives, as input data, image data relating to at least one of a phase-contrast image containing cells and a bright-field image obtained by imaging;
an output unit for outputting estimated image data related to an estimated image of a stained image including a region of interest of the cell obtained by imaging the cell without staining the cell, based on the input data received by the input unit; 16. The detector of claim 15, comprising: A detection method using a detector that detects cells,
The detector is
Accepting as input data image data relating to at least one of a phase-contrast image or a bright-field image containing cells, obtained by imaging;
Based on the received input data, outputting estimated image data related to an estimated image of a stained image including a region of interest of the cell obtained by imaging without staining the cell,
binarizing the stained image;
identifying a plurality of pixel blocks each composed of a predetermined number of pixels corresponding to each binary value, which is the pixel value of the pixel obtained by the binarization;
A pixel block of at least one of the phase-contrast image and the bright-field image corresponding to the specified pixel block is associated with one of the binarized pixel values of predetermined pixels of the specified pixel block, and the detector A detection method that generates training data for learning . A method for providing cells cultured under a predetermined environment, comprising:
to the computer,
A process of acquiring image data related to at least one of a phase-contrast image or a bright-field image containing the cell, obtained by imaging;
Estimated image data related to an estimated image of a stained image including a region of interest of the cell obtained by inputting the acquired image data into a detector trained by the learning data and imaging the cell without staining the cell. The distribution server distributes the output processing and the computer program for executing
When the providing server accepts a request from the user's terminal device to which the computer program was delivered, the user's terminal device is provided with cells cultured in the same culture environment as the cells used for learning the detector. The provision server generates provision information for
A provision method for providing the cells to the user based on the generated provision information .

Images