JP2019091307A - Computer program, detecting device, imaging device, server, image data, detector, detecting method, and providing method - Google Patents

Abstract

To provide a computer program, a detecting device, an imaging device, a server, image data, a detector, a detecting method, and a providing method, which can recognize fine specimens such as every single one of cells and bacteria, by using a phase-contrast microscope image or a fluorescent dyeing image of cells.SOLUTION: A computer program causes a computer to execute the steps of: processing for obtaining first image data relating to a first image which includes a specimen for learning and that is obtained by a first method; processing for obtaining second image data relating to a second image which includes an area of interest in the specimen for learning, is obtained by a second method different from the first method, and is able to recognize the area of interest within the specimen for learning; processing for generating learning data on the basis of the first image data and the second image data; and processing for prompting a detector to learn on the basis of the generated learning data in such a manner that the detector is able to detect the specimen on the basis of the first image data relating to the first image which includes the specimen and that is obtained by the first method.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to a computer program, a detection device, an imaging device, a server, image data, a detector, a detection method, and a provision method.

  There is a strong demand for an image analysis technology for automating the observation of minute samples such as cells and bacteria in order to industrialize the regenerative medicine field and streamline research. Among the image analysis techniques, the technique of detecting cells is the most basic one, but the nature of the microscope image is different from that of a general camera image, or the cells grow densely and so on. There are various technological developments.

  In Patent Document 1, an edge is extracted by applying a second-order differential filter to a phase difference image of cells acquired using a phase contrast microscope, and the one having the largest area of the circle formed by the extracted edge is suspended. Techniques for extraction as cell shapes are disclosed.

  In addition, Patent Document 2 discloses a technique for measuring the number of specific bacteria by irradiating excitation light to a microorganism that is fluorescently stained using a compound that specifically reacts with a specific bacterium, and analyzing the developed fluorescence. It is disclosed.

Unexamined-Japanese-Patent No. 2008-212017 Japanese Patent Application Publication No. 2001-340072

  However, in the phase difference image as in Patent Document 1, the difference in luminance between the cell and the background is small since the cell is often colorless and transparent in the first place, and an appropriate filter model is designed according to various cell shapes. It is very difficult to do in practice. For this reason, although the presence of cells can be recognized in the phase contrast image, it is difficult to recognize each cell.

  Further, as in Patent Document 2, when the cells are subjected to fluorescent staining to observe details, the cells are degraded or killed by the staining, and the original behavior of the cells can not be recognized. In addition, operations and costs for fluorescent staining occur.

  The present disclosure has been made in view of such circumstances, and is a computer program, a detection device, an imaging device, a server, image data, a detector, and the like, which can recognize minute samples such as individual cells and bacteria. Identify methods of detection and delivery.

  The present application includes a plurality of means for solving the above problems, and the computer program is, for example, a computer program for causing a computer to detect a sample, and the computer program is obtained by the first method. A process of acquiring first image data relating to a first image including a learning sample, and a second method different from the first method enabling identification of a region of interest in the learning sample A process of acquiring second image data relating to a second image including a region of interest of the learning sample, a process of generating learning data based on the first image data and the second image data, and the first method The detector is trained based on the generated learning data so that the detector can detect the specimen based on the obtained first image data of the first image including the specimen. To perform the processing and to.

  According to the present disclosure, it is possible to recognize minute samples such as cells and bacteria.

It is a block diagram which shows an example of a structure of the detection apparatus of this Embodiment. It is an explanatory view showing an example of a phase contrast image and a nuclear staining image. It is explanatory drawing which shows typically the 1st example of a structure of the detection model part of this Embodiment. It is explanatory drawing which shows the 1st example of the production | generation method of the learning data for making the detection model part of this Embodiment learn. It is explanatory drawing which shows the 1st example of learning by the learning process part of this Embodiment. It is a flowchart which shows the 1st example of the procedure of the learning process of the process part of this Embodiment. It is explanatory drawing which shows the 1st example of the production | generation method of the input data by the input data production | generation part of this Embodiment. It is explanatory drawing which shows the 1st example of the detection process by the detection model part of this Embodiment. It is a flowchart which shows the 1st example of the procedure of the detection process by the process part of this Embodiment. It is explanatory drawing which shows an example of the presumed nuclear-staining image produced | generated by the process part of this Embodiment. It is explanatory drawing which shows an example of the synthetic | combination image which synthesize | combined the phase difference image and the presumed nuclear stain image. It is explanatory drawing which shows the other example of a presumed nuclear-staining image and a synthesized image. It is explanatory drawing which shows typically the 2nd example of a structure of the detection model part of this Embodiment. It is explanatory drawing which shows the 2nd example of the production | generation method of the learning data for making the detection model part of this Embodiment learn. It is explanatory drawing which shows the 2nd example of learning by the learning process part of this Embodiment. It is a flowchart which shows the 2nd example of the procedure of the learning process of the process part of this Embodiment. It is explanatory drawing which shows the 2nd example of the detection process by the detection model part of this Embodiment. It is a flowchart which shows the 2nd example of the procedure of the detection process by the process part of this Embodiment. It is explanatory drawing which shows the example of the nuclear staining image after binarization obtained by binarizing a nuclear staining image by a different threshold value. It is a flowchart which shows an example of the procedure of the pre-processing of the binarization process by the process part of this Embodiment. It is explanatory drawing which shows the example of the image in the pre-processing of the binarization process by the process part of this Embodiment. It is a block diagram which shows the other example of a structure of the detection apparatus of this Embodiment. It is a block diagram which shows an example of a structure of the microscope of this Embodiment. It is a schematic diagram which shows an example of a structure of the detection system of this Embodiment. It is a schematic diagram which shows an example of the provision method of the epidermal cell of the mouse | mouth of this Embodiment. It is a flowchart which shows an example of a process sequence of the provision method of the epidermal cell of the mouse | mouth of this Embodiment.

  Hereinafter, embodiments of the present disclosure will be described based on the drawings. FIG. 1 is a block diagram showing an example of the configuration of a detection apparatus 50 according to the present embodiment. The detection apparatus of the present embodiment is an apparatus for detecting a sample. The specimens include all targets that can be identified by labeling according to the second method described later. The sample includes, for example, cells in various parts of a living body (animal) (for example, epidermal cells of a mouse) and minute identification targets such as bacteria. The second method is a method that makes it possible to identify the region of interest of the sample. By staining cell nuclei or cell membranes by the second method, cell nuclei can be identified, and as a result, cells can be detected as specimens. In this case, the region of interest is a cell nucleus. The second method also includes a method of emphasizing the cell nucleus by daringly staining the cytoplasm in order to distinguishably emphasize the cell nucleus. Further, when only the nuclei of living cells are stained by the second method, living cells can be identified, and as a result, living cells can be detected as a sample. In this case, the region of interest is a living cell. Further, the sample is not limited to a single tissue of a living organism such as a cell or a fungus, and a powder which is an aggregate of individual particles containing specific molecules or proteins can also be included as a target. The sample includes the case where one or both are distinguished when not distinguishing between live cells and dead cells. A cell nucleus (also referred to simply as "nucleus") is present in almost all cells, and the cell nucleus is one of the organelles that constitute eukaryotic cells, and genetic information is stored in the nucleus. There is. Of the protoplasm, which is the part surrounded by the cell membrane of the cell, the region other than the cell nucleus is called cytoplasm. The detection device of the present embodiment is capable of recognizably detecting each cell in a state in which a large number of cells are densely packed, and by identifying cell nuclei of individual cells, each cell can be The position and the number of cells can be accurately detected. The sample may include bacteria, solid particles, gel particles and the like. Hereinafter, in the present specification, a sample is described as a cell.

  The detection device 50 includes a processing unit 51, a first image data acquisition unit 52, a second image data acquisition unit 53, a memory 54, an output unit 55, a display unit 56, and the like. The processing unit 51 includes a detection model unit 511, an input data generation unit 512, an estimated image generation unit 513, a learning processing unit 514, a learning data generation unit 515, and the like. Also, the detection device 50 can be connected to a display device 60 capable of displaying an image. The detection device 50 can be incorporated into, for example, an information processing device such as a personal computer used by a user, a server providing various services and data, or a microscope for observing a sample.

  The first image data acquisition unit 52 has a function as a first acquisition unit and an acquisition unit, and both the cells for learning (the cells for learning of the learning processing unit 514 and the cells for detection obtained by the first method). To obtain first image data relating to the first image. The first method is a method in which it is difficult to separately and clearly recognize a region of interest (e.g., a nucleus or the like) of a cell without performing individual processing on an image. The first image obtained by the first method includes, for example, at least one of a phase difference image and a bright field image obtained by a microscope.

  The phase contrast image is an image obtained by phase contrast microscopy, and is an image in which colorless and transparent cells are visible by light and dark contrast using two properties of light diffraction and interference. When light enters the cell, the phase change of light occurs (diffracted light), while the light not affected by the cell does not change phase (direct light), so the phase difference between both lights causes the shape of the cell to be changed. Information can be obtained as light and dark contrast.

  The bright field image is an image obtained by a transmission electron microscope, and is an image formed by only a transmitted electron beam (transmission wave) among electron beams incident on a sample containing many cells. In the phase contrast image and the bright field image, since it is not necessary to stain the cells, it is possible to detect a living appearance such as cell differentiation. In the present specification, the first image obtained by the first method will be described as a phase difference image.

  The second image data acquisition unit 53 has a function as a second acquisition unit, and acquires second image data related to a second image including a learning cell, which is obtained by a second method different from the first method. . The second method is a method different from the first method, in which a region of interest (e.g., a nucleus or the like) of a cell can be separately and clearly recognized. In other words, according to the first method, it is difficult to identify the individual regions of interest of the cells individually according to the first method, while according to the second method It can be said that one can be identified individually. In the present specification, the second image obtained by the second method is described as a stained image.

  Nuclear stained images are images obtained by fluorescence microscopy. Cells are subjected to nuclear staining (also referred to as fluorescent staining) in advance, and the nuclear-stained cells are irradiated with excitation light, and a filter that transmits only fluorescence of a required wavelength among the fluorescence generated from the nuclear-stained cells is A nuclear staining image can be obtained by using only the fluorescence.

  The memory 54 is an image memory, and the image data (first image data) of the phase difference image acquired by the first image data acquisition unit 52 and the image data of the nuclear staining image acquired by the second image data acquisition unit 53 2) Store the image data).

  FIG. 2 is an explanatory view showing an example of a phase difference image and a nuclear staining image. FIG. 2A is a phase contrast image, and FIG. 2B is a nuclear staining image. The phase difference image and the nuclear staining image shown in FIG. 2 are obtained by photographing the same observation area at the same time. The resolution of the phase difference image and the nuclear staining image can be 800 × 600, but is not limited thereto. The phase contrast image can recognize the presence of cells or the outline of cell shape, but it is not possible to accurately observe whether it is a single cell or a plurality of cells are concentrated. It is difficult to distinguishably observe individual cells for all cells in the area. In the nuclear staining image, a portion of the cell nucleus appears as high brightness as compared to the background, and the position and number of the cell nucleus can be accurately observed as compared to the phase contrast image. Note that the luminance emitted from each cell nucleus is not uniform but varies depending on the cell nucleus.

  The processing unit 51 can be configured by hardware or software, or a combination of both.

  The detection model unit 511 has a function as a detector, and is configured by, for example, a plurality of processors (CPU: Central Processing Unit), a plurality of graphics processing units (GPU: Graphics Processing Unit), or a combination of CPU and GPU be able to. A detection model for detecting cells is created by a required programming language, and data for learning is used to tune the configuration of parameters of the detection model such that detection accuracy by the created detection model is enhanced. Learning of the model unit 511 can be performed.

  FIG. 3 is an explanatory view schematically showing a first example of the configuration of the detection model unit 511 of the present embodiment. FIG. 3 shows the detection model unit 511 configured as a neural network model, but the configuration of the detection model unit 511 is not limited to a multi-layered neural network (depth learning) as in the example of FIG. 3. Other algorithms of machine learning (eg, SVM: support vector machine, Random Forest: random forest) can also be used.

  As shown in FIG. 3, the detection model unit 511 includes an input layer, an output layer, and a plurality of intermediate layers. Although two intermediate layers are illustrated in FIG. 3 for the sake of convenience, the number of intermediate layers is not limited to two, and may be three or more. The number of nodes in the input layer is N (x1, x2,..., XN), and the number of nodes in the output layer is 1 (y1).

  One or more nodes (neurons) are present in the input layer, the output layer, and the intermediate layer, and the nodes of each layer are coupled to nodes present in the previous and subsequent layers in one direction with a desired weight. A vector having the same number of components as the number N of nodes in the input layer is provided as input data of the learned detection model unit 511.

  The data given to each node of the input layer is given as an input to the first intermediate layer, and the output of the intermediate layer is calculated using the weight and the activation function, and the calculated value is given to the next intermediate layer Thereafter, the output layer is sequentially transmitted to the subsequent layers (lower layer) until the output of the output layer is determined. Note that all the weights connecting the nodes are calculated by the learning algorithm of the learning processing unit 514.

  The output layer outputs the luminance value of one pixel based on the input data (x1, x2,... XN). Assuming that the luminance range of the pixel is 0 to 255, the output value can take any value (not limited to an integer) of 0 to 255. The luminance value 0 is black (dark), and the luminance value 255 is white (bright).

  Next, the learning procedure of the detection model unit 511 will be described.

  The learning data generation unit 515 uses the image data of the phase difference image including the learning cell obtained by imaging and the image data of the nuclear staining image including the region of interest of the learning cell obtained by imaging. Generate learning data based on it.

  FIG. 4 is an explanatory view showing a first example of a method of generating learning data for causing the detection model unit 511 of the present embodiment to learn. The method of generating learning data shown in FIG. 4 corresponds to the first example of the detection model unit 511 shown in FIG.

  As shown in FIG. 4A, the learning data generation unit 515 acquires, from the memory 54, image data of a nuclear stained image (for example, having a resolution of 800 × 600) including the region of interest of the learning cell in which the observation region is imaged.

  As shown in FIG. 4B, the learning data generation unit 515 identifies a plurality of pixel blocks (for example, g × g, g = 28, etc.) configured by a predetermined number of pixels from the nuclear staining image. In the example of FIG. 4B, four pixel blocks G1, G2, G3, and G4 are shown for convenience. In practice, a required number of pixel blocks can be identified from one nuclear stain image. The required number may be, for example, a number equal to the total number of pixels of one nuclear stain image, or may be a number smaller than the total number of pixels to reduce the image processing load. Although specification of a pixel block can choose a position on a nuclear stain picture at random, it can specify not only a pixel block containing a cell nucleus but a pixel block not containing a cell nucleus. By specifying a pixel block that does not contain cell nuclei, it is possible to prevent erroneous detection such that cell nuclei are erroneously present even if cell nuclei are not present, and to improve cell detection accuracy. Also, it is not necessary to distinguish whether the cell is a positive cell or a dead cell. The pixel blocks G1 and G3 contain cell nuclei (colored spots), and the pixel blocks G2 and G4 do not contain cell nuclei. Further, feature amounts of the pixel blocks G1, G2, G3 and G4 (in the example of FIG. 4B, luminance values which are one of the pixel values) are set to 159, 82, 255 and 90, respectively. Here, the feature amount of the pixel of the pixel block may be, for example, a luminance value which is one of the pixel values of the pixels present at the center of the pixel block, but is not limited thereto. In addition, a representative value representing the pixel block can be included, such as an average value of pixel values in the pixel block, a median value of the pixel values, and the like. That is, the representative value is also one of the pixel values.

  The learning data generation unit 515 acquires, from the memory 54, image data of a phase difference image (for example, having a resolution of 800 × 600) including the learning cells, in which the observation area is imaged. The phase difference image and the nuclear staining image are obtained by imaging the same observation area including the learning cells at the same time.

  The learning data generation unit 515 generates learning data by associating the feature amount of the pixel of the pixel block identified on the nuclear stain image with the pixel block of the phase difference image corresponding to the pixel block identified on the nuclear stain image. More specifically, the learning data generation unit 515 uses the pixel block of the phase difference image as one of the pixel values of a predetermined pixel (the center pixel in the example of FIG. 4) in the pixel block specified on the nuclear stain image. To generate learning data in association with the luminance values. Since pixel values are used as learning data (teacher data), for example, there is no need to design a suitable feature vector as compared to the case where a feature vector or the like is calculated and used as teacher data, and a feature vector is calculated. In comparison, since the pixel value can be used as it is, processing is simplified and highly accurate cell detection is enabled.

  As shown in FIG. 4C, the learning data generation unit 515 cuts out regions X1, X2, X3, and X4 (cut-out images) of the phase difference image corresponding to the pixel blocks G1, G2, G3, and G4 on the nuclear staining image. The luminance values of the pixel blocks G1 to G4 are associated with the regions X1 to X4 as teacher labels. Regions X1 to X4 are each composed of g × g pixels. Therefore, for region X1, a vector (x11, x12, N) has N (= g × g) components as learning input data of sample X1. ..., x1N) are generated, and the teacher label is 159.

  For the region X2, a vector (x21, x22,..., X2N) having N (= g × g) components is generated as learning input data of the sample X2, and the teacher label is 82. For the region X3, a vector (x31, x32,..., X3N) having N (= g × g) components is generated as learning input data of the sample X3, and the teacher label is 255. Also, for the region X4, a vector (x41, x42,..., X4N) having N (= g × g) components is generated as learning input data of the sample X4, and the teacher label is 90.

  FIG. 5 is an explanatory view showing a first example of learning by the learning processing unit 514 of the present embodiment. The learning processing unit 514 causes the detection model unit 511 to learn based on the learning data generated by the learning data generation unit 515. Specifically, the learning processing unit 514 inputs learning input data (x11, x12,..., X1N) of the sample X1 to each input node of the input layer, and the output value y1 of the output node of the output layer is The learning is performed to approach the teacher data 159.

  Further, the learning processing unit 514 inputs learning input data (x21, x22,..., X2N) of the sample X2 to each input node of the input layer, and the output value y1 of the output node of the output layer is teacher data. Learn to approach 82. Further, the learning processing unit 514 inputs learning input data (x31, x32,..., X3N) of the sample X3 to each input node of the input layer, and the output value y1 of the output node of the output layer is teacher data. It learns to approach a certain 255. Further, the learning processing unit 514 inputs learning input data (x41, x42,..., X4N) of the sample X4 to each input node of the input layer, and the output value y1 of the output node of the output layer is teacher data. Learn to approach 90.

  FIG. 6 is a flowchart showing a first example of the procedure of the learning process of the processing unit 51 according to the present embodiment. Hereinafter, for the sake of convenience, the subject of the process will be described as the processing unit 51. The processing unit 51 acquires the image data of the phase difference image including the cells for learning, which images the observation area (S11), and images the same observation area at the same time, the nucleus including the region of interest of the cells for learning Image data of a stained image is acquired (S12).

  The processing unit 51 specifies a plurality of pixel blocks at random from the nuclear staining image (S13), and records the luminance value of the central pixel of the specified pixel block (S14). The processing unit 51 extracts the cutout image of the position corresponding to the pixel block from the phase difference image (S15).

  The processing unit 51 generates learning data with the luminance value of each pixel of the extracted cutout image as learning input data and the luminance value of the central pixel of the pixel block corresponding to the cutout image as a teacher label (S16). The processing unit 51 causes the detection model unit 511 to learn based on the generated learning data (S17).

  The processing unit 51 determines whether the accuracy of the output value of the detection model unit 511 is within the allowable range (S18), and if it is not within the allowable range (NO in S18), the processing from step S11 is continued. If the accuracy of the output value is within the allowable range (YES in S18), the processing unit 51 ends the process.

  In the case of generating learning data, the phase difference image and the nuclear staining image are not limited to one, and a plurality of phase difference images and a nuclear staining image can be used. For example, in order to learn a state according to the activity of a cell, it is possible to capture a phase difference image and a nuclear staining image each time a predetermined time has elapsed. For example, every 15 minutes, one phase difference image and one nuclear staining image may be taken, and finally about 10 phase difference images and a nuclear staining image may be acquired.

  Next, a procedure for detecting a cell using the learned detection model unit 511 will be described.

  The first image data acquisition unit 52 acquires a phase difference image including cells for detection (not for learning), and stores the acquired phase difference image in the memory 54.

  The input data generation unit 512 acquires a phase difference image from the memory 54, and specifies a plurality of pixel blocks configured with a predetermined number of pixels from the acquired phase difference image. The input data generation unit 512 generates input data based on the luminance value of each pixel of the specified pixel block.

  FIG. 7 is an explanatory drawing showing a first example of a method of generating input data by the input data generation unit 512 of the present embodiment. The resolution of the phase difference image is m × n (for example, m = 800, n = 600, etc.). For example, the input data generation unit 512 scans in the horizontal direction from the upper left of the phase difference image, shifts the scanning position in the vertical direction when scanning in the horizontal direction is completed, and repeats scanning from left to right again. By scanning to the lower right and specifying pixel blocks (g × g, g = 28) for the number of pixels of all pixels of the phase difference image, M (M = m × n) cutout images A1 and A2 are obtained. Cut out AM. Each cutout image is composed of N (N = g × g) pixels. Thus, the input data is composed of M vectors each having N components. The number of cut-out images is not limited to the total number of pixels of the phase difference image, but may be the number of pixels when a specific region of the phase difference image is scanned, and the image to be cut out is along the scanning direction It may be thinned out.

  FIG. 8 is an explanatory view showing a first example of detection processing by the detection model unit 511 of the present embodiment. When input data (a11, a12, a13,... A1N) for the cutout image A1 is input to each input node of the input layer, an output value b1 is output to the output node of the output layer. Further, when input data (a21, a22, a23,... A2N) for the cutout image A2 is input, an output value b2 is output to the output node of the output layer. Hereinafter, similarly, when input data (aM1, aM2, aM3,... AMN) for the cutout image AM is input, the output value bM is output to the output node of the output layer.

  The estimated image generation unit 513 generates an estimated image of the nuclear staining image based on the output values b1, b2,..., BM output from the output node of the detection model unit 511 without capturing the nuclear staining image. As illustrated in FIG. 7, assuming that the resolution of the phase difference image is m × n, the estimated image generation unit 513 associates M (= m × n) output values with m × n pixels. , M × n estimated nuclear staining images.

  The output unit 55 can output the output values b1, b2,..., BM output from the output node of the detection model unit 511 to an external device or the like.

  FIG. 9 is a flowchart showing a first example of the procedure of detection processing by the processing unit 51 of the present embodiment. The processing unit 51 acquires image data of a phase difference image including cells to be detected (S31), scans the acquired phase difference image, and extracts a cutout image (S32). The processing unit 51 generates input data based on the luminance value of each pixel of the extracted cutout image (S33).

  The processing unit 51 inputs the generated input data to the input node of the detection model unit 511 (S34), and sets the output value output by the detection model unit 511 for each cutout image as the luminance value of each pixel of the estimated nuclear staining image ( S35) An estimated nuclear staining image is generated (S36).

  The processing unit 51 contrasts and displays the phase difference image and the estimated nuclear staining image (S37), and ends the processing.

  FIG. 10 is an explanatory view showing an example of an estimated nuclear staining image generated by the processing unit 51 of the present embodiment. FIG. 10A is a phase difference image including cells to be detected, FIG. 10B is an estimated image (estimated nuclear stain image) of a nuclear staining image generated by the processing unit 51, and FIG. 10C is a phase contrast image. It is a corresponding nuclear staining image, and the same observation area as the phase difference image is captured at the same time. As shown in FIG. 10, according to the present embodiment, it is possible to obtain an estimated image of a nuclear staining image with high accuracy only by acquiring a phase contrast image without performing nuclear staining on cells and acquiring a nuclear staining image. be able to.

  Image data related to the estimated nuclear staining image shown in FIG. 10 is image data in which a cell (cell nucleus) is detected, and estimated image data related to an estimated image of a stained image including a region of interest of cells obtained by imaging A detection model for detecting cells, input data based on at least one of a phase-contrast image containing cells and a bright-field image, obtained by imaging, The detection model unit 511 is generated on the basis of an output value output from the detection model unit 511 by inputting to the unit 511, and the detection model unit 511 obtains at least one of a phase difference image including learning cells or a bright field image obtained by imaging. The learning is performed by learning data generated based on the first image data and the second image data related to the stained image including the region of interest of the learning cell obtained by imaging.

  FIG. 11 is an explanatory view showing an example of a combined image obtained by combining the phase difference image and the estimated nuclear staining image. The display unit 56 can display the estimated nuclear stain image generated by the estimated image generation unit 513 and the acquired phase difference image on the display screen of the display device 60 in comparison. Although the phase difference image (FIG. 11A) and the estimated nuclear staining image (FIG. 11B) are not shown in FIG. 11C, the phase difference image and the estimated nuclear staining image are not only displayed side by side but also displayed in FIG. As described above, overlapping the phase difference image with the estimated nuclear stain image such that the upper left position and the lower right position of the phase difference image respectively coincide with the upper left position and the lower right position of the estimated nuclear stain image And editing the phase difference image by a predetermined operation based on the information of the estimated nuclear staining image. Thereby, since the position of the cell recognized on the phase difference image is determined by the position of the cell nucleus recognized on the estimated nuclear staining image, the position of each cell, without acquiring the nuclear staining image, It is possible to observe the phase difference image while accurately recognizing the number of cells and the number of cells. In particular, in the case of being recognized as one cell, it can be grasped and observed that, for example, two cells are in close contact with each other, in the case of being recognized as one cell, in particular.

  According to the present embodiment, it is possible to obtain an estimated nuclear staining image in which a cell nucleus is clearly seen without actually acquiring a nuclear staining image, so that an image suitable for image analysis can be obtained. In addition, the operation and cost for nuclear staining imaging become unnecessary, there is no invasiveness to the cells, and it becomes possible to observe the original behavior of the cells.

  According to the present embodiment, since teacher data (data for learning) is automatically generated based on a nuclear staining image, for example, a mask image (teacher data) of a region (cell region) to be detected is manually created. There is no need, there is no room for the subjectivity of the creator of the teacher data to enter, and it is possible to generate learning data without variation or deviation due to subjectivity.

  According to this embodiment, once learning is completed, an image equivalent to a nuclear staining image can be obtained without performing nuclear staining of cells, so various risks due to nuclear staining (for example, damage to cells, etc.) There is no

  According to the present embodiment, since the image data of the phase difference image is input to the detection model unit learned by the learning data based on the nuclear staining image to detect the cells, it is necessary when only the phase difference image is used There is no need to design various and appropriate filter models.

  According to the present embodiment, since it is not necessary to estimate whether each pixel is a cell or not based on the optical system model, it is not necessary to determine many unknown parameters of the optical system model by trial and error.

  According to the present embodiment, when identifying a plurality of pixel blocks from a nuclear staining image, not only pixel blocks including cell nuclei but also pixel blocks not including cell nuclei are identified to generate learning data. The false detection can be prevented and the detection accuracy of the cell can be enhanced as the cell nucleus is erroneously present even though it is not present.

  FIG. 12 is an explanatory view showing another example of the estimated nuclear staining image and the composite image. In FIG. 12, only one cell (cell nucleus) is illustrated for convenience. The first image data acquisition unit 52 acquires a plurality of phase difference images captured in time series and including cells to be detected. For example, one phase difference image obtained by imaging the observation region can be acquired each time 10 minutes or 20 minutes have elapsed. The input data generation unit 512 extracts (specifies) a plurality of cutout images each having a predetermined number of pixels (for example, g × g) from each of the plurality of phase difference images. The input data generation unit 512 generates a plurality of sets of input data for each of the plurality of phase difference images, based on the luminance value of each pixel of the plurality of cutout images. The plurality of sets of input data correspond to a plurality of phase difference images.

  Since the estimated image generation unit 513 generates one estimated nuclear staining image based on a single phase difference image, the time-series estimated nuclei of the same number as the number of phase difference images are generated for a plurality of phase difference images. Stained images can be generated.

  The display unit 56 can display the estimated movement trajectory of the cell (cell nucleus) based on the output value output from the detection model unit 511 to which a plurality of sets of input data is input. That is, in the estimated nuclear staining image of FIG. 12A, the range from the past estimated position of the cell nucleus to the current (most recent) estimated position is represented as the estimated movement trajectory. Further, in the synthetic image of FIG. 12B, estimated moving trajectories represent from the past estimated position of both the cell and the cell nucleus in the cell to the current (most recent) estimated position. This enables accurate understanding of the dynamic behavior of the cells.

  In the above-mentioned example, although detection device 50 was composition provided with one detection model part 511, it is not limited to this, A composition provided with a plurality of detection model parts may be sufficient. In this case, for example, depending on the type of the region of interest of the cell to be detected, etc., a plurality of detection model units may be provided which have different detection models for each type. Thereby, an optimal detection model unit can be used according to the type of the region of interest of cells, and the detection accuracy of cells can be further enhanced.

  Further, the detection model unit 511 can be configured to be capable of relearning. The learning processing unit 514 updates and detects a plurality of parameters (weights between the input layer and the intermediate layer, weights between the intermediate layer and the output layer, weights between the intermediate layers, etc.) that the detection model unit 511 has, The model unit 511 can be made re-learnable. Thereby, the detection accuracy of cells can be maintained.

  Next, another configuration of the detection model unit 511 will be described.

  FIG. 13 is an explanatory view schematically showing a second example of the configuration of the detection model unit 511 of the present embodiment. The difference with the first example illustrated in FIG. 3 is that the number of nodes in the output layer is 2 (y1, y2). The number of input nodes in the input layer and the configuration of the middle layer are the same as in the first example. The output layer outputs the probability that the luminance value is 0 (black) based on the input data (x1, x2,... XN), and the node 1 has a probability that the luminance value is 255 (white) Output The output value may take any value between 0 and 1 (not limited to an integer).

  FIG. 14 is an explanatory view showing a second example of a method of generating learning data for causing the detection model unit 511 of the present embodiment to learn. The method of generating learning data in FIG. 14 corresponds to the second example of the detection model unit 511 shown in FIG.

  As shown in FIG. 14A, the learning data generation unit 515 acquires, from the memory 54, image data of a nuclear stained image (for example, having a resolution of 800 × 600) including the region of interest of the learning cell, in which the observation region is imaged.

  As shown in FIG. 14B, the learning data generation unit 515 performs a binarization process on the nuclear staining image to generate a binarized image. By binarizing, for example, the color development position can be made into the foreground. In addition, binarization makes it possible to clearly distinguish between the region of interest of the cell and the region other than the region of interest. For example, by binarizing, the distinction between the location corresponding to the cell nucleus and the location not corresponding to the cell nucleus becomes clear, and the detection model unit 511 can be made easy to learn.

  As shown in FIG. 14C, the learning data generation unit 515 identifies a plurality of pixel blocks (for example, g × g, g = 28, etc.) configured with a predetermined number of pixels from the nuclear staining image. In the example of FIG. 14C, four pixel blocks G6, G7, G8 and G9 are shown for convenience. In practice, a required number of pixel blocks can be identified from one nuclear stain image. The required number may be, for example, a number equal to the total number of pixels of one nuclear stain image, or may be a number smaller than the total number of pixels to reduce the image processing load. Although specification of a pixel block can choose a position on a nuclear stain picture at random, it can specify not only a pixel block containing a cell nucleus but a pixel block not containing a cell nucleus. By specifying a pixel block that does not contain cell nuclei, it is possible to prevent erroneous detection such that cell nuclei are erroneously present even if cell nuclei are not present, and to improve cell detection accuracy. In addition, it is not necessary to distinguish whether the cell is a positive cell or a dead cell. The pixel blocks G6 and G8 contain cell nuclei (colored spots), and the pixel blocks G7 and G9 do not contain cell nuclei. Further, the feature quantities of the pixel blocks G6, G7, G8, and G9 are binary, and are set to 1, 0, 1, 0, respectively. Binary values are pixel values of pixels. The binary may be, for example, a binary of a pixel present at the center of the pixel block, but is not limited thereto.

  The learning data generation unit 515 acquires, from the memory 54, image data of a phase difference image (for example, having a resolution of 800 × 600) including the learning cells, in which the observation area is imaged. The phase difference image and the nuclear staining image are obtained by imaging the same observation area of the learning cell at the same time.

  The learning data generation unit 515 sets the pixel block of the phase difference image corresponding to the pixel block specified on the nuclear stain image to the pixel value of the binarized predetermined pixel of the pixel block specified on the nuclear stain image. A value (0 or 1) is associated to generate learning data. More specifically, the learning data generation unit 515 generates learning data by associating the pixel block of the phase difference image with the binary value which is the pixel value of the central pixel in the pixel block identified on the nuclear stain image. Since binary values that are pixel values are used as learning data (teacher data), for example, there is no need to design a suitable feature vector as compared to the case where a feature vector or the like is calculated and used as teacher data. Since pixel values can be used as they are, calculation can be simplified, and highly accurate cell detection can be performed.

  As shown in FIG. 14D, the learning data generation unit 515 cuts out regions X6, X7, X8, X9 (cut-out images) of the phase difference image corresponding to the pixel blocks G6, G7, G8, G9 on the nuclear staining image. The binary values of the pixel blocks G6 to G9 are associated with the regions X6 to X9 as teacher labels. Regions X6 to X9 are each composed of g × g pixels. Therefore, for region X6, a vector (x61, x62, N) having N (= g × g) components as learning input data of sample X1. ..., x 6 N) are generated, and the teacher label is 1.

  For the region X7, a vector (x71, x72,..., X7N) having N (= g × g) components is generated as learning input data of the sample X7, and the teacher label is zero. For the region X8, a vector (x81, x82,..., X8N) having N (= g × g) components is generated as learning input data of the sample X8, and the teacher label is 1. Further, for the region X9, a vector (x91, x92,..., X9N) having N (= g × g) components is generated as learning input data of the sample X9, and the teacher label is zero.

  FIG. 15 is an explanatory view showing a second example of learning by the learning processing unit 514 of the present embodiment. The learning processing unit 514 causes the detection model unit 511 to learn based on the learning data generated by the learning data generation unit 515. Specifically, the learning processing unit 514 inputs learning input data (x61, x62,..., X6N) of the sample X6 of which the teacher label is 1 to each input node of the input layer, and outputs the output node of the output layer The learning is performed so that the output value y2 of 1 approaches 1 and the output value y1 of the output node 0 approaches 0.

  Although not shown, learning input data (x71, x72,..., X7N) of sample X7 having a teacher label of 0 is input to each input node of the input layer, and the output of output node 0 of the output layer The learning is performed so that the value y1 approaches 1 and the output value y2 of the output node 1 approaches 0. The same applies to learning input data of the samples X8 and X9.

  That is, when the input data of the teacher label 1 is input, learning is performed so that the node 0 approaches 0 and the node 1 approaches 1 and when the input data of the teacher label 0 is input, the node 0 The learning is performed so that the node 1 approaches 0 to 1 and 0.

  FIG. 16 is a flowchart showing a second example of the procedure of the learning process of the processing unit 51 of the present embodiment. Hereinafter, for the sake of convenience, the subject of the process will be described as the processing unit 51. The processing unit 51 acquires the image data of the phase difference image including the cells for learning, which images the observation area (S51), and images the same observation area at the same time, including the region of interest of the cells for learning Image data of a stained image is acquired (S52).

  The processing unit 51 binarizes the nuclear staining image (S53), randomly identifies a plurality of pixel blocks from the binarized nuclear staining image (S54), and records the binary value of the central pixel of the identified pixel block (S53) S55). The processing unit 51 extracts the cutout image of the position corresponding to the pixel block from the phase difference image (S56).

  The processing unit 51 generates learning data with the luminance value of each pixel of the extracted clipped image as learning input data and the binary value of the central pixel of the pixel block corresponding to the clipped image as a teacher label (S57). The processing unit 51 causes the detection model unit 511 to learn based on the generated learning data (S58).

  The processing unit 51 determines whether the accuracy of the output value of the detection model unit 511 is within the allowable range (S59), and if it is not within the allowable range (NO in S59), the processing from step S51 is continued. If the accuracy of the output value is within the allowable range (YES in S59), the processing unit 51 ends the process.

  The method of generating input data in the second example is the same as that of the first example illustrated in FIG. 7, and M (M = m × n) cutout images A1, A2,..., AM are cut out from the phase difference image. .

  FIG. 17 is an explanatory view showing a second example of detection processing by the detection model unit 511 of the present embodiment. When input data (a11, a12, a13,... A1N) for the cutout image A1 is input to each input node of the input layer, output data B1 is output to an output node of the output layer. In the output data B1, the probability of the output node 0 is 0.4, and the probability of the output node 1 is 0.1. If the probability of the output node 0 is equal to or higher than the probability of the output node 1, the output value is set to 0. When the probability of the output node 0 is less than the probability of the output node 1, the output value is set to 1. As for the output data B1, since the probability of the output node 0 is equal to or higher than the probability of the output node 1, the output value b1 is zero.

  When input data (a21, a22, a23,... A2N) for the cutout image A2 is input, output data B2 is output to the output node of the output layer. In the output data B2, the probability of the output node 0 is 0.3, and the probability of the output node 1 is 0.8. As for the output data B2, since the probability of the output node 0 is less than the probability of the output node 1, the output value b2 is 1.

  Similarly, when input data (aM1, aM2, aM3,... AMN) for the cutout image AM is input, the output data BM is output to the output node of the output layer. In the output data BM, the probability of the output node 0 is 0.5, and the probability of the output node 1 is 0.2. As for the output data BM, since the probability of the output node 0 is equal to or higher than the probability of the output node 1, the output value bM is zero.

  The estimated image generation unit 513 generates an estimated image of the nuclear staining image based on the output values b1, b2,..., BM output from the output node of the detection model unit 511 without capturing the nuclear staining image. As illustrated in FIG. 7, assuming that the resolution of the phase difference image is m × n, the estimated image generation unit 513 associates M (= m × n) output values with m × n pixels. , M × n estimated nuclear staining images.

  FIG. 18 is a flowchart showing a second example of the procedure of detection processing by the processing unit 51 of the present embodiment. The processing unit 51 acquires image data of a phase difference image including cells to be detected (S71), and scans the acquired phase difference image to extract a cutout image (S72). The processing unit 51 generates input data based on the luminance value of each pixel of the extracted cutout image (S73).

  The processing unit 51 inputs the generated input data to the input node of the detection model unit 511 (S74), and the value of node 0 (probability) output by the detection model unit 511 for each cutout image 値 the value of node 1 (probability) In this case, the output value is 0 (black), and if the value of node 0 (probability) <the value of node 1 (probability), the output value is 1 (white), and the pixel value (brightness value) of the estimated nuclear staining image (S75).

  The processing unit 51 generates an estimated nuclear staining image (S76), contrasts and displays the phase difference image and the estimated nuclear staining image (S77), and ends the processing.

  The processing unit 51 and the display unit 56 in the present embodiment can be configured by a CPU, a GPU, a ROM, a RAM, a recording medium reading unit, and the like. The computer program recorded on the recording medium can be read by the recording medium reading unit and stored in the RAM. The processing performed by the processing unit 51 and the display unit 56 can be executed by causing the CPU and the GPU to execute the computer program stored in the RAM. Note that the CPU is a generic term of a processor, a processor device, a processing unit, and the like, and can be configured of one or more. The computer program can also be downloaded via a network such as the Internet, instead of the configuration in which the recording medium reading unit reads it.

  In the second example shown in FIG. 14 and FIG. 16 etc. described above, the configuration for performing the binarization processing on the nuclear staining image is described, but in the following, the pre-processing performed before the binarization processing will be described.

  FIG. 19 is an explanatory view showing an example of a nuclear staining image after binarization obtained by binarizing a nuclear staining image with different threshold values. FIG. 19A shows a nuclear staining image (original image before binarization) obtained by imaging. When nuclear staining is performed on cells, the region of interest (for example, the cell nucleus) of the cells does not necessarily emit light at the same intensity, so differences in coloring between cells or unevenness in brightness of the entire image as shown in FIG. 19A. Exists.

  As shown in FIG. 19B, when the threshold value: 140 is binarized, pixels with relatively high luminance in the non-cell region in the image become foreground as if color development positions, and non-cell locations are erroneously treated as cells. It will be

  Further, as shown in FIG. 19C, when binarizing with the threshold value: 150, it is possible to somewhat suppress that a region which is not a cell in the image is erroneously treated as a cell, but it is colored with a weak intensity Cell nuclei are removed, and there are cases where cell nuclei can not be detected.

  Further, as shown in FIG. 19D, when binarizing with the threshold value: 160, it is possible to prevent that a region which is not a cell in the image is mistakenly treated as a cell, but it is colored with a slightly higher intensity Even some cell nuclei may be removed. Thus, there may be cases where it is not possible to accurately detect all cell nuclei, whatever threshold is used.

  FIG. 20 is a flow chart showing an example of a pre-processing procedure of the binarization processing by the processing unit 51 of the present embodiment. The process shown in FIG. 20 includes the process to be added between step S52 and step S53 shown in FIG.

  The processing unit 51 acquires image data of a nuclear staining image (S101). The processing unit 51 generates an intermediate image with the number of pixels where the luminance value of the peripheral pixel of the target pixel of the nuclear staining image is smaller than the luminance value of the target pixel as the value of the target pixel (S102). The pixel of interest scans in the horizontal direction from the upper left of the nuclear staining image, shifts the scanning position in the vertical direction when scanning in the horizontal direction is finished, repeats scanning from left to right again, and finally scans to the lower right . The peripheral pixels of the target pixel may be, for example, 21 × 21 pixels including the target pixel, but the number of peripheral pixels is not limited to this.

  In the generated intermediate image, a pixel having a large pixel value represents that the luminance around the target pixel of the nuclear staining image is low, and a pixel having a small pixel value has a high luminance around the target pixel of the nuclear staining image Represents That is, it can be said that a pixel having a small pixel value of the intermediate image includes a pixel having a high relative value of the luminance value, and corresponds to a cell nucleus having a relatively high light emission intensity than the surrounding.

  The processing unit 51 Fourier-transforms the intermediate image to calculate a frequency spectrum (S103), and removes high frequency components using a low pass filter (S104). As a result, it is possible to remove fine pattern parts or edge parts where the luminance value changes rapidly, etc., and to remove parts in the nuclear staining image that show a change different from the change in luminance value in cell nuclei. it can.

  The processing unit 51 generates a preprocessed nuclear stain image by inverse Fourier transform (S105), binarizes the nuclear stain image (S106), and ends the process. The nuclear staining image referred to in steps S105 and S106 is strictly an intermediate image corresponding to the nuclear staining image, but in the present specification, it will be described as a nuclear staining image for convenience.

  FIG. 21 is an explanatory view showing an example of an image in preprocessing of the binarization processing by the processing unit 51 of the present embodiment. FIG. 21A shows a nuclear staining image which is an original image. FIG. 21B shows an intermediate image in which the number of pixels having a luminance value lower than that of the pixel of interest within the range (for example, 21 × 21 pixels) of the peripheral pixels of the pixel of interest is the value of the pixel of interest. FIG. 21C shows a frequency spectrum after Fourier transform. In FIG. 21C, the vicinity of the center is a low frequency component, and the farther from the center, the higher the frequency.

  FIG. 21D shows a frequency spectrum in which high frequency components are removed by a low pass filter. In the example of FIG. 21D, except the circular shape near the center is removed as the high frequency component. FIG. 21E shows an intermediate image after inverse Fourier transform. FIG. 21F shows an image after binarization. Comparing FIG. 21A with FIG. 21F, it can be seen that in FIG. 21A, cell nuclei having different emission intensities (different luminance values) appear as the foreground.

  As described above, the processing unit 51 acquires the second image. The processing unit 51 compares the luminance value of the peripheral pixel (for example, 21 × 21 pixels including the target pixel) of the target pixel of the acquired second image with the luminance value of the target pixel, and the pixel having the luminance value smaller than that of the target pixel. The intermediate image is generated by calculating the number of pixels of and the calculated number of pixels as the pixel value of the target pixel. The processing unit 51 performs Fourier transform on the intermediate image, and removes high frequency components in the frequency spectrum after the Fourier transform. The processing unit 51 performs inverse Fourier transform on the frequency spectrum from which high frequency components have been removed to generate an intermediate image (nuclear staining image). The processing unit 51 binarizes the generated intermediate image.

  In addition, as another method of obtaining a binarized image, for example, the learning data generation unit 515 does not use cells as a method of background subtraction with a background image of brightness unevenness without cells (as a method of obtaining a background image, A binarized image can be generated by photographing a state, obtaining an image having a median value in a time-series direction from a time-lapse image, or the like, or using an adaptive binarization process or the like. The adaptive binarization process is a method of setting a threshold locally for each region configured of a plurality of pixels without fixing the threshold for binarization. For example, an average value of pixel values of pixels in a certain area is set as a threshold value of the area.

  FIG. 22 is a block diagram showing another example of the configuration of the detection apparatus 50 of the present embodiment. The difference from the configuration illustrated in FIG. 1 is that the learning processing unit 514, the learning data generation unit 515, and the second image data acquisition unit 53 are not provided in the example of FIG. That is, in the detection apparatus 50 shown in FIG. 22, when the detection model unit 511 has already learned and does not need further learning, the learning processing unit 514, the learning data generation unit 515, and the second image data acquisition The part 53 is not an essential component. In addition, in the detection apparatus 50 illustrated in FIG. 22, since the other part is the same as that of the structure of FIG. 1, description is abbreviate | omitted.

  FIG. 23 is a block diagram showing an example of the configuration of the microscope 100 according to the present embodiment. Below, microscope 100 is explained as an example of an imaging device of this embodiment. The microscope 100 includes the detection device 50 illustrated in FIG. 1 or FIG. The detection device 50 is not limited to the configuration incorporated into the microscope 100.

  In addition to the detection device 50, the microscope 100 is a white light source 11 that emits a white color when capturing a phase difference image, and a well plate in which cells C to which a nuclear stain agent that generates fluorescence by excitation light is accommodated. 10, an objective lens 13, a light source 12 for fluorescence excitation, a mirror unit 14 for fluorescence disposed below the well plate 10, an imaging device 15, an AD converter for converting analog values output from the imaging device 15 into digital values D) 16, the CPU 20, the memory 23, the operation unit 22 for the user to operate the microscope 100, the light source 12 for fluorescence excitation, the objective lens 13, the mirror unit 14 for fluorescence, the imaging device 15 etc. A drive control unit 21 for moving in the direction is provided.

  When imaging a phase difference image, the white light from the white light source 11 is emitted downward from above the well plate 10, and the light transmitted through the cell C (diffracted light) and the light not transmitted through the cell C (directly Due to the phase difference with light, the shape information of the cell can be obtained by the imaging device 15 as the contrast of light and dark. The microscope 100 can also capture a bright field image instead of or in addition to the phase difference image.

  When imaging a nuclear staining image, the excitation light emitted from the fluorescence excitation light source 12 is reflected by the fluorescence mirror unit 14 toward the well plate 10 and enters the cell C through the objective lens 13. Fluorescent light is emitted from the cell C by the incident excitation light. The fluorescence emitted from the cell C is transmitted through the objective lens 13 and the mirror unit 14 for fluorescence and is incident on the imaging device 15, whereby a nuclear staining image can be obtained.

  The switching between the imaging of the phase difference image and the imaging of the nuclear staining image can be performed at the same time (observation region) under the control of the CPU 20. In addition, according to the operation from the operation unit 22, for example, it is possible to set an imaging location and imaging timing and to automatically perform imaging.

  FIG. 24 is a schematic view showing an example of the configuration of a detection system according to the present embodiment. The detection system can include, for example, a microscope 100, a terminal device 200 used by a user, a server 300 that provides an application or a database, and the like. The terminal device 200 is connected to a display device 60 capable of displaying an image. The display device 60 can also be incorporated into the terminal device 200. In the detection system shown in FIG. 24, the detection device 50 of the present embodiment can be incorporated into any one of the microscope 100, the terminal device 200, and the server 300. In the example of FIG. 23 described above, the example in which the detection device 50 is incorporated in the microscope 100 has been described.

  The microscope 100 and the terminal device 200 can be connected, for example, via a communication line. The terminal device 200 and the server 300 can be connected via the communication network 1 such as the Internet.

  When the detection device 50 is incorporated in the server 300, the server 300 includes a detection model unit 511 that detects cells, an acquisition unit that acquires image data related to at least one of a phase difference image including cells and a bright field image, and an acquired image Data is input to a detection model unit 511, and an output unit or the like is provided which outputs estimated image data related to an estimated nuclear staining image including a region of interest of cells without staining the cells.

  In the above-mentioned embodiment, the nuclear staining image was described as an example of the second image obtained by the second method. Nuclear staining images include not only staining of cell nuclei but also images capable of recognizing organelles in cells. In addition, the second image is not limited to fluorescence imaging such as nuclear staining image using a staining agent etc. For example, it is obtained by a method by RI (radioisotope) imaging using a contrast agent etc. Images are also included.

  Next, a method for providing mouse epidermal cells will be described. In addition, although the epidermal cell of a mouse | mouth is demonstrated below as a cell to provide, the cell to provide is not limited to the epidermal cell of a mouse | mouth.

  FIG. 25 is a schematic view showing an example of the method for providing epidermal cells of the mouse of the present embodiment. As shown in FIG. 25, a distribution server 310, a provision server 320, a terminal device 201 used by a user, a tablet terminal 202, and the like are connected via a communication network. The tablet terminal 202 may be a smartphone.

  The transmitting unit 311 of the distribution server 310 distributes the computer program of the present embodiment to the terminal device 201 of a specific user. The terminal device 201 can install the distributed computer program, whereby the terminal device 201 has a configuration in which the detection device 50 of the present embodiment is incorporated. The specific user is, for example, a user of a company, a research institution, a medical institution or the like who analyzes cells using the computer program of the present embodiment, and can be a user registered in advance.

  The user requests the provision server 320 to provide the epidermal cells of the mouse from the tablet terminal 202. The request can include an ID for authenticating the user, a password, and the like.

  When the reception unit 321 of the providing server 320 receives a request from the user, the authentication unit 322 performs an authentication process as to whether the user is a legitimate user. As a result of the authentication process, if the request is from a legitimate user, the information generation unit 323 generates provided information for providing the user with the epidermal cells of the mouse. The provided information for providing mouse epidermal cells includes, for example, destination information for transporting mouse epidermal cells and the like.

  As a result, mouse epidermal cells can be provided with certainty to a specific user to whom the computer program of the present embodiment has been distributed. Further, by providing cells cultured in the same culture environment as the cells used for learning together with the computer program, it is possible to secure the detection accuracy of the cells by the computer program.

  The distribution server 310 and the providing server 320 can be integrated and configured as one server (for example, the providing server 320).

  FIG. 26 is a flow chart showing an example of the processing procedure of the method for providing epidermal cells of the mouse of the present embodiment. Hereinafter, for convenience, the subject of the process will be described as the providing server 320. The providing server 320 distributes the program (computer program) of the present embodiment to a specific user (S91). The provision server 320 determines whether or not there is a request for provision of epidermal cells of the mouse from the user (S92), and if there is no request (NO in S92), the process of step S92 is continued.

  If there is a request (YES in S92), the provider server 320 performs user authentication processing (S93), and determines whether or not the user is a legitimate user (S94). If the user is a legitimate user (YES in S94), the providing server 320 generates the provided information for providing the user with the epidermal cells of the mouse (S95), and ends the process. If the user is not a legitimate user (NO in S94), the provider server 320 ends the process.

  The computer program according to the present embodiment is a computer program for causing a computer to detect a sample, and the first image data relating to a first image including a learning sample obtained by the first method on a computer A second image relating to a second image including a region of interest of the learning sample, obtained by a second method different from the first method and capable of identifying the region of interest in the learning sample; A process of acquiring image data, a process of generating learning data based on the first image data and the second image data, and first image data of a first image including a specimen obtained by the first method And a process of making the detector learn based on the generated learning data so that the detector can detect the sample based on

  The computer program according to the present embodiment includes, in a computer, a process of specifying a plurality of pixel blocks including a predetermined number of pixels from the second image, and a pixel block of the first image corresponding to the specified pixel block. And processing for generating the learning data by associating feature amounts of pixels of the specified pixel block.

  The computer program according to the present embodiment causes the computer to execute the process of generating the learning data by associating the pixel block of the first image with the pixel value of the predetermined pixel in the specified pixel block.

  The computer program according to the present embodiment corresponds to the computer processing of binarizing the second image and binary values which are pixel values of pixels obtained by the binarizing processing. Any of a process of specifying a plurality of pixel blocks including the number of pixels, and a pixel value of a binarized predetermined pixel of the specified pixel block in the pixel block of the first image corresponding to the specified pixel block Are associated to execute the process of generating the learning data.

  The computer program according to the present embodiment causes the computer to execute processing of displaying the estimated position of the sample based on the result of learning the detector.

  The computer program according to the present embodiment includes, in a computer, processing of acquiring first image data related to a first image including a sample obtained by the first method, and a predetermined number of pixels from the first image. A process of specifying a plurality of pixel blocks to be configured, a process of generating input data based on the pixel value of each pixel of the specified pixel block, and an output value output from the detector to which the generated input data is input On the basis thereof, the process of generating an estimated image of a second image including the region of interest of the sample obtained by the second method is executed without relying on the second method.

  The computer program according to the present embodiment causes a computer to execute a process of comparing and displaying the first image and the estimated image.

  The computer program according to the present embodiment is a process for specifying on a computer a plurality of pixel blocks each having a predetermined number of pixels from each of a plurality of first images including a sample obtained in time series by the first method. And processing for generating a plurality of sets of input data based on the pixel value of each pixel of the specified pixel block for each of the plurality of first images, and the detector to which the plurality of sets of input data are input And a process of displaying the estimated movement trajectory of the sample based on the output value output by

  The computer program according to the present embodiment relates to a process of acquiring the first image data relating to at least one of a phase difference image containing cells and a bright field image on a computer, and a stained image containing a region of interest of the cells. And a process of acquiring the second image data.

  The computer program according to the present embodiment is a computer program for causing a computer to detect a sample, and the computer acquires the first image data related to the first image including the sample obtained by the first method. The second method without depending on the second method different from the first method, which makes it possible to identify the region of interest in the sample based on the processing to be performed and the acquired first image data and the learned detector. And a process of generating an estimated image of a second image including the region of interest of the sample obtained by

  The detection device according to the present embodiment is a detection device that detects a sample, and a first acquisition unit that acquires first image data related to a first image including a learning sample, obtained by the first method. Acquiring second image data related to a second image including a region of interest of the learning sample, obtained by a second method different from the first method, which makes it possible to identify the region of interest in the learning sample A first image related to a first image including a specimen, obtained by the first method, and a learning data generation unit that generates learning data based on a second acquisition unit, the first image data and the second image data And a learning processing unit that causes the detector to learn based on the generated learning data so that the detector can detect the sample based on the data.

  The detection device according to the present embodiment includes an acquisition unit for acquiring first image data related to a first image including a specimen obtained by the first method, and a predetermined number of pixels from the first image. The input data generation unit that generates input data based on the pixel value of each pixel of the pixel block identified by the identification unit, and the input data generated by the input data generation unit An image for generating an estimated image of a second image including a region of interest of the sample obtained by the second method based on the input output value of the detector and without relying on the second method And a generation unit.

  The detection apparatus according to the present embodiment includes a plurality of the detectors for each type of region of interest of a sample.

  In the detection device according to the present embodiment, the detector has a plurality of parameters for converting input data into an output value, and the learning processing unit updates the plurality of parameters of the detector to perform the detection. Makes it possible to relearn

  The imaging device according to the present embodiment includes a first output unit that outputs at least one of a phase difference image including a cell or a bright field image obtained by imaging, and a region of interest of the cell obtained by imaging And a first acquisition unit for acquiring at least one of a phase contrast image including learning cells and a bright field image output from the first output unit and a second output unit outputting a stained image including And a second acquisition unit for acquiring second image data related to a stained image including the region of interest of the learning cell, which is output by the second output unit, and based on the first image data and the second image data. A detector can detect a cell based on first data of at least one of a phase difference image including a cell and a bright field image output from the first data output unit and a learning data generation unit that generates learning data On the generated learning day And a learning section for learning the detector based on.

  The imaging device according to the present embodiment includes a specifying unit that specifies a plurality of pixel blocks each having a predetermined number of pixels from at least one of a phase difference image including a cell or a bright field image, and the pixel block specified by the specifying unit. Cells are stained based on an input data generation unit that generates input data based on the pixel value of each of the pixels, and an output value output from the detector to which the input data generated by the input data generation unit is input. And an image data generation unit that generates estimated image data related to the estimated image of the stained image including the region of interest of the cells.

  The server according to the present embodiment includes a detector for detecting cells, an acquisition unit for acquiring image data relating to at least one of a phase difference image or bright field image including cells, and image data acquired by the acquisition unit. And an output unit that outputs estimated image data related to an estimated image of a stained image including a region of interest of the cells without staining the cells.

  The image data according to the present embodiment is image data in which a cell is detected, and has a structure including estimated image data related to an estimated image of a stained image including a region of interest of the cell obtained by imaging. The estimated image is input to input data based on first image data relating to at least one of a phase difference image including cells and a bright field image obtained by imaging, input to a detector for detecting cells, and the detector And the first image data relating to at least one of a phase difference image including a learning cell and a bright field image obtained by imaging and obtained by imaging. Learning is performed using learning data generated based on the second image data related to the stained image including the region of interest of the learning cell.

  The detector according to the present embodiment is a detector that detects a cell, and is obtained by imaging, first image data relating to at least one of a phase difference image including a learning cell and a bright field image; Learning is performed using learning data generated based on second image data related to a stained image including the region of interest of the learning cell, which is obtained by imaging.

  The detector according to the present embodiment identifies a plurality of pixel blocks each having a predetermined number of pixels from the stained image, and at least one pixel block of the phase difference image or bright field image corresponding to the identified pixel block. The learning data is generated by associating the feature amounts of the pixels of the specified pixel block with each other.

  The detector according to the present embodiment includes an input unit that receives, as input data, image data related to at least one of a phase difference image including cells and a bright field image obtained by imaging, and an input received by the input unit. And an output unit that outputs estimated image data related to an estimated image of a stained image including a region of interest of the cell obtained by imaging without staining the cell based on data.

  The detection method according to the present embodiment is a detection method using a detector that detects a cell, and the detector relates to at least one of a phase difference image containing cells and a bright field image obtained by imaging. Image data is received as input data, and estimated image data concerning an estimated image of a stained image including a region of interest of the cells obtained by imaging without staining the cells based on the received input data is output .

  A providing method according to the present embodiment is a method of providing a cell cultured under a predetermined environment, wherein at least one of a phase difference image including the cell and a bright field image obtained by imaging in a computer. Based on the process of acquiring the image data and the acquired image data, the estimated image data of the estimated image of the stained image including the region of interest of the cells obtained by imaging without staining the cells is output. A computer program for executing the processing to be performed, and when receiving a request from a user to which the computer program is distributed, to generate information for providing the user with cells cultured under a predetermined environment.

50 detection device 51 processing unit 511 detection model unit 512 input data generation unit 513 estimated image generation unit 514 learning processing unit 515 learning data generation unit 52 first image data acquisition unit 53 second image data acquisition unit 54 memory 55 output unit 56 display Part 60 Display device 100 Microscope 200, 201 Terminal device 300 Server 310 Distribution server 320 Provision server

Claims (23)

A computer program for causing a computer to detect a sample,
On the computer
A process of acquiring first image data relating to a first image including a learning sample, obtained by the first method;
A process of acquiring second image data related to a second image including a region of interest of the learning sample, obtained by a second method different from the first method, which makes it possible to identify the region of interest in the learning sample When,
A process of generating learning data based on the first image data and the second image data;
Processing the device to learn the detector based on the generated learning data so that the detector can detect the specimen based on the first image data of the first image including the specimen obtained by the first method Computer program to run. On the computer
A process of specifying a plurality of pixel blocks each having a predetermined number of pixels from the second image;
The computer program according to claim 1, wherein the processing of generating the learning data by associating the feature amount of the pixel of the specified pixel block with the pixel block of the first image corresponding to the specified pixel block. On the computer
The computer program according to claim 2, wherein the process of generating the learning data by associating the pixel value of a predetermined pixel in the specified pixel block with the pixel block of the first image is performed. On the computer
A process of binarizing the second image;
A process of specifying a plurality of pixel blocks each having a predetermined number of pixels corresponding to each of binary values which are pixel values of pixels obtained by the binarizing process;
Processing for associating the pixel block of the first image corresponding to the specified pixel block with any of pixel values of predetermined pixels binarized of the specified pixel block to generate the learning data; The computer program according to Item 2. On the computer
The computer program according to any one of claims 1 to 4, wherein a process of displaying an estimated position of a sample is executed based on a result of learning the detector. On the computer
A process of acquiring first image data related to a first image including a sample obtained by the first method;
A process of specifying a plurality of pixel blocks each having a predetermined number of pixels from the first image;
A process of generating input data based on the pixel value of each pixel of the identified pixel block;
An estimation of a second image including a region of interest of the specimen obtained by the second method based on an output value output from the detector, to which the generated input data is input, without relying on the second method The computer program according to any one of claims 1 to 5, wherein a process of generating an image is performed. On the computer
The computer program according to claim 6, which executes processing of comparing and displaying the first image and the estimated image. On the computer
A process of specifying a plurality of pixel blocks each having a predetermined number of pixels from each of a plurality of first images including a specimen obtained in time series by the first method;
A process of generating a plurality of sets of input data based on the pixel value of each pixel of the specified pixel block for each of the plurality of first images;
The process according to any one of claims 1 to 7, wherein the process of displaying the estimated movement trajectory of the sample is performed based on an output value output from the detector to which the plurality of sets of input data are input. Computer program. On the computer
A process of acquiring the first image data relating to at least one of a phase contrast image or a bright field image including cells;
The computer program according to any one of claims 1 to 8, further comprising: executing the process of acquiring the second image data related to the stained image including the region of interest of the cells. A computer program for causing a computer to detect a sample,
On the computer
A process of acquiring first image data related to a first image including a sample obtained by a first method;
The above-mentioned obtained by the second method without relying on the second method different from the first method, which makes it possible to identify the region of interest in the sample based on the acquired first image data and the learned detector. And generating an estimated image of a second image including a region of interest of the sample. A detection device for detecting a sample,
A first acquisition unit configured to acquire first image data related to a first image including a learning sample, obtained by the first method;
Acquiring second image data relating to a second image including a region of interest of the sample for learning obtained by a second method different from the first method, which makes it possible to identify the region of interest in the sample for learning 2 acquisition parts,
A learning data generation unit that generates learning data based on the first image data and the second image data;
A learning processing unit that causes the detector to learn based on the generated learning data so that the detector can detect the specimen based on the first image data of the first image including the specimen obtained by the first method And a detector. An acquisition unit configured to acquire first image data relating to a first image including a specimen obtained by the first method;
An identifying unit that identifies a plurality of pixel blocks each having a predetermined number of pixels from the first image;
An input data generation unit that generates input data based on the pixel value of each pixel of the pixel block identified by the identification unit;
The region of interest of the sample obtained by the second method is included without relying on the second method based on the output value output from the detector at which the input data generated by the input data generation unit is input. The image generation part which produces | generates the presumed image of a 2nd image, The detection apparatus of Claim 11.   The detection device according to claim 11 or 12, further comprising a plurality of the detectors for each type of region of interest in a sample. The detector
Have multiple parameters to convert input data to output value,
The learning processing unit
The detection device according to any one of claims 11 to 13, wherein the plurality of parameters of the detector are updated to make the detector re-learnable. A first output unit that outputs at least one of a phase-contrast image containing cells and a bright-field image obtained by imaging;
A second output unit that outputs a stained image including a region of interest of cells obtained by imaging;
A first acquisition unit that acquires first image data related to at least one of a phase difference image including a learning cell and a bright field image output from the first output unit;
A second acquisition unit that acquires, from the second output unit, second image data related to a stained image including a region of interest of the learning cell;
A learning data generation unit that generates learning data based on the first image data and the second image data;
The detection is performed based on the generated learning data so that the detector can detect the cell based on the first image data related to at least one of the phase difference image including the cell and the bright field image output by the first output unit. An imaging device including a learning processing unit for learning a learning device. An identifying unit that identifies a plurality of pixel blocks each having a predetermined number of pixels from at least one of a phase difference image or bright field image including cells;
An input data generation unit that generates input data based on the pixel value of each pixel of the pixel block identified by the identification unit;
Estimated image data related to an estimated image of a stained image including a region of interest of the cell without staining the cell based on the output value output from the detector, to which the input data generated by the input data generation unit is input The image pickup apparatus according to claim 15, further comprising: an image data generation unit configured to generate A detector for detecting cells,
An acquisition unit configured to acquire image data relating to at least one of a phase difference image including a cell and a bright field image;
A server that outputs image data acquired by the acquisition unit to the detector, and outputs estimated image data related to an estimated image of a stained image including a region of interest of the cells without staining the cells; . Image data in which cells are detected,
It has a structure including estimated image data related to an estimated image of a stained image including a region of interest of cells obtained by imaging;
The estimated image is
Input value based on first image data relating to at least one of a phase difference image including cells and a bright field image obtained by imaging, input value to a detector for detecting cells, and an output value output from the detector Generated based on the
The detector
The first image data according to at least one of a phase difference image including a learning cell and a bright field image obtained by imaging, and a stained image including a region of interest of the learning cell obtained by imaging Image data being learned by learning data generated based on the second image data. A detector for detecting cells,
The first image data according to at least one of a phase difference image including a learning cell and a bright field image obtained by imaging, and a stained image including a region of interest of the learning cell obtained by imaging A detector being trained by training data generated based on the second image data. Identifying a plurality of pixel blocks each having a predetermined number of pixels from the stained image;
20. The image processing device according to claim 19, wherein at least one pixel block of the phase difference image or the bright field image corresponding to the specified pixel block is learned by learning data generated by associating the feature amount of the pixel of the specified pixel block. Detector. An input unit that receives, as input data, image data related to at least one of a phase difference image including cells and a bright field image obtained by imaging;
An output unit for outputting estimated image data of an estimated image of a stained image including a region of interest of the cells obtained by imaging without staining the cells based on the input data received by the input unit; 21. A detector according to claim 19 or 20 comprising. A detection method using a detector for detecting cells, comprising:
The detector
Accept image data relating to at least one of a phase difference image containing cells and a bright field image obtained by imaging as input data;
A detection method for outputting estimated image data concerning an estimated image of a stained image including a region of interest of the cell obtained by imaging without staining the cell based on the received input data. A method for providing cells cultured under a predetermined environment, comprising:
On the computer
A process of acquiring image data related to at least one of a phase difference image including the cells and a bright field image obtained by imaging;
A computer program for executing a process of outputting estimated image data related to an estimated image of a stained image including a region of interest of the cells obtained by imaging without staining the cells based on the acquired image data; Deliver
A provision method for generating information for providing cells cultured under a predetermined environment to a user when receiving a request from a user to which the computer program has been distributed.