WO2023145917A1

WO2023145917A1 - Learning model generation method, image processing method, image processing device, computer program, and learning model

Info

Publication number: WO2023145917A1
Application number: PCT/JP2023/002769
Authority: WO
Inventors: 星児鈴木; 祐香嶋津; 匡記松村; 紘一郎頼
Original assignee: テルモ株式会社
Priority date: 2022-01-31
Filing date: 2023-01-30
Publication date: 2023-08-03

Abstract

Provided are a learning model generation method, an image processing method, an image processing device, a computer program, and a learning model with which it is possible to recognize a cell with high accuracy even when there is a change in the imaging environment.　The learning model generation method includes: acquiring first training data that includes a cell image in which a cell is imaged and a nuclear stained image in which the aforementioned cell is stained; and generating a learning model, on the basis of the acquired first training data, so as to output the nuclear stained image when the cell image is inputted.

Description

LEARNING MODEL GENERATING METHOD, IMAGE PROCESSING METHOD, IMAGE PROCESSING APPARATUS, COMPUTER PROGRAM AND LEARNING MODEL

The present invention relates to a learning model generation method, an image processing method, an image processing device, a computer program, and a learning model.

Recognizing and counting the nuclei of cells, or diagnosing pathology from the shape of the nuclei has traditionally been done visually by specialists. there is

Non-Patent Document 1 discloses a method of converting an electron microscope image into an image showing a nucleus using a neural network.

However, since images taken with an optical microscope change depending on the actual shooting environment, high accuracy cannot be obtained simply by using a learning model obtained by deep learning.

The present invention has been made in view of such circumstances, and provides a learning model generation method, an image processing method, an image processing apparatus, a computer program, and a learning model capable of recognizing cells with high accuracy even when the imaging environment fluctuates. intended to provide

The present application includes a plurality of means for solving the above problems. To give an example, the learning model generation method includes first training data including a cell image obtained by imaging a cell and a nuclear staining image obtained by staining the cell. is obtained, and based on the obtained first training data, a learning model is generated so as to output a stained nuclear image when a cell image is input.

According to the present invention, cells can be recognized with high accuracy even when the imaging environment fluctuates.

It is a figure showing an example of composition of an image processing device of this embodiment. It is a figure which shows an example of a normalization process. FIG. 4 is a diagram showing the relationship between normalized images and small images; It is a figure which shows an example of a sliding window method. It is a figure which shows an example of a structure of a learning model. It is a figure which shows an example of the count result screen by an image processing apparatus. FIG. 3 shows a cell mini-image and a corresponding nuclear staining mini-image; FIG. 10 is a diagram showing an example of small cell images in which brightness, contrast, and amount of blurring are varied based on normalized small cell images. FIG. 10 is a diagram showing an example of homogenization processing of nuclear-stained small images; FIG. 3 shows a medium small image and a corresponding nuclear stain small image. FIG. 10 is a diagram showing an example of medium small images with varying brightness, contrast, and blur amount based on normalized medium small images. It is a figure which shows the generation method in the 1st stage of a learning model. It is a figure which shows the generation method in the 1st stage of a learning model. It is a figure which shows the generation method in the 2nd step of a learning model. It is a figure which shows the generation method in the 2nd stage of a learning model. It is a figure which shows the generation method in the 3rd stage of a learning model. It is a figure which shows the generation method in the 3rd stage of a learning model. FIG. 10 is a diagram showing a comparison result between a case where training data includes a medium image containing no cells and a case where the training data does not include the medium image. It is a figure which shows the procedure of the cell counting process by an image processing apparatus. It is a figure which shows the procedure of the production|generation process of the 1st training data by an image processing apparatus. It is a figure which shows the procedure of the production|generation process of the 2nd training data by an image processing apparatus. FIG. 4 is a diagram showing the procedure of a learning model generation process by an image processing device;

Hereinafter, embodiments of the present invention will be described based on the drawings. FIG. 1 is a diagram showing an example of the configuration of an image processing apparatus 50 of this embodiment. The image processing device 50 includes a control unit 51 that controls the entire device, a communication unit 52, a memory 53, a normalization unit 54, an image processing unit 55, a display unit 56, an operation unit 57, a counting unit 58, a storage unit 59, and a learning unit. A processing unit 62 is provided. The storage unit 59 stores a computer program 60, a learning model 61, and required information.

The normalization unit 54, the image processing unit 55, the counting unit 58, and the learning processing unit 62 may be configured by hardware, may be realized by a computer program (software) 60, or may be hardware and software. may be realized in combination with The learning processing unit 62 may be incorporated in an external device, and the learning model 61 generated by the device may be stored in the storage unit 59 . Also, the image processing device 50 may be configured by a plurality of devices.

The control unit 51 can be configured with a CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), or the like. The control unit 51 can execute processing defined by the computer program 60 . That is, the processing by the control unit 51 is also the processing by the computer program 60 .

The communication unit 52, for example, includes a communication module and can transmit and receive information to and from an external device using wireless communication or wired communication. The communication unit 52 has a function as an acquisition unit, and can acquire a cell image obtained by capturing cells, a culture medium image containing no cells, and a stained nuclear image obtained by staining cells from an external device.

As used herein, cells are not particularly limited, and include, for example, adherent cells (adherent cells) and floating cells. Adherent cells include, for example, adherent somatic cells and the like. Examples of somatic cells include myoblasts (e.g., skeletal myoblasts), muscle satellite cells, mesenchymal stem cells (e.g., bone marrow, adipose tissue, peripheral blood, skin, hair root, muscle tissue, intrauterine tissue stem cells such as cardiomyocytes, fibroblasts, cardiac stem cells, embryonic stem cells, pluripotent stem cells such as iPS (induced pluripotent stem) cells, synovial cells, chondrocytes , Epithelial cells (e.g., oral mucosal epithelial cells, retinal pigment epithelial cells, nasal mucosal epithelial cells, etc.), endothelial cells (e.g., vascular endothelial cells, etc.), hepatocytes (e.g., hepatocytes, etc.), pancreatic cells (e.g., pancreatic islet cells, etc.), renal cells, adrenal cells, periodontal ligament cells, gingival cells, periosteal cells, skin cells and the like. Somatic cells may be those differentiated from iPS cells (iPS cell-derived cells), iPS cell-derived cardiomyocytes, fibroblasts, myoblasts, epithelial cells, endothelial cells, hepatocytes, pancreatic cells, kidney cells, adrenal cells, periodontal ligament cells, gingival cells, periosteal cells, skin cells, synovial cells, chondrocytes, and the like. Floating cells include lymphocytes (T lymphocytes, B lymphocytes) and the like. As the cells, nucleated cells having nuclei inside the cells are preferable.

The culture medium in the early stages of culture contains foreign substances such as tissue fragments that are easily mistaken for cells. A medium image is an image that does not contain cells, but contains foreign substances such as tissue fragments.

The memory 53 can be composed of semiconductor memory such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), and flash memory. The computer program 60 can be developed in the memory 53 and the control unit 51 can execute the computer program 60 .

The normalization unit 54 converts the cell image and medium image acquired via the communication unit 52 into grayscale. In addition, the normalization unit 54 normalizes the grayscale-converted images (cell image, culture medium image) converted to grayscale. A normalized grayscale transformed image is also referred to as a normalized image.

FIG. 2 is a diagram showing an example of normalization processing. The horizontal axis of the graph in FIG. 2 indicates the pixel value (luminance value: 0 to 255), and the vertical axis indicates the abundance ratio of each pixel value to the most frequent pixel value. As shown in FIG. 2, attention is paid to the grayscale transformed images Gi, Gj. The grayscale-converted images Gi and Gj are images captured in different shooting environments, such as different shooting timings and shooting locations, and represent images captured by an imaging device (eg, an optical phase-contrast microscope, etc.). The vertical and horizontal resolution (m×n) of an image differs depending on the imaging device. The resolution m or n can be, for example, about 1000, 1200, or 1400, but is not limited to these.

A histogram of the pixel values (for example, 0 to 255) of the specific region Ri of the grayscale converted image Gi is created, and the average value ( Let ai be the brightness) and σi be the variance (contrast). Similarly, a histogram of pixel values (eg, 0 to 255) in a specific region Rj of the grayscale converted image Gj is created, and the distribution of the created histogram is approximated by a predetermined probability distribution (eg, normal distribution). Let aj be the average value (brightness) and σj be the variance (contrast). The specific regions Ri and Rj may be any regions in the grayscale-converted image, but the sizes of the regions are the same.

Let a be the average value of a predetermined probability distribution (for example, normal distribution, etc.), and let σ be the variance. The average value a may be 128, for example. The variance σ may be 15, for example. The average value (brightness) a and variance (contrast) σ can be set based on the values obtained when the image is captured in a shooting environment in which the lighting conditions are good and condensation does not occur on the flask or the like.

The normalization unit 54 converts the grayscale-converted image so that the pixel value distribution of the obtained grayscale-converted (cell image) approximates a predetermined probability distribution. More specifically, the normalization unit 54 converts the grayscale-converted image so that the average value of the pixel value distribution of the acquired grayscale-converted image becomes the average value of the probability distribution, and the acquired grayscale-converted image A grayscale-converted image is transformed so that the variance of the pixel value distribution of the image becomes the variance of the probability distribution. In the example of FIG. 2, the pixel values of all the pixels of the grayscale-converted image Gi are converted so that the average value ai of the distribution of the grayscale-converted image Gi becomes the average value a of the normal distribution. Further, the pixel values of all the pixels of the grayscale-converted image Gi are converted so that the variance σi of the distribution of the grayscale-converted image Gi becomes the variance σ of the normal distribution. Similarly, for the grayscale-converted image Gj, the pixel values of all the pixels of the grayscale-converted image Gj are converted so that the average value aj of the distribution of the grayscale-converted image Gj becomes the average value a of the normal distribution. Further, the pixel values of all the pixels of the grayscale-converted image Gj are converted so that the variance σj of the distribution of the grayscale-converted image Gj becomes the variance σ of the normal distribution.

By normalizing grayscale-converted images (cell images, culture medium images), the impact of changes in the imaging environment on the image quality is reduced. is obtained, and the cells can be recognized with high accuracy by the processing described later.

The image processing unit 55 performs processing to divide the normalized image (cell image and culture medium image) into small images of a predetermined size. A small cell image after grayscale conversion is called a small cell image (recognition target image). there is The predetermined size can be, for example, a resolution of 256×256, but is not limited to this. The image processing unit 55 uses the sliding window method when dividing the normalized image into small images.

FIG. 3 is a diagram showing the relationship between normalized images and small images. In FIG. 3, normalized images are explained as cell images. The normalized image may be the medium image. Small images (small cell images) can be extracted by scanning the normalized image so that a region of a given size is partially overlapped. In the example of FIG. 3, small cell images D1, D2, . . . , Dm, D(m+1), . That is, one normalized image is divided into a plurality of small cell images.

In the example of FIG. 3, the information of the peripheral portion of the small cell image Dm extracted from the area other than the vicinity of the periphery of the normalized image is obtained by using the adjacent small cell image D(m+1) by the sliding window method. Since it can be recognized by the center part such as (m+1), the information on the peripheral part is input to the learning model 61, and there is no problem. On the other hand, in the small cell image D1 extracted from the vicinity of the periphery of the normalized image, the edge E1 of the 256×256 area is adjacent to the boundary of the normalized image, and the information from the image outside the normalized image is Since (hint) cannot be obtained, the accuracy of cell recognition decreases. Similarly, in the small cell image D2 extracted from the vicinity of the periphery of the normalized image, the edge E2 of the 256×256 area cannot obtain information (hints) from the image outside the normalized image. the recognition accuracy of The image processing unit 55 uses a sliding window method when dividing the normalized image into small images so that the cell recognition accuracy is not lowered.

FIG. 4 is a diagram showing an example of the sliding window method. In FIG. 4, normalized images are explained as cell images. The normalized image may be the medium image. As shown in FIG. 4, the image processing unit 55 inserts a frame-shaped area W having a required pixel value around the normalized image (or a non-normalized cell image may be used). The required pixel values include statistical values (eg, mean, mode, or median) of pixel values in the outer peripheral region of the normalized image adjacent to the frame-shaped region W. FIG.

In the small cell image D1 extracted from the vicinity of the outer periphery of the normalized image, the edge of the 256×256 area is adjacent to the frame-shaped area W, and the pixel values of the frame-shaped area W are all 0 or As compared with the case of H.255, it is possible to suppress deterioration in cell recognition accuracy. Similarly, in the small cell image D2 extracted from the vicinity of the periphery of the normalized image, the end of the 256×256 area is adjacent to the frame-shaped area W, and similarly, the cell recognition accuracy (detection accuracy) can suppress the decrease in

The display unit 56 can be configured with a liquid crystal display, an organic EL display, or the like. Note that an external display device may be connected to the image processing device 50 instead of the display unit 56 .

The operation unit 57 is composed of, for example, a keyboard, a mouse, a touch pad, a touch panel, or the like, and can accept operations for information displayed on the display unit 56.

The image processing device 50 divides the acquired normalized image into small cell images of a predetermined size, generates a virtual nuclear staining small image of a predetermined size based on each divided cell small image, and generates each virtual nuclear staining image. The sub-images are stitched together to generate a virtual nuclear staining image of the same size as the original normalized image. In this specification, a stained nuclear image is an image obtained by imaging a cell in which only the nucleus inside the cell is actually stained using a reagent that binds to nucleic acids abundantly contained in the nucleus of the cell. The virtual nuclear-stained image is an image similar to the nuclear-stained image, but it is an image that represents the nuclear-stained image virtually, instead of the image that is actually stained. A learning model 61 is used to generate a virtual nuclear-stained image (virtual nuclear-stained small image). The learning model 61 will be described below.

FIG. 5 is a diagram showing an example of the configuration of the learning model 61. FIG. The learning model 61 comprises an input layer 611 , a first network layer 612 , a second network layer 613 and an output layer 614 . The learning model 61 can use, for example, an autoencoder, but is not limited to this. A 256×256 small cell image (cell image) is input to the input layer 611 . Cell subimages contain cells. A 256×256 virtual nuclear staining small image (virtual nuclear staining image) is output from the output layer 614 . The virtual nuclear staining mini image contains cell nuclei. The first network layer 612 comprises a convolutional layer that weights the information about the input small image of cells, selects information with high importance, and removes other information, thereby compressing the input data and reducing the size of the cells. Extract image features. The second network layer 613 comprises a deconvolution layer, and based on the features extracted in the first network layer 612, generates a virtual nucleus-stained small image as an output image from the compressed data. A first network layer 612 includes, for example, an encoder, and a second network layer 613 includes a decoder. The learning model 61 is not limited to an autoencoder, and may be any model capable of image generation, such as U-net.

In FIG. 5, the small cell image and the virtual nuclear staining small image are shown schematically, and the shape and number of cells in the small cell image and the shape and number of cell nuclei in the virtual nuclear staining small image may differ from the actual one.

The counting unit 58 identifies the position of the cell nucleus based on the virtual nuclear staining image of the same size as the original normalized image (for example, 1200 × 1200), and calculates the number of cells per cell image (cell nucleus number) are counted.

FIG. 6 is a diagram showing an example of the counting result screen 100 by the image processing device 50. FIG. The counting result screen 100 is displayed on the display unit 56, for example. The count result screen 100 displays an uploader area 101, an image selection area 102, a count result display area 104, an "evaluation" icon 103, and the like. By dragging and dropping a cell image file to the uploader area 101, the image processing apparatus 50 displays the counting result in the counting result display area 104 based on the cell image. Further, by clicking the uploader area 101, an image list can be displayed in the image selection area 102. FIG. The image selection area 102 displays columns for a selection box, cell image creation date, cell image file name, cell image source, and cell count. The cell number column displays the number of cells when the cell image has been evaluated, and displays "not evaluated" when the cell image has not been evaluated. By selecting a desired cell image and operating the “evaluation” icon 103 , the counting result is displayed in the counting result display area 104 .

A cell image on which the position of the cell nucleus is superimposed is displayed in the counting result display area 104 . In the figure, the cell nuclei are indicated by black circles, but they may be indicated in other manners, for example, with an identification code such as a cross mark. In addition, in FIG. 6, the cell image is shown schematically, and the number of cell nuclei in the cell image may differ from the actual number. Also, the file name of the original cell image and the number of detected cells may be displayed in the counting result display area 104 .

The number of cells is important information for managing the state of cell culture, for example, determining whether the culture is complete or whether additional culture is required. The number of cells in one cell image may reach several hundred (for example, about 400), and counting by visual inspection is a heavy burden on human beings. There is variability in judgment. In addition, foreign substances such as tissue fragments may be misidentified as cells. However, as described above, according to the present embodiment, not only is the number of cells automatically counted using deep learning, but also normalization processing is performed without relying on visual observation. Absorbed, a cell image with image quality according to a predetermined probability distribution is obtained, and cells can be recognized (detected) with high accuracy. In addition, sample cleaning is not required, and cells can be recognized with high accuracy even when the sample condition fluctuates. Further, according to the present embodiment, by using the sliding window method, it is possible to suppress the decrease in cell recognition accuracy in the vicinity of the periphery of the small cell image (recognition target image), and add a frame-shaped area to the recognition target image. By doing so, it is possible to suppress the decrease in accuracy of recognizing cells in the vicinity of the periphery of the normalized image.

Next, a method for generating the learning model 61 will be described. First, a method of generating training data for generating the learning model 61 will be described, and then a method of learning the learning model 61 will be described.

FIG. 7 is a diagram showing a small cell image Di and a corresponding nuclear staining small image Si. The small cell image Di contains cells, and the stained nuclear small image Si is a nuclear stained image of the small image and contains stained cell nuclei. The staining state of the cell nucleus is not uniform, and the degree of staining varies. Small cell images and small nuclear staining images are schematic representations, and the shape and number of cells in the small cell images and the shape and number of cell nuclei in the small nuclear staining images may differ from the actual ones. There is

The small cell image Di is an image obtained by dividing the original cell image. That is, the small cell image Di is a normalized cell image obtained by performing the normalization process shown in FIG. 2 on the original grayscale-converted image and dividing it using the sliding window method shown in FIG. . The stained nuclear small image Si is obtained by dividing the stained nuclear image obtained by staining the cells into an image of the same size as the small cell image Di. The index i is the image index and indicates that both images correspond.

FIG. 8 is a diagram showing an example of a small cell image in which the brightness, contrast, and amount of blur are varied based on the normalized small cell image. As shown in FIG. 8, the image processing unit 55 independently varies the brightness, contrast, and blur amount of the normalized small cell image Di over at least three stages. For example, the image processing unit 55 may process a dark, low-contrast, and blurred image, a bright, normal-contrast, and non-blurred image, and a normal-brightness, high-contrast, and blurred image based on these three parameters ( Brightness, contrast, and amount of blurring) are processed to generate multiple (3 x 3 x 3 = 27 types in the example shown) small cell images based on the normalized small cell images. do. This makes it possible to augment the learning data.

As a method for increasing the number of images, the image processing unit 55 may perform processing such as rotation or flipping in addition to variations in brightness, contrast, and amount of blurring. For example, there are 4 types of rotation for each 90 degrees, and 8 types including vertical, horizontal, and diagonal flips (horizontal flips and vertical flips are equivalent to rotating 180 degrees, so 8 types because there are overlaps) ) may be generated. With these, 27×8=216 small cell images can be generated from one normalized small cell image.

FIG. 9 is a diagram showing an example of homogenization processing of nuclear-stained small images. The image processing unit 55 converts the stained cell nucleus into a predetermined shape and pixel value to generate a stained nuclear small image Si. Specifically, as shown in FIG. 9, the image processing unit 55 performs binarization processing on the stained nuclear small image Si, and binarizes the stained nuclear small image Si to a binarized image containing dotted pixels. Convert to image. A one-dot dot image in the binarized image represents the position of the cell nucleus. The image processing unit 55 applies Gaussian convolution (blurring) to the binarized image to convert the peripheries (for example, 5 to 6 dots) of the dotted pixels of the binarized image to the required pixel values. A nuclear stained small image Si that has been set is generated.

Convolution moves a pixel block (convolution operator) of a predetermined size on the nuclear staining small image, and multiplies the value of each element of the convolution operator by the corresponding pixel value of the nuclear staining small image (pixel value). This is a process of using the value obtained by summing the values as the central pixel value. Gaussian convolution uses a Gaussian filter with a Gaussian function as the convolution operator.

The required pixel value may be a constant pixel value, or may have a gradation by changing the pixel value outward from a point-like pixel. In the generated nuclear-stained small image Si, the pixels around the cell nucleus are uniform with respect to any cell nucleus, and the pixels have a diameter of 5 to 6 dots. In the nuclear-stained small image before homogenization processing, the brightness of cell nuclei is not uniform depending on the degree of staining. The flow of the equalization process is as follows: (1) Pixel values of pixels below a certain threshold are set to 0 for noise removal. (2) Detect a light spot and convert it into one dot. (3) Blur the 1-dot image to convert it into a pixel with a diameter of 5 to 6 dots.

When small images of nuclear staining before homogenization with non-uniform cell nuclei are used as training data, a characteristic of deep learning is the case where the cell nucleus is lost in the small images of nuclear staining and learning occurs. Decrease accuracy. As in the present embodiment, by using the uniformized nuclear staining small image Si, it is possible to prevent learning by losing sight of the cell nucleus, and to prevent erroneous recognition due to variations in the staining state of the cell nucleus, thereby improving the recognition accuracy of the cell. can be improved.

The control unit 51 acquires first training data including a small cell image obtained by imaging a cell and a nuclear staining small image obtained by staining the cell. That is, the control unit 51 can acquire the increased number of cell image small images illustrated in FIG. 8 and the nuclear staining small images Si (uniformed) corresponding thereto as the first training data. This allows a large amount of training data to be collected.

FIG. 10 is a diagram showing the medium small image Ci and the corresponding nuclear staining small image Si. Medium small images Ci do not contain cells. The corresponding nuclear-stained small image Si does not contain stained cell nuclei, so it is, for example, a black image. The medium small image Ci is shown schematically, and the shape and number of foreign substances such as tissue fragments and medium scratches may differ from the actual ones.

The medium small image Ci is an image obtained by dividing the original medium image. That is, the medium small image Ci is a normalized medium image obtained by performing the normalization process shown in FIG. 2 on the original grayscale-converted image and dividing it using the sliding window method shown in FIG. . The stained nuclear small image Si is an image (for example, a black image) of the same size as the medium small image Ci. The index i is the image index and indicates that both images correspond.

FIG. 11 is a diagram showing an example of small culture medium images in which the brightness, contrast, and amount of blurring are varied based on the normalized small culture medium images. As shown in FIG. 11, the image processing unit 55 independently varies the brightness, contrast, and amount of blurring of the normalized small culture medium images Ci in at least three steps. For example, the image processing unit 55 may process a dark, low-contrast, and blurred image, a bright, normal-contrast, and non-blurred image, and a normal-brightness, high-contrast, and blurred image based on these three parameters ( Brightness, contrast, and amount of blurring) are processed to generate multiple (3 x 3 x 3 = 27 types in the example shown) small medium images based on the normalized small medium images. do. This makes it possible to augment the learning data.

As a method for increasing the number of images, the image processing unit 55 may perform processing such as rotation or flipping in addition to variations in brightness, contrast, and amount of blurring. For example, there are 4 types of rotation for each 90 degrees, and 8 types including vertical, horizontal, and diagonal flips (horizontal flips and vertical flips are equivalent to rotating 180 degrees, so 8 types because there are overlaps) ) may be generated. With these, 27×8=216 small medium images can be generated from one normalized small medium image.

The control unit 51 acquires the second training data including medium small images containing no cells and nuclear staining small images corresponding to the medium small images. That is, the control unit 51 can acquire the increased small culture medium images illustrated in FIG. 11 and the stained nuclear images Si (black images) corresponding to them as the second training data. This allows a large amount of training data to be collected.

Next, a method for generating the learning model 61 using the first training data and the second training data will be described. The first training data and the second training data are also collectively referred to as third training data. Note that the learning model 61 may be generated through three steps.

Based on the acquired first training data, the learning processing unit 62 generates the learning model 61 so that when a cell small image is input, a stained nuclear small image is output. The first training data may include small cell images whose pixel values have been transformed so as to approximate the pixel value distribution of the small cell images to a predetermined probability distribution (eg, normal distribution). Also, the first training data may include small cell images whose pixel values are converted such that the average value or variance of the pixel value distribution of the small cell images becomes the average value or variance of a predetermined probability distribution. Also, the first training data may include small cell images obtained by varying at least one of the mean value and variance of the pixel value distribution of the small cell images and the amount of blurring of the small cell images. Also, the small cell image may include a small cell image in which a frame-shaped area having a required pixel value is inserted in the periphery, as illustrated in FIG. Further, the nuclear staining small image may include a nuclear staining small image in which a frame-shaped area having a required pixel value is inserted in the outer circumference, as illustrated in FIG.

Based on the obtained second training data, the learning processing unit 62 generates the learning model 61 so as to output a nuclear staining small image when a medium small image containing no cells is input. The second training data may include culture medium small images whose pixel values have been transformed so as to approximate the pixel value distribution of the culture medium small images to a predetermined probability distribution (eg, normal distribution). Also, the second training data may include medium small images whose pixel values are converted such that the average value or variance of the pixel value distribution of the medium small images becomes the average value or variance of a predetermined probability distribution. Also, the second training data may include medium small images obtained by varying at least one of the mean value and variance of the pixel value distribution of the medium small images and the amount of blurring of the medium small images. Also, the small culture medium image may include a small culture medium image in which a frame-shaped area having a required pixel value is inserted in the periphery, as illustrated in FIG.

FIGS. 12A and 12B are diagrams showing the method of generating the learning model 61 in the first stage. As shown in FIGS. 12A and 12B, the learning processing unit 62 outputs the small cell image and the small medium image when the small cell image and the small medium image are input, so that the learning model 61 (first network Layer 612) is trained. Specifically, as shown in FIG. 12A, the learning processing unit 62 compares the small cell image Di received in the input layer with the small cell image output from the output layer, and Adjust the parameters so that Di is as reproducible as possible. In addition, as shown in FIG. 12B, the learning processing unit 62 compares the small culture medium images Ci received in the input layer and the small culture medium images output from the output layer, and the input small culture medium images Ci are available. Adjust the parameters so that they are as reproducible as possible. Error backpropagation, for example, can be used as a method of adjusting parameters by comparing input data and output data. Training data can be obtained at low cost by learning to match the input and output of the learning model 61 .

13A and B are diagrams showing a method of generating the learning model 61 in the second stage. As shown in FIGS. 13A and B, the learning processing unit 62 fixes the parameters of the first network layer 612 learned in the first stage, initializes the parameters of the second network layer 613, is input, the learning model 61 (second network layer 613) is trained so as to output a stained nuclear image. Specifically, as shown in FIG. 13A, the learning processing unit 62 sets the parameters of the second network layer 613 so that when the cell small image Di is input, the virtual stained small image Si is output from the output layer. to adjust. Further, as shown in FIG. 13B, the learning processing unit 62 sets the second network so that when the medium small image Ci is input, the virtual stained small image Si (for example, a black image) is output from the output layer. Adjust the parameters of layer 613 .

FIGS. 14A and 14B are diagrams showing the method of generating the learning model 61 in the third stage. As shown in FIGS. 14A and 14B, the learning processing unit 62 cancels the parameter fixation of the first network layer 612, and outputs a stained nuclear small image when a small cell image and a small medium image are input. The learning model 61 is generated by adjusting the parameters of the first network layer 612 and the second network layer 613 . Specifically, as shown in FIG. 14A , the learning processing unit 62 sets the first network layer 612 and the 2 Adjust the network layer 613 parameters. In addition, as shown in FIG. 14B, the learning processing unit 62, when the medium small image Ci is input, the first network The parameters of layer 612 and second network layer 613 are adjusted.

FIG. 15 is a diagram showing the results of comparison between the training data including medium images containing no cells and the training data not including medium images. In FIG. 15, the horizontal axis indicates the true value of the cell count value in one captured image, and the vertical axis indicates the ratio of the cell count value to the true value when the learning model 61 is used. In FIG. 15 , the solid line graph shows the cell count value by the learning model 61 trained using the medium image (small medium image) that does not contain cells, and the broken line graph shows the medium image that does not contain cells (small medium image ), the cell count value by the learning model 61 learned without using. The culture medium in the early stage of culture contains foreign substances such as tissue pieces that are easily mistaken for cells. When the number of cells is approximately 100 or less in one captured image, a large number of tissue fragments are included, and in the case of the learning model 61 learned without using the culture medium image, these tissue fragments are included. The ratio exceeds 100% because there are many errors of misidentification as cells. On the other hand, in the case of the learning model 61 learned using the culture medium image, it can be seen that there are few errors in misidentifying tissue fragments as cells. As described above, according to the present embodiment, since the learning model 61 is generated by including the culture medium image and the corresponding nuclear staining image (for example, black image) in the training data, the cell recognition accuracy in the early stage of culture is improved. can be made

FIG. 16 is a diagram showing the procedure of cell counting processing by the image processing device 50. FIG. For the sake of convenience, the main body of processing will be described as the control unit 51 . The controller 51 acquires a cell image (S11). A cell image is an image with a vertical and horizontal resolution (m×n) of about 1000, 1200, or 1400, for example. The control unit 51 converts the acquired cell image into grayscale (S12), and normalizes the grayscale converted image (S13). For normalization, the method illustrated in FIG. 2 can be used.

The control unit 51 divides the normalized image into small cell images using a sliding window method (S14). The resolution (size) of each segmented small cell image can be, for example, 256×256. The control unit 51 inputs the cell small image to the learning model 61 and acquires the virtual nuclear staining small image output by the learning model 61 (S15).

The control unit 51 joins the acquired virtual nuclear staining small images into a virtual nuclear staining image of the same size as the cell image (S16). The control unit 51 identifies the positions of cell nuclei based on the virtual nuclear staining image (S17), and counts the number of cells (number of positions of cell nuclei) (S18). The control unit 51 superimposes the position of the cell nucleus on the cell image, outputs the counting result (S19), and ends the process.

FIG. 17 is a diagram showing the procedure for generating the first training data by the image processing device 50. FIG. The control unit 51 acquires the cell image and the corresponding nuclear staining image (S31). The cell image and the corresponding nuclear staining image are images having vertical and horizontal resolutions (m×n) of about 1000, 1200, and 1400, for example. The control unit 51 converts the obtained cell image into grayscale (S32), and normalizes the converted cell image (S33). For normalization, the method illustrated in FIG. 2 can be used.

Based on the normalized cell image, the control unit 51 generates a cell image with varying brightness, contrast, and blurring amount (S34). A method illustrated in FIG. 8 can be used to generate a cell image. The control unit 51 binarizes the obtained stained nuclear image to generate a binary image (S35), and performs Gaussian convolution on the generated binary image to generate a stained nuclear image (S36). The process of steps S35 and S36 can use the method illustrated in FIG.

The control unit 51 generates training data (first training data) by dividing the generated cell image and nuclear staining image into small images (cell small image and nuclear staining small image) while making them correspond to each other by a sliding window method. (S37), the process ends. The resolution (size) of the divided cell small image and nuclear staining small image can be, for example, 256×256.

FIG. 18 is a diagram showing the procedure for generating the second training data by the image processing device 50. FIG. The control unit 51 acquires a medium image containing no cells (S41). The culture medium image is an image with a vertical and horizontal resolution (m×n) of about 1000, 1200, and 1400, for example. The control unit 51 converts the obtained culture medium image into grayscale (S42), and normalizes the converted culture medium image (S43). For normalization, the method illustrated in FIG. 2 can be used.

Based on the normalized culture medium image, the control unit 51 generates a culture medium image with varying brightness, contrast, and blurring amount (S44). The method illustrated in FIG. 11 can be used to generate the culture medium image. The control unit 51 acquires an image in which all pixel values have a predetermined value as a stained nuclear small image (S45). A nuclear-stained small image can be, for example, a black image (pixel value is 0).

The control unit 51 divides the generated culture medium image into small images (medium small images) by a sliding window method, associates the acquired nuclear staining small images, and generates training data (second training data) (S46). , terminate the process. The resolution (size) of the segmented medium small image and the corresponding nuclear stain small image can be, for example, 256×256.

FIG. 19 is a diagram showing the procedure for generating the learning model 61 by the image processing device 50. FIG. The learning model 61 is generated using the first training data and second training data (collectively referred to as third training data) generated by the processing shown in FIGS. 17 and 18 . The control unit 51 acquires a small cell image and a small medium image (S51). The size of the cell small image and medium small image can be, for example, 256×256. The control unit 51 initializes the parameters of the learning model 61, and when a small cell image is input to the learning model 61, adjusts the parameters so as to output the small cell image and generates the learning model 61 (S52). .

When the small culture medium image is input to the learning model 61, the control unit 51 adjusts the parameters so as to output the small culture medium image and generates the learning model 61 (S53). The control unit 51 acquires a nuclear staining small image corresponding to the cell small image and a nuclear staining small image corresponding to the medium small image (S54), fixes the parameters of the first network layer 612 of the learning model 61, The parameters of the 2 network layer 613 are initialized (S55).

When the small cell image is input to the learning model 61, the control unit 51 adjusts the parameters so as to output the nuclear staining small image corresponding to the input small cell image and generates the learning model 61 (S56). When the culture medium small image is input to the learning model 61, the control unit 51 adjusts the parameters so as to output the nuclear staining small image corresponding to the input culture medium small image and generates the learning model 61 (S57). The control unit 51 releases the fixed parameters of the first network layer 612 of the learning model 61 (S58).

When the small cell image is input to the learning model 61, the control unit 51 adjusts the parameters so as to output the nuclear staining small image corresponding to the input small cell image and generates the learning model 61 (S59). When the medium small image is input to the learning model 61, the control unit 51 adjusts the parameters so as to output the nuclear staining small image corresponding to the input medium small image and generates the learning model 61 (S60). The control unit 51 stores the generated learning model 61 in the storage unit 59 (S61), and terminates the process.

The image processing method of the present embodiment acquires a cell image obtained by imaging a cell, transforms the cell image so that the distribution of pixel values of the acquired cell image approximates a predetermined probability distribution, and based on the transformed cell image to generate a virtual nuclear staining image, and count the number of cells based on the generated virtual nuclear staining image.

The image processing method of this embodiment converts the cell image so that the average value of the pixel value distribution of the acquired cell image becomes the average value of the probability distribution.

The image processing method of this embodiment converts the cell image so that the variance of the pixel value distribution of the obtained cell image becomes the variance of the probability distribution.

The image processing method of the present embodiment inserts a frame-shaped region having a required pixel value around the periphery of an acquired cell image or a converted cell image, and generates a virtual nuclear staining image based on the cell image in which the frame-shaped region is inserted. Generate.

In the image processing method of the present embodiment, the required pixel values include statistical values of pixel values in the peripheral area of the cell image adjacent to the frame-shaped area.

The image processing method of this embodiment identifies the nuclear positions of cells based on the generated virtual nuclear staining image, and counts the number of identified nuclear positions as the number of cells.

The image processing method of this embodiment superimposes the identified nucleus position on the acquired cell image and displays it.

The image processing apparatus of the present embodiment includes an acquisition unit that acquires a cell image obtained by imaging a cell, a conversion unit that converts the cell image so that the pixel value distribution of the acquired cell image is approximated to a predetermined probability distribution, A generation unit that generates a virtual nuclear staining image based on the converted cell image, and a counting unit that counts the number of cells based on the generated virtual nuclear staining image.

The computer program of the present embodiment provides a computer with a cell image obtained by imaging a cell, converts the cell image so that the distribution of pixel values of the obtained cell image is approximated to a predetermined probability distribution, and converts the cell image. A process of generating a virtual nuclear staining image based on and counting the number of cells based on the generated virtual nuclear staining image is executed.

The learning model generation method of the present embodiment acquires first training data including a cell image obtained by imaging a cell and a nuclear staining image obtained by staining the cell, and inputs a cell image based on the acquired first training data. A learning model is generated so as to output a stained nuclear image when

In the learning model generation method of the present embodiment, the first training data includes cell images in which the pixel values have been converted so as to approximate the pixel value distribution of the cell images to a predetermined probability distribution.

In the learning model generation method of the present embodiment, the pixel values of the first training data are converted such that the average value or variance of the pixel value distribution of the cell image becomes the average value or variance of a predetermined probability distribution. including cell images.

In the learning model generation method of the present embodiment, the first training data includes cell images obtained by varying at least one of the average value, the variance, and the amount of blurring of the cell images of pixel value distribution of the cell images.

The learning model generation method of the present embodiment acquires second training data including a medium image that does not contain cells and a nuclear staining image corresponding to the medium image, and based on the acquired second training data, The learning model is generated so as to output a stained nuclear image when an image of a culture medium without a medium is input.

In the learning model generation method of this embodiment, the second training data includes a culture medium image in which the pixel values have been converted to approximate the pixel value distribution of the culture medium image to a predetermined probability distribution.

In the learning model generation method of the present embodiment, the pixel values of the second training data are converted such that the average value or variance of the pixel value distribution of the culture medium image becomes the average value or variance of a predetermined probability distribution. Contains medium images.

In the learning model generation method of the present embodiment, the second training data includes medium images obtained by varying at least one of the mean value, the variance, and the amount of blurring of the medium image of pixel value distribution of the medium image.

In the learning model generation method of this embodiment, the culture medium image includes a culture medium image in which a frame-shaped area having a required pixel value is inserted in the periphery.

In the learning model generation method of this embodiment, the nuclear staining image includes a nuclear staining image obtained by converting the stained cell nucleus into a predetermined shape and pixel value.

In the learning model generation method of this embodiment, the cell image or nuclear staining image includes a cell image or nuclear staining image in which a frame-shaped area having a required pixel value is inserted in the periphery.

In the learning model generation method of the present embodiment, the learning model includes a first network layer for extracting feature values of an input image and a second network layer for outputting an output image based on the extracted feature values. Acquiring third training data including a cell image captured, a medium image containing no cells, and a nuclear staining image corresponding to the cell image or medium image, and based on the acquired third training data, the cell image and the medium When an image is input, the first network layer is learned so that the cell image and the medium image are output, the learned parameters of the first network layer are fixed, and the obtained third training data is used. and learning the second network layer so as to output a nuclear staining image when a cell image and a medium image are input, canceling the fixation of the parameters of the first network layer, and obtaining third training data Based on, the learning model is generated by learning the first network layer and the second network layer so as to output a stained nuclear image when a cell image and a culture medium image are input.

The image processing method of the present embodiment acquires a cell image obtained by imaging a cell, inputs the acquired cell image into the learning model generated by the learning model generation method described above, acquires a nuclear staining image, and acquires output the stained nuclear image.

The image processing apparatus of the present embodiment includes a learning model generated by the learning model generation method described above, a first acquisition unit that acquires a cell image obtained by imaging a cell, and an acquired cell image that is input to the learning model. It comprises a second acquiring unit that acquires the stained nuclear image, and an output unit that outputs the acquired stained nuclear image.

The computer program of the present embodiment acquires a cell image obtained by imaging a cell into a computer, inputs the acquired cell image into the learning model generated by the learning model generation method described above, and acquires a stained nuclear image. , to output the acquired nuclear staining image, and to execute processing.

The learning model of this embodiment is generated by the learning model generation method described above.

50 image processing device 51 control unit 52 communication unit 53 memory 54 normalization unit 55 image processing unit 56 display unit 57 operation unit 58 counting unit 59 storage unit 60 computer program 61 learning model 611 input layer 612 first network layer 613 second network Layer 614 Output layer 62 Learning processing unit

Claims

Acquiring first training data including a cell image obtained by imaging a cell and a nuclear staining image obtained by staining the cell,
generating a learning model based on the acquired first training data so as to output a stained nuclear image when a cell image is input;
Learning model generation method.
The first training data is
including a cell image in which the pixel values are converted to approximate a distribution of pixel values of the cell image to a predetermined probability distribution;
The learning model generation method according to claim 1.
The first training data is
A cell image in which the pixel values are converted such that the average value or variance of the pixel value distribution of the cell image is the average value or variance of a predetermined probability distribution,
The learning model generation method according to claim 1 or 2.
The first training data is
including a cell image obtained by varying at least one of the average value, the variance, and the amount of blurring of the cell image of the pixel value distribution of the cell image;
The learning model generation method according to any one of claims 1 to 3.
Obtaining second training data including medium images containing no cells and nuclear staining images corresponding to the medium images;
generating the learning model based on the obtained second training data so as to output a nuclear staining image when a medium image containing no cells is input;
The learning model generation method according to any one of claims 1 to 4.
The second training data is
Including a culture medium image in which the pixel values are converted so as to approximate the distribution of the pixel values of the culture medium image to a predetermined probability distribution,
The learning model generation method according to claim 5.
The second training data is
A culture medium image in which the pixel values are converted such that the average value or variance of the pixel value distribution of the culture medium image is the average value or variance of a predetermined probability distribution,
The learning model generation method according to claim 5 or 6.
The second training data is
including a culture medium image in which at least one of the average value of the pixel value distribution of the culture medium image, the variance, and the amount of blurring of the culture medium image is varied,
The learning model generation method according to any one of claims 5 to 7.
The medium image includes a medium image in which a frame-shaped area having a required pixel value is inserted in the outer periphery,
The learning model generation method according to any one of claims 5 to 8.
The nuclear staining image is
Including a nuclear staining image in which the stained cell nucleus is converted into a predetermined shape and pixel value,
The learning model generation method according to any one of claims 1 to 9.
The cell image or nuclear staining image includes a cell image or nuclear staining image in which a frame-shaped area having a required pixel value is inserted in the outer periphery,
The learning model generation method according to any one of claims 1 to 10.
The learning model is
a first network layer for extracting features of an input image;
a second network layer that outputs an output image based on the extracted feature amount,
Acquiring third training data including a cell image capturing a cell, a medium image containing no cell, and a nuclear staining image corresponding to the cell image or the medium image,
Based on the acquired third training data, learning the first network layer so as to output the cell image and the medium image when the cell image and the medium image are input,
Fixing the learned parameters of the first network layer, and learning the second network layer based on the acquired third training data so as to output a stained nuclear image when a cell image and a culture medium image are input. let
The first network layer and the training a second network layer to generate the learned model;
The learning model generation method according to any one of claims 1 to 11.
Acquire a cell image that captures the cell,
inputting the obtained cell image into a learning model generated by the learning model generation method according to any one of claims 1 to 12 to obtain a stained nuclear image;
Output the acquired nuclear staining image,
Image processing method.
A learning model generated by the learning model generation method according to any one of claims 1 to 12;
a first acquisition unit that acquires a cell image obtained by imaging a cell;
a second acquisition unit that acquires a stained nuclear image by inputting the acquired cell image to the learning model;
and an output unit that outputs the acquired nuclear staining image,
Image processing device.
to the computer,
Acquire a cell image that captures the cell,
inputting the obtained cell image into a learning model generated by the learning model generation method according to any one of claims 1 to 12 to obtain a stained nuclear image;
Output the acquired nuclear staining image,
A computer program that causes a process to be performed.
Generated by the learning model generation method according to any one of claims 1 to 12,
learning model.