JP7174371B2 - DATASET GENERATION SYSTEM AND DATASET GENERATION METHOD - Google Patents

Description

Embodiments of the present invention relate to dataset generation systems and dataset generation methods.

Conventionally, in pathological diagnosis, living tissues such as cells are first stained with HE (hematoxylin and eosin). HE staining is a technique for visualizing colorless and transparent living tissue, and stains living tissue without lesions as well as living tissue with lesions. After HE staining, a pathologist observes the living tissue and determines whether or not there is a lesion in each living tissue based on his/her knowledge and experience. In this way, since the presence or absence of lesions in living tissue is determined based on the knowledge and experience of pathologists, lesions may be overlooked. Therefore, a more accurate pathological diagnosis has been demanded.

At present, in order to realize more accurate pathological diagnosis, it is proposed to assist pathological diagnosis using a machine learning method. Specifically, it is a method of having a computer learn a machine learning model for estimating information indicating the diseased tissue, and having the computer generate information indicating the diseased tissue based on the learned machine learning model. However, in machine learning, it is necessary to prepare a large amount of data sets of input-side learning data and teacher data in order to create a machine learning model. In the proposed machine learning method, it is proposed to perform machine learning using image data of HE-stained living tissue images as learning data on the input side and pathologist's judgment results as teaching data. However, such a method requires the pathologist to discriminate the diseased tissue each time one set of teacher data is created, which may increase the burden on the pathologist. Furthermore, contamination of erroneous training data due to misunderstandings or oversights by pathologists is also a problem.

WO2019/008935 JP 2018-180794 A

A problem to be solved by the present invention is to provide a dataset generation system and a dataset generation method that can reduce the burden of creating a dataset.

A data set generation system of an embodiment has a first staining section, a destaining section, a second staining section, an imaging section, and a management section. The first staining section stains the living tissue with a first staining method. The decolorizing section decolorizes the biological tissue dyed by the first staining method. The second staining unit gives a living tissue that satisfies a predetermined condition regarding the state of the living tissue among the living tissue that constitutes the living tissue after decolorization by the decolorizing unit a different color from other living tissue that does not satisfy the condition. A biological tissue that satisfies the above conditions is stained by the second staining method, which is a staining method. The imaging unit images the living tissue after being stained by the first staining unit and the living tissue after being stained by the second staining unit. The management unit stores first image data, which is image data of a first image that is an image of the living tissue after being stained by the first staining unit, and an image of the living tissue after being stained by the second staining unit. The second image data, which is the image data of the second image, is associated with the second image data and recorded in the storage unit.

Explanatory drawing of the data set generation method of embodiment. The figure which shows an example of the system configuration|structure of the dataset generation system 100 of embodiment. The figure which shows an example of the functional structure of the control part 201 in embodiment. FIG. 2 is an explanatory diagram for explaining an overview of the flow of data set generation processing executed by the data set generation system 100 of the embodiment; 4 is a flowchart showing an example of the flow of data set generation processing executed by the data set generation system 100 of the embodiment; FIG. 2 is an explanatory diagram for explaining an example of processing contents of machine learning using a dataset generated by the dataset generation system 100 of the embodiment; FIG. 4 is an explanatory diagram for explaining processing contents of estimating an immunostained image by a device using a learned model learned by machine learning using a data set in the embodiment;

Hereinafter, a dataset generation system and a dataset generation method according to embodiments will be described with reference to the drawings. The main purpose of this dataset (including the "training dataset" described below) is to use it for learning machine learning, but since it is a collection of data, it can be used for other purposes as well. can be used.

FIG. 1 is an explanatory diagram of the data set generation method of the embodiment. A data set generation method generates a data set (hereinafter referred to as a "learning data set") for learning a machine learning model having image data of images of living tissue as explanatory variables and information indicating lesions as objective variables. The method. A machine learning model means a machine learning model in machine learning including deep learning.

In the method of generating the learning data set, a part of the biological tissue that is the sample is cut out, and HE (hematoxylin and eosin) staining is performed on a single section that is a part of the sample that has been cut out. In the method of generating a learning data set, an image of an HE-stained section (hereinafter referred to as "HE-stained image") is photographed, and image data of the HE-stained image (hereinafter referred to as "HE-stained image data") is acquired. do. In the learning data set generation method, the acquired HE stained image data is recorded in an auxiliary storage device or the like.

In the method of generating a learning data set, a section after taking an HE-stained image is destained, and the section after destaining is re-stained by immunostaining. A living tissue to be stained by immunostaining is a living tissue that satisfies a predetermined condition regarding the state of the living tissue (hereinafter referred to as "state condition"). A state condition is, for example, a condition of having a lesion. More specifically, the state condition is, for example, a condition that a protein or gene that is not expressed in normal tissue is expressed. In this case, the biological tissue that satisfies the condition condition is cancer cells or the like. In the method of generating a learning data set, an image of an immunostained section (hereinafter referred to as "immunostained image") is photographed, and image data of the immunostained image (hereinafter referred to as "immunostained image data") is acquired. do.

Note that FIG. 1 shows that there is living tissue inside the cell 901 that is not stained by immunostaining. A biological tissue inside the cell 901 that is not stained by immunostaining is, for example, a cell nucleus. Cell nuclei are stained by HE staining. Figure 1 also shows that extracellular locations may be stained. Locations 902, 903 and 904 in FIG. 1 are extracellular locations that are stained. Extracellular locations are stained, for example, by a non-specific reaction of the stain.

In the learning data set generation method, the acquired immunostained image data is recorded in an auxiliary storage device or the like in association with the HE stained image data.

In the learning data set generation method, image processing for generating corrected immunostained image data based on the immunostained image (hereinafter referred to as “corrected immunostained image data generation processing”) may be performed. The immunostained image data to be corrected is image data of an image having substantially the same arrangement and shape as the arrangement and shape of the living tissue shown in the HE-stained image, and the diseased tissue has a different hue, brightness, or saturation from other living tissues. is image data of an image shown in the color of . An example of the procedure for generating corrected immunostained image data will be described below.

First, alignment processing is performed. In the alignment process, affine transformation is performed so that the position and shape of the living tissue indicated by the HE-stained image and the position and shape of the living tissue indicated by the immunostained image are substantially the same. In the registration process, for example, based on the HE-stained image and the immunostained image, affine transformation is performed on the immunostained image so that the degree of similarity between the images (hereinafter referred to as "image similarity") is maximized. The image similarity is indicated by coefficients such as the Jaccard coefficient, the Dice coefficient, and the Simpson coefficient. In this case, the immunostained image is affine-transformed so as to maximize the coefficient indicating image similarity such as the Jaccard coefficient, Dice coefficient, and Simpson coefficient. The immunostained image may be affine-transformed, for example, so that the difference in color histogram between the HE-stained image and the affine-transformed immunostained image is minimized. The immunostained image may be affine-transformed, for example, so that the distance of predetermined feature points between the HE-stained image and the affine-transformed immunostained image is minimized.

Next, image conversion processing is performed. By executing the image conversion process, the affine-transformed immunostained image is converted into a binarized image. A binarized image is an image obtained by binarizing an affine-transformed immunostained image. Any binarization method may be used. In the binarization method, for example, after projecting an immunostained image onto the HSV color space, the pixel values of pixels within a predetermined range of hue or saturation are determined to be 1, and the pixel values of other pixels are set to 0. It may be a method of determining The binarization method may be, for example, a method of setting the pixel value of pixels representing the contour of the region stained by immunostaining to 1 and setting the pixel value of pixels representing the region other than the contour to 0.

Finally, denoising image processing is performed. Noise removal image processing is performed on the immunostained image on which the image conversion processing has been performed. Denoising image processing removes noise in an image. Noise in an image is the pixel value of a pixel that satisfies the noise condition. The noise condition is a condition that, of the living tissues that do not satisfy the state conditions, those living tissues that satisfy the state conditions are stained with the same hue, brightness, or saturation as those that satisfy the state conditions. The noise in the image is, for example, the pixel values of the pixels representing extracellular locations stained by staining fluid residue. Removing noise in an image is a process of changing the pixel value of a pixel that satisfies the noise condition to the pixel value of a pixel that does not satisfy the noise condition. The denoising image processing may be, for example, morphological processing. Morphological processing is performed on binary images. Specifically, in the morphology processing, an operation of inverting the pixel value of a pixel having a pixel value of 1 adjacent to a pixel having a pixel value of 0 to 0 is performed multiple times. After the inverting operation is performed multiple times, the pixel values of the pixels with pixel values of 0 adjacent to the pixels with pixel values of 1 are inverted to 1. In the morphology processing, if the number of pixels to be flipped is one size smaller than the cell, isolated noise smaller than the cell is removed.

The noise removal image processing may be, for example, processing based on conditional random field theory. The denoising image processing may be, for example, filtering with a kernel. The filter may be a filter that outputs an average value or a filter that outputs a median value.

The noise removal image processing may be, for example, the following processing. First, cell contours are detected based on the HE-stained image. Next, it is determined whether or not the area inside the outline is stained, and depending on whether or not the area is stained, the pixel value of the pixels inside the cell is set to 1, and the pixel value of the other pixels is set to 0.

Noise removal image processing is, for example, a process in which, in an HE-stained image, based on pixel values or feature amounts for each pixel or for each predetermined region, it is determined that pixels with highly similar pixel values or feature amounts belong to the same category. There may be.

The immunostained image after executing the registration processing, the image conversion processing, and the noise removal image processing is the corrected immunostained image data.
In addition, in the corrected immunostained image data generation process, it is not always necessary to perform all of the alignment process, the image conversion process, and the noise removal image process. For example, in the corrected immunostained image data generating process, only the registration process may be executed. For example, in the corrected immunostained image data generation process, only the image conversion process may be executed. For example, only noise removal image processing may be performed in the corrected immunostained image data generation processing. For example, in the corrected immunostained image data generation process, only the alignment process and the image conversion process may be executed. For example, in the corrected immunostained image data generation process, only the alignment process and the noise removal image process may be executed. For example, in the corrected immunostaining image data generation process, only the image conversion process and the noise removal image process may be executed.
This is the end of the explanation of an example of the procedure of the corrected immunostained image data generation process.

In the learning data set generation method, when the corrected immunostained image data generation process is executed, the acquired corrected immunostained image data is recorded in an auxiliary storage device or the like in association with the HE stained image data.

A set of HE-stained image data and immunostained image data or corrected immunostained image data obtained by using the learning data set generation method is an example of a learning data set. In particular, the HE-stained image data is learning data on the input side of the learning data set. Also, the immunostained image data or the corrected immunostained image data is teacher data of the learning data set.
The description of FIG. 1 ends here.

Next, the data set generation system 100, which is a system for generating a learning data set according to a learning data set generation method, will be described.
FIG. 2 is a diagram showing an example of the system configuration of the dataset generation system 100 of the embodiment. The dataset generation system 100 generates a training dataset. Hereinafter, the process of generating the training data set by the data set generation system 100 will be referred to as data set generation process. A data set generation system 100 includes a stained image data generation device 1 and an image processing device 2 .

The staining image data generating apparatus 1 includes a control unit 101 including a processor such as a CPU (Central Processing Unit) connected via a bus and a memory, and executes a program. The stained image data generation apparatus 1 includes a control unit 101, a storage unit 102, a section preparation unit 103, a first staining unit 104, an imaging unit 105, a destaining unit 106, a second staining unit 107, and a communication unit 108 by executing a program. It works as a device. The control unit 101 controls the operation of each functional unit included in the stained image data generation device 1 . The control unit 101 records various information in the storage unit 102, for example. The storage unit 102 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 102 stores various information related to the operation of the stained image data generating apparatus 1 .

The section preparation unit 103 prepares a section from a specimen such as a piece of tissue. The slice preparation unit 103 prepares a slice by, for example, performing processing related to slice preparation twice, a primary process and a secondary process. The primary process is a process of fixing a specimen and producing a block in which the fixed specimen is embedded. Examples of fixatives for fixing specimens include formalin. The primary treatment is, for example, a treatment for producing FFPE (formalin-fixed paraffin-embedded) blocks of specimens. The primary treatment may be, for example, embedding a frozen specimen in an embedding medium and rapidly freezing the specimen to embed the specimen. The secondary process is a process of cutting out a section from the block produced by the primary process.

<FFPE block preparation procedure>
An FFPE block is produced, for example, by the following procedure.
Specimens are fixed with formalin.

Next, a process of paraffin replacement is performed on the specimen. The process of paraffin replacement performed in the process of producing FFPE blocks is performed, for example, by the following procedure. In paraffin replacement, the specimen is first soaked in ethanol. Water in the sample immersed in ethanol is replaced by ethanol. The specimen whose moisture has been replaced with ethanol is then immersed in xylene. Specimens soaked in xylene replace ethanol with xylene. Specimens in which ethanol is replaced by xylene are soaked in paraffin. Specimens soaked in paraffin have xylene replaced by paraffin. Note that each replacement may be performed stepwise using a mixed liquid with the liquid to be replaced. This is the end of the description of the paraffin replacement process.

The paraffin-substituted specimen is cooled. Cooling causes the paraffin to harden and the specimen to be embedded. An FFPE block is thus produced. This completes the description of the procedure for fabricating the FFPE block.

The secondary process is a process of cutting a section from the block embedded by the primary process and placing the cut section on a fixing member such as silane-coated glass. When the section preparation unit 103 prepares FFPE blocks in the primary treatment, the section preparation unit 103 performs deparaffinization after execution of the secondary treatment, and rinses the sections with running water after execution of deparaffinization. Deparaffinization is the reverse operation of paraffinization.

Note that the section placed on the fixing member by the secondary processing is not separated from the fixing member until the learning data set is generated by the stained image data generating apparatus 1 . Since it is not separated from the fixing member, the position and shape in the image of the living tissue appearing in the HE-stained image acquired by the stained image data generation device 1 and the living tissue appearing in the immunostained image acquired by the stained image data generation device 1 are substantially identical. Therefore, machine learning using the learning data set generated by the stained image data generating apparatus 1 can learn a highly accurate machine learning model.

The first staining unit 104 HE-stains the section placed on the fixing member. The first staining unit 104 encloses the section after staining with a cover glass. The first staining unit 104 may HE-stain the section by any method as long as the section can be HE-stained. For example, the first staining unit 104 may be a commercially available fully automatic HE staining device. Sections may be HE-stained, for example, by the following procedure.

<Procedure for HE staining>
First, the sections are rinsed with distilled water. The washed sections are immersed in a hematoxylin solution. After soaking in the hematoxylin solution, the sections are washed with running water. The sections are colored by washing with running water. The section may be immersed in alcohol after washing with distilled water. Colored sections are immersed in eosin solution. Sections soaked in eosin solution are dehydrated in alcohol multiple times. Sections after dehydration are immersed in xylene. Sections immersed in xylene will clear. This is the end of the description of the HE staining procedure.

The imaging unit 105 includes a first scanner 501 and a second scanner 502 . A first scanner 501 images a section HE-stained by the first staining unit 104 . The first scanner 501 is, for example, a digital slide scanner. Image data (i.e., HE-stained image data) of the section image (i.e., HE-stained image) captured by the first scanner 501 is recorded in the storage unit 102 by the control unit 101 . Details of the second scanner 502 will be described later.

A destaining unit 106 destains the HE-stained section. The destaining unit 106 may destain the HE-stained section by any method as long as it can be destained. An example of the destaining procedure will be shown, taking as an example the case where the section is sealed with a cover glass.

<Decolorization procedure>
First, the sections are soaked in xylene. By soaking in xylene, the mounting medium such as marinol is removed. The sections are then dipped in alcohol. The alcohol immersion causes two chemical reactions: xylene substitution and eosin decolorization. Sections after xylene substitution and eosin destaining are then immersed in dilute alcohol and hydrochloric acid. Hematoxylin is decolorized by soaking in dilute alcohol and hydrochloric acid. After the hematoxylin has been destained, the section is washed with running water. This completes the description of an example of the decolorization procedure.

The second staining unit 107 immunostains the section destained by the destaining unit 106 . Immunostaining by the second staining unit 107 stains cancer cells in a color different from that of other living tissues. The second staining section 107 encloses the stained biological tissue with a cover glass. The second dyeing section 107 may be dyed by any dyeing method other than the first dyeing section 104, if possible. For example, the second staining section 107 may be a commercially available fully automatic staining device. Sections may be immunostained, for example, by the following procedure.

<Procedure for immunostaining>
Sections are enzymatically treated with Proteinase K. Enzymatically treated sections are soaked in oxide. Endogenous peroxidase is removed by exposure to oxidol. Note that if endogenous peroxidase is removed, the section does not necessarily need to be immersed in oxidol. A primary antibody such as CK CAM5.2 (Cytokeratin CAM5.2 (Cytokeratin is a type of protein, CAM5.2 is the identification number of an antibody that reacts with a specific Cytokeratin)) is applied to the section from which endogenous peroxidase has been removed. Antibodies are applied. A secondary antibody is applied to the living tissue after application of the primary antibody. A color developer such as DAB (33,3'-diaminobenzidine tetrahydrochloride) is applied to the sections after application of the secondary antibody. Sections are colored with a coloring agent. After coloring, the section is immersed in a hematoxylin solution. After soaking in hematoxylin solution, the sections are washed with running water for color development. The flushed sections are dehydrated multiple times with alcohol. Dehydrated sections are soaked in xylene. Sections immersed in xylene will clear.

If the section is not immersed in the hematoxylin solution, the nucleus of the section is not stained. By immersing the section in a hematoxylin solution, nuclei and cancer cells are stained with different colors. When the section is immersed in hematoxylin solution, nuclei, cancer cells, and other living tissue are color-coded, respectively. Therefore, the respective positions and shapes of the nuclei, cancer cells, and other living tissue become clearer than when the section is not immersed in the hematoxylin solution. This concludes the description of the immunostaining procedure.

A second scanner 502 images a section immunostained by the second staining unit 107 . The image data (ie, immunostained image data) of the section image (ie, immunostained image) captured by the second scanner 502 is stored in the storage unit 102 in association with the image data of the HE-stained image captured by the first scanner 501. Recorded. The second scanner 502 is, for example, a digital slide scanner.

The communication unit 108 includes a communication interface for connecting its own device to the image processing device 2 . The communication unit 108 communicates with the image processing apparatus 2 via a wire or wireless.

The image processing apparatus 2 includes a control unit 201 including a processor such as a CPU and a memory connected via a bus, and executes a program. The image processing device 2 functions as a device including a control unit 201, a storage unit 202, and a communication unit 203 by executing programs.

The control unit 201 generates corrected immunostained image data by executing corrected immunostained image data generation processing on the immunostained image. The immunostained image data to be corrected is transmitted to the stained image data generation apparatus 1 via the communication unit 203 and recorded in the storage unit 102 in association with the image data of the HE stained image captured by the first scanner 501 .

FIG. 3 is a diagram showing an example of the functional configuration of the control unit 201 in the embodiment.
The control unit 201 includes an image acquisition unit 211 , an alignment unit 212 and an image correction unit 213 . The image acquisition unit 211 acquires the HE stained image and the immunostained image from the stained image data generation device 1 via the communication unit 203 . The alignment unit 212 executes alignment processing. The image correction unit 213 executes image conversion processing and noise removal image processing.

Returning to the description of FIG. The storage unit 202 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 202 stores various information regarding the operation of the image processing apparatus 2 . The storage unit 202 may store HE-stained image data, immunostained image data, and corrected immunostained image data. The communication unit 203 includes a communication interface for connecting the own device to the stained image data generation device 1 . The communication unit 203 communicates with the stained image data generation device 1 via a wire or wireless.

FIG. 4 is an explanatory diagram illustrating an overview of the flow of data set generation processing executed by the data set generation system 100 of the embodiment.
The data set generation system 100 cuts a section from the block embedding the specimen. The dataset generation system 100 places the section on a fixed member such as a slide glass. The dataset generation system 100 performs HE staining on the section placed on the fixing member, and images the HE-stained section with the first scanner 501 . The data set generation system 100 decolorizes and then immunostains the section that has been photographed by the first scanner 501 . The immunostained section is imaged by the second scanner 502 . The control unit 101 associates the image data of the slice image captured by the first scanner 501 with the image data of the slice image captured by the second scanner 502 and stores them in the storage unit 102 . The image processing device 2 performs noise removal image processing on the image data of the image of the section captured by the second scanner 502 . The image processing device 2 may perform an affine transformation on the slice image acquired by the second scanner 502 .

FIG. 5 is a flowchart showing an example of the flow of data set generation processing executed by the data set generation system 100 of the embodiment.
The section preparation unit 103 prepares a section (step S101). Next, the section preparation unit 103 places the prepared section on a fixed member such as a slide glass (step S102). Next, the first staining unit 104 HE-stains the section placed on the fixing member (step S103). Next, the first scanner 501 images the HE-stained section (step S104). Thus, an HE-stained image is obtained. Here, silane-coated glass is preferable for the slide glass in the normal flow of immunostaining, but by fixing the slide on the silane-coated glass in advance before HE staining, the specimen remains stuck to the surface during destaining and re-staining. (Other conditions often result in the specimen coming off).
Next, the decolorizing unit 106 decolorizes the section (step S105). Next, the second staining unit 107 immunostains the section (step S106). That is, the same specimen is destained and then re-stained. Next, the second scanner 502 images the immunostained section (step S107). An immunostained image is thus acquired. Next, the control unit 101 associates the image data of the HE-stained image acquired in step S104 with the image data of the immunostained image acquired in step S107, and records them in the storage unit 102 (step S108).
In the above method, when the section is placed on a slide glass or the like, it is fixed more firmly than before. feature point matching, etc.) (Since only the cell nucleus is stained, eosin (E), which stains the entire cell, is not used). That is, when aligning two images of "H staining + E staining" (HE staining) and "Cytokeratin + H staining" (immunostaining), the position of the H-stained cell nucleus is corrected more accurately. be able to. Here, in the operation of immunostaining, nuclear staining with hematoxylin is performed after coloring with cytokeratin.
Therefore, according to the above-described method in which the section is fixed to a slide glass or the like more strongly than before, when minimizing the deviation between images obtained by HE staining and immunostaining in S109 described later, HE staining By comparing the area stained with hematoxylin and the area stained with immunostaining in the image that has undergone staining, it becomes easy to recognize how many pixels should be moved.

Next, the control unit 201 included in the image processing apparatus 2 performs alignment processing on the immunostained image acquired in step S107 (step S109). Next, the control unit 201 included in the image processing apparatus 2 executes image conversion processing on the immunostained image after executing the alignment processing (step S110). Next, the control unit 201 included in the image processing apparatus 2 executes noise removal image processing on the immunostained image after execution of the image conversion processing (step S111). Next, the control unit 101 associates the image data of the HE-stained image acquired in step S104 with the image data of the immunostained image after execution of the process of step S111, and records them in the storage unit 102 (step S112). . In FIG. 5, a series of processes from step S109 to step S111 is an example of corrected immunostaining image data generation process.

FIG. 6 is an explanatory diagram illustrating an example of the processing contents of machine learning using the learning dataset generated by the dataset generation system 100 of the embodiment. In other words, the same specimen is destained, restained, and then used for machine learning.
In the machine learning using the learning data set of the embodiment, HE-stained image data is used as learning data on the input side, and the corresponding immunostained image data is used as teacher data to suitably adjust the parameters of the machine learning model. A machine learning model may be a neural network. The neural network may be, for example, an encoder and a decoder. The neural network may be a convolutional neural network. The parameters of the machine learning model may be adjusted by a back-propagation algorithm.
A machine learning model at the time when the termination condition is satisfied is a trained model. The end condition may be any condition as long as it is related to the end of learning. The termination condition may be, for example, the condition that learning has been performed using a predetermined number of data sets, or the termination condition may be, for example, the condition that the amount of change in the parameter due to learning is less than a predetermined size. There may be.

FIG. 7 is an explanatory diagram for explaining processing contents of estimation of an immunostained image by an apparatus using a learned model learned by machine learning using a learning data set in the embodiment. A device that estimates an immunostained image using a learned model learned by executing the processing shown in FIG. 6 acquires new HE-stained image data and estimates an immunostained image based on the learned model. The description of FIG. 7 ends here.

The data set generation system 100 configured in this manner includes a stained image data generation device 1 that generates data (i.e., immunostained image data or corrected immunostained image data) indicating a lesion portion in a living tissue shown in an HE-stained image. Prepare. Therefore, the data set generation system 100 configured in this manner can reduce the burden on the pathologist who performs the task of marking the diseased tissue shown in the HE-stained image.

Further, the data set generation system 100 of the embodiment configured as described above includes the stained image data generation device 1 that generates a learning data set by staining a single biological tissue multiple times. Therefore, the data set generation system 100 configured in this manner can suppress a decrease in learning accuracy due to the fact that the living tissue shown in the HE-stained image and the living tissue shown in the immunostained image are not the same.

In addition, a device that indicates a lesioned tissue based on a trained model that has been trained using a learning data set configured in this way can indicate a lesioned portion by inexpensive HE staining without using expensive immunostaining. can.

(Modification)
The dataset generation system 100 does not necessarily have to include the image processing device 2 . In this case, the teacher data of the learning data set is not the corrected immunostained image data but the immunostained image data.

The first staining unit 104 may stain the living tissue by any method other than HE staining as long as the living tissue can be stained and decolorized after staining. The first staining unit 104 may stain the living tissue by a method such as Azan staining or Masson staining, for example.

The second staining unit 107 does not necessarily dye the living tissue by immunostaining if the living tissue that satisfies the state condition can be stained with a different hue, brightness, or saturation from that of the living tissue that does not meet the state condition. good. The second staining unit 107 may stain a specific base sequence (gene) as a target by a method such as fluorescence in-situ hybridization (FISH method).

The imaging unit 105 does not necessarily have to have two scanners, the first scanner 501 and the second scanner 502 . The imaging unit 105 may include, for example, only the first scanner 501, and the first scanner 501 may image the living tissue after HE staining and the living tissue after immunostaining.

Note that the stained image data generation device 1 and the image processing device 2 do not necessarily have to be mounted in different housings. The stained image data generation device 1 and the image processing device 2 may be configured in one housing. The living tissue is transported between the section preparation section 103, the first staining section 104, the destaining section 106 and the second staining section 107 by a transport mechanism such as a belt conveyor.

Also, the learning data set generation method does not necessarily have to be executed by an entirely automated device like the data set generation system 100 shown in FIG. Human work may be partially included in the training data set generation method.

Furthermore, HE staining is an example of the first staining method. Immunostaining is an example of the second staining method. Note that the control unit 101 is an example of a management unit. Note that the HE-stained image is an example of the first image. Note that the first image data is an example of HE-stained image data. Note that the immunostained image is an example of the second image. The second image data is an example of immunostaining image data. The third image data is an example of corrected immunostaining image data. Note that the image processing device 2 is an example of an image processing unit.

In each of the above embodiments, the control unit 101 and the control unit 201 are software function units, but they may be hardware function units such as LSI.

According to at least one embodiment described above, the data set generation system 100 includes the first staining unit 104 that stains the living tissue with HE staining, and the destaining unit 106 that decolorizes the living tissue stained with HE staining. , of the living tissues constituting the living tissue after decolorization by the decolorization unit 106, the living tissue that satisfies the condition condition is subjected to immunostaining, which is a method of staining the living tissue that does not satisfy the condition condition, in a color different from that of the other living tissue that does not satisfy the condition condition. A second staining unit 107 that stains the filling biological tissue, an imaging unit 105 that captures the biological tissue after being stained by the first staining unit 104 and the biological tissue after being stained by the second staining unit 107, and the first staining unit HE-stained image data that is image data of an HE-stained image that is an image of the living tissue after staining by the second staining unit 104, and immunostaining that is image data of an immunostained image that is an image of the living tissue after staining by the second staining unit 107 and a control unit 101 that records the data in the storage unit 102 in association with the image data. Therefore, the dataset generation system 100 configured in this way can reduce the burden of creating datasets. In addition, the data set generation system 100 configured in this way can suppress a decrease in the accuracy of learning due to the fact that the living tissue shown in the HE-stained image and the living tissue shown in the immunostained image are not the same.

All or part of each function of the control unit 101 and the control unit 201 is realized using hardware such as ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), FPGA (Field Programmable Gate Array), and the like. may The program may be recorded on a computer-readable recording medium. Computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs and CD-ROMs, and storage devices such as hard disks incorporated in computer systems. The program may be transmitted over telecommunications lines.

While several embodiments of the invention have been described, these embodiments have been presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, as well as the scope of the invention described in the claims and equivalents thereof.

DESCRIPTION OF SYMBOLS 100... Data set generation system 1... Stained image data generation apparatus 2... Image processing apparatus 101... Control part 102... Storage part 103... Section preparation part 104... First staining part 105... Imaging part 106 Decolorization unit 107 Second dyeing unit 108 Communication unit 201 Control unit 202 Storage unit 203 Communication unit 501 First scanner 502 Second scanner

Claims (6)

A data set generation system for generating a data set for learning a machine learning model having image data of an image of a living tissue as an explanatory variable and information indicating a lesion as an objective variable,
a first staining unit that stains a living tissue with a first staining method;
a decolorization unit that decolorizes the biological tissue that has been stained by the first staining method;
A method of staining a biological tissue satisfying a predetermined condition regarding the state of the biological tissue among biological tissues constituting the biological tissue after decolorization by the decolorization unit in a color different from that of other biological tissue not satisfying the condition. a second staining unit that stains the living tissue that satisfies the conditions by the staining method of 2;
an imaging unit that captures images of the living tissue after being stained by the first staining unit and the living tissue after being stained by the second staining unit;
First image data that is image data of a first image that is an image of the living tissue after being stained by the first staining unit, and second image that is an image of the living tissue after being stained by the second staining unit a management unit that associates and records second image data, which is image data, in a storage unit;
Based on the second image data, image data of an image having substantially the same arrangement and shape as the arrangement and shape of the living tissue shown in the first image, wherein the living tissue satisfying the conditions is different from the other living tissues. an image processing unit that generates third image data, which is image data of an image represented by hue, brightness, or saturation;
with
The biological tissue is fixed in advance to a fixing member for fixing the biological tissue with a strength that suppresses displacement of the biological tissue within a predetermined range before staining by the first staining method ,
The image processing unit generates the third image data by performing an affine transformation on the second image such that the degree of similarity between the first image data and the second image data is maximized,
The management unit records the first image data and the third image data in association with each other in the storage unit,
The first image data recorded in the storage unit is learning data on the input side of the data set, and the third image data recorded in the storage unit is teacher data of the data set.
Dataset generation system. The image processing unit performs a process of removing noise from the image indicated by the second image data after the execution of the affine transformation.
The data set generation system according to claim 1 . The image processing unit performs a process of removing noise in the image indicated by the second image data.
The data set generation system according to claim 1 . The first staining method is HE (hematoxylin and eosin) staining,
Data set generation system according to any one of claims 1 to 3 . The second staining method is immunostaining,
Data set generation system according to any one of claims 1 to 4 . A data set generation method for generating a data set for learning a machine learning model having image data of an image of a living tissue as an explanatory variable and information indicating a lesion as an objective variable,
a first staining step of staining the living tissue with a first staining method;
a decolorizing step of decolorizing the biological tissue stained by the first staining method;
A method of staining a biological tissue satisfying a predetermined condition regarding the state of the biological tissue among biological tissues constituting the biological tissue after decolorization in the decolorization step in a color different from that of other biological tissue not satisfying the condition. a second staining step of staining the living tissue that satisfies the conditions by the staining method of 2;
an imaging step of photographing the living tissue after being stained by the first staining step and the living tissue after being stained by the second staining step;
First image data that is image data of a first image that is an image of the living tissue after being stained by the first staining step, and second image that is an image of the living tissue that is an image of the living tissue after being stained by the second staining step. a management step of correlating with second image data, which is image data, and recording in a storage unit;
Based on the second image data, image data of an image having substantially the same arrangement and shape as the arrangement and shape of the living tissue shown in the first image, wherein the living tissue satisfying the conditions is different from the other living tissues. an image processing step of generating third image data, which is image data of an image represented by colors of hue, lightness or saturation;
has
The biological tissue is fixed in advance to a fixing member for fixing the biological tissue with a strength that suppresses displacement of the biological tissue within a predetermined range before staining by the first staining method ,
The image processing step generates the third image data by performing an affine transformation on the second image such that the degree of similarity between the first image data and the second image data is maximized;
The managing step associates and records the first image data and the third image data in the storage unit;
The first image data recorded in the storage unit is learning data on the input side of the data set, and the third image data recorded in the storage unit is teacher data of the data set.
Dataset generation method.

Images