JP2021032705A - Dataset generation system and dataset generation method - Google Patents

Abstract

To provide a dataset generation system and a dataset generation method with which it is possible to reduce the burden of a pathologist pertaining to creation of the dataset used in training of a machine learning model for estimating information that indicates a diseased tissue.SOLUTION: The dataset generation system comprises a first dyeing unit, a decolorization unit, a second dyeing unit, an imaging unit and a management unit. The first dyeing unit dyes a biological tissue by a first dyeing method. The decolorization unit decolorizes the biological tissue dyed by the first dyeing method. The second dyeing unit dyes a biological tissue, among biological tissues constituting the biological tissue after being decolorized by the decolorization unit, that satisfies a prescribed condition relating to the state of biological tissue to a color different from other biological tissues that do not satisfy the condition. The imaging unit images the biological tissue. The management unit records, in a storage unit, the image of the biological tissue before decolorization and the image of the biological tissue after being dyed by the second dyeing unit in association.SELECTED DRAWING: Figure 2

Description

Embodiments of the present invention relate to a dataset generation system and a dataset generation method.

Conventionally, in pathological diagnosis, first, a living tissue such as a cell is stained with HE (hematoxylin / eosin). HE staining is a technique for visualizing colorless and transparent living tissue, and a living tissue without a lesion is stained in the same manner as a living tissue having a lesion. After HE staining, the pathologist observes the biological tissue, and the pathologist determines whether or not each biological tissue has a lesion based on knowledge and experience. In this way, since the presence or absence of a lesion in the living tissue is determined based on the knowledge and experience of the pathologist, the lesion may be overlooked. Therefore, more accurate pathological diagnosis has been required.

At present, in order to realize a more accurate pathological diagnosis, it has been proposed to assist the pathological diagnosis by using a machine learning method. Specifically, it is a method in which a computer is trained in a machine learning model that estimates information indicating the lesion tissue, and the computer is made to generate information indicating the lesion tissue based on the learned machine learning model. However, machine learning requires a large amount of data sets of input-side learning data and teacher data to be prepared in order to create a machine learning model. In the proposed machine learning method, it is proposed that the image data of the image of the living tissue stained with HE is used as the training data on the input side, and the judgment result of the pathologist is used as the teacher data for machine learning. However, such a method may increase the burden on the pathologist because the pathologist needs to discriminate the lesion tissue each time one teacher data is created. Further, there is a problem that incorrect teacher data is mixed in due to a pathologist's mistake or oversight.

International Publication No. 2019/008935 Japanese Unexamined Patent Publication No. 2018-180794

An object to be solved by the present invention is to provide a data set generation system and a data set generation method capable of reducing the burden of creating a data set.

The data set generation system of the embodiment includes a first dyeing unit, a bleaching unit, a second dyeing unit, an imaging unit, and a management unit. The first staining section stains the biological tissue by the first staining method. The decolorized portion decolorizes the biological tissue stained by the first staining method. The second dyed portion changes the color of the biological tissue constituting the biological tissue after decolorization by the decolorized portion to a color different from that of other biological tissues that do not satisfy the above conditions. The living tissue satisfying the above conditions is stained by the second staining method, which is a staining method. The imaging unit photographs the biological tissue after staining with the first staining unit and the biological tissue after staining with the second staining unit. The management unit is a first image data which is an image data of a first image which is an image of the living tissue after being stained by the first staining unit, and an image of the living tissue after being stained by the second staining unit. The second image data, which is the image data of the second image, is associated with the second image data and recorded in the storage unit.

The explanatory view of the data set generation method of an embodiment. The figure which shows an example of the system structure of the data set generation system 100 of embodiment. The figure which shows an example of the functional structure of the control part 201 in embodiment. The explanatory view explaining the outline of the flow of the data set generation processing executed by the data set generation system 100 of embodiment. The flowchart which shows an example of the flow of the data set generation processing executed by the data set generation system 100 of embodiment. The explanatory view explaining an example of the processing content of the machine learning using the data set generated by the data set generation system 100 of embodiment. The explanatory view explaining the processing content of the estimation of the immunostaining image by the apparatus which uses the trained model trained by the machine learning using the data set in embodiment.

Hereinafter, the data set generation system and the data set generation method of the embodiment will be described with reference to the drawings. The main purpose of this data set (including the "learning data set" described below) is to use it for learning machine learning, but since it is a collection of data, it can be used for other purposes as well. Can be used.

FIG. 1 is an explanatory diagram of a data set generation method of the embodiment. The data set generation method generates a data set (hereinafter referred to as "learning data set") for learning a machine learning model using image data of an image of a living tissue as an explanatory variable and information indicating a lesion as an objective variable. The method. Machine learning model means a machine learning model in machine learning including deep learning.

In the method of generating a learning data set, a part of a living tissue as a sample is cut out, and a single section which is a part of the cut out sample is stained with HE (hematoxylin / eosin). In the method of generating a learning data set, an image of an HE-stained section (hereinafter referred to as "HE-stained image") is taken, and image data of the HE-stained image (hereinafter referred to as "HE-stained image data") is acquired. To do. In the method of generating the learning data set, the acquired HE-stained image data is recorded in an auxiliary storage device or the like.

In the method of generating the learning data set, the section after taking the HE-stained image is decolorized, and the decolorized section is re-stained by immunostaining. The biological tissue stained by immunostaining is a biological tissue that satisfies a predetermined condition (hereinafter referred to as "state condition") relating to the state of the biological tissue. The condition condition is, for example, the condition of having a lesion. More specifically, the state condition is, for example, a condition in which a protein or gene that is not expressed in normal tissues is expressed. In this case, the biological tissue satisfying the state condition is a cancer cell or the like. In the method of generating a learning data set, an image of an immunostained section (hereinafter referred to as "immunostaining image") is taken, and image data of the immunostaining image (hereinafter referred to as "immunostaining image data") is acquired. To do.

In addition, FIG. 1 shows that there is a living tissue inside the cell 901 that is not stained by immunostaining. Living tissues that are not stained by immunostaining inside cells 901 are, for example, cell nuclei. Cell nuclei are stained by HE staining. In addition, FIG. 1 shows that extracellular positions may be stained. Positions 902, 903 and 904 in FIG. 1 are stained extracellular positions. The extracellular location is stained, for example, by a non-specific reaction of the stain.

In the method of generating the learning data set, the acquired immunostained image data is recorded in an auxiliary storage device or the like in association with the HE-stained image data.

In the method of generating the learning data set, an image process for generating corrected immunostained image data based on the immunostained image (hereinafter referred to as “corrected immunostained image data generation process”) may be executed. The corrected immunostaining image data is image data of an image having substantially the same arrangement and shape as the arrangement and shape of the living tissue shown in the HE-stained image, and the lesion tissue is different in hue, brightness or saturation from other living tissues. It is the image data of the image shown by the color of. The following is an example of the procedure for generating corrected immunostaining image data.

First, the alignment process is executed. In the alignment process, the affine transformation is performed so that the position and shape of the living tissue shown by the HE-stained image and the position and shape of the living tissue shown by the immunostained image are substantially the same. In the alignment process, for example, based on the HE-stained image and the immunostained image, the affine transformation for the immunostained image is executed so that the degree of similarity of the images (hereinafter referred to as "image similarity") is the highest. The image similarity is indicated by, for example, a coefficient indicating the similarity such as a Jaccard coefficient, a Dice coefficient, and a Simpson coefficient. In this case, the immunostained image is affine-transformed so that a coefficient indicating image similarity such as a Jaccard coefficient, a Dice coefficient, and a Simpson coefficient is maximized. The immunostained image may be affine-transformed so that the difference between the color histograms of the HE-stained image and the immunostained image after the affine transformation is the smallest, for example. The immunostained image may be affine-transformed so that the distance between the predetermined feature points between the HE-stained image and the immunostained image after the affine transformation is the smallest, for example.

Next, the image conversion process is executed. By executing the image conversion process, the affine-transformed immunostaining image is converted into a binarized image. The binarized image is an image obtained by binarizing the affine-transformed immunostaining image. The binarization method may be any method. In the binarization method, for example, after projecting an immunostained image into the HSV color space, the pixel values of pixels within a predetermined hue or saturation range are determined to be 1, and the pixel values of other pixels are set to 0. It may be a method of determining. The binarization method may be, for example, a method in which the pixel value of the pixel representing the contour of the region stained by immunostaining is set to 1, and the pixel value of the pixel representing the region other than the contour is set to 0.

Finally, noise removal image processing is performed. The noise reduction image processing is performed on the immunostained image on which the image conversion processing has been performed. Noise removal image processing removes noise in an image. The noise in the image is a pixel value of a pixel satisfying the noise condition. The noise condition is a condition that represents a living tissue stained with a color having the same hue, lightness, or saturation as the living tissue satisfying the state condition among the living tissues that do not satisfy the state condition. The noise in the image is, for example, the pixel value of a pixel representing the extracellular position stained by the residual stain. Removing noise in an image is a process of changing the pixel value of a pixel satisfying the noise condition to a pixel value of a pixel not satisfying the noise condition. The noise removal image processing may be, for example, morphology processing. The morphology process is performed on the binary image. Specifically, in the morphology process, an operation of reversing the pixel value of a pixel having a pixel value of 1 adjacent to a pixel having a pixel value of 0 to 0 is executed a plurality of times. After the inversion operation is executed a plurality of times, the pixel value of the pixel having a pixel value of 0 adjacent to the pixel having a pixel value of 1 is inverted to 1. In the morphology process, if the number of times the inversion operation is performed is one size smaller than the number of pixels, the isolated noise smaller than the cell is removed.

The noise removal image processing may be, for example, processing based on the theory of conditional random fields. The noise reduction image processing may be, for example, a process of filtering by the kernel. The filter may be a filter that outputs an average value, or may be a filter that outputs a median value.

The noise removal image processing may be, for example, the following processing. First, the contour of the cell is detected based on the HE-stained image. Next, it is determined whether or not the region inside the contour is stained, the pixel value of the pixel in the cell is set to 1 depending on whether or not it is stained, and the pixel value of the other pixels is set to 0.

The noise removal image processing is, for example, a process of determining that pixels having a high similarity pixel value or feature amount belong to the same category in an HE-stained image based on the pixel value or feature amount for each pixel or a predetermined area. There may be.

The immunostained image after the execution of the alignment process, the image conversion process, and the noise removal image processing is the corrected immunostained image data.
In the corrected immunostaining image data generation process, it is not always necessary to execute all the processes of the alignment process, the image conversion process, and the noise removal image process. For example, in the corrected immunostaining image data generation process, only the alignment process may be executed. For example, in the corrected immunostaining image data generation process, only the image conversion process may be executed. For example, in the corrected immunostaining image data generation processing, only the noise removal image processing may be executed. For example, in the corrected immunostaining image data generation process, only the alignment process and the image conversion process may be executed. For example, in the corrected immunostaining image data generation process, only the alignment process and the noise removal image process may be executed. For example, in the corrected immunostaining image data generation processing, only the image conversion processing and the noise removing image processing may be executed.
This completes the description of an example of the procedure for generating corrected immunostaining image data.

In the method of generating the learning data set, when the corrected immunostaining image data generation process is executed, the acquired corrected immunostaining image data is recorded in the auxiliary storage device or the like in association with the HE-stained image data.

An example of a training data set is a set of HE-stained image data and immunostained image data or corrected immunostained image data acquired by using the method for generating a training data set. In particular, the HE-stained image data is training data on the input side of the training data set. The immunostaining image data or the corrected immunostaining image data is the teacher data of the learning data set.
This is the end of the description of FIG.

Next, a data set generation system 100, which is a system for generating a learning data set by a method for generating a learning data set, will be described.
FIG. 2 is a diagram showing an example of the system configuration of the data set generation system 100 of the embodiment. The data set generation system 100 generates a training data set. Hereinafter, the process in which the data set generation system 100 generates the learning data set is referred to as a data set generation process. The data set generation system 100 includes a stained image data generation device 1 and an image processing device 2.

The stained image data generation device 1 includes a control unit 101 including a processor such as a CPU (Central Processing Unit) connected by a bus and a memory, and executes a program. The stained image data generation device 1 includes a control unit 101, a storage unit 102, a section preparation unit 103, a first dyeing unit 104, an imaging unit 105, a decolorization unit 106, a second dyeing unit 107, and a communication unit 108 by executing a program. Functions as a device. The control unit 101 controls the operation of each functional unit included in the dyed image data generation device 1. The control unit 101 records various information in the storage unit 102, for example. The storage unit 102 is configured by using a storage device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 102 stores various information regarding the operation of the stained image data generation device 1.

The section preparation unit 103 prepares a section from a sample such as a tissue piece. The section preparation unit 103 prepares a section by, for example, executing a process related to the preparation of the section twice, that is, the primary treatment and the secondary treatment. The primary treatment is a process of fixing the sample and producing a block in which the fixed sample is embedded. Examples of the fixative for fixing the sample include formalin. The primary treatment is, for example, a treatment for preparing an FFPE (formalin-fixed paraffin-embedded) block of a sample. The primary treatment may be, for example, a treatment of embedding a sample by embedding a frozen sample in an embedding agent and quick freezing. The secondary treatment is a treatment for cutting out a section from a block produced by the primary treatment.

<Procedure for making FFPE block>
The FFPE block is produced, for example, by the following procedure.
The specimen is fixed by formalin.

Next, a paraffin replacement process is performed on the sample. The paraffin replacement process performed in the process of producing the FFPE block is performed, for example, by the following procedure. In paraffin replacement, the sample is first immersed in ethanol. The water content of the sample soaked in ethanol is replaced with ethanol. Specimens whose water content has been replaced with ethanol are then immersed in xylene. In the sample soaked in xylene, ethanol is replaced with xylene. Specimens in which ethanol is replaced with xylene are soaked in paraffin. In the sample soaked in paraffin, xylene is replaced with paraffin. Each substitution may be carried out stepwise using a mixed solution with the solution to be replaced. This is the end of the description of the paraffin replacement process.

The paraffin-substituted specimen is cooled. By cooling, the paraffin solidifies and the sample is embedded. In this way, the FFPE block is produced. This completes the description of the procedure for producing the FFPE block.

The secondary treatment is a treatment in which a section is cut out from the block embedded by the primary treatment, and the cut out section is placed on a fixing member such as silane coated glass. When the section preparation unit 103 prepares an FFPE block in the primary treatment, the section preparation unit 103 executes a deparaffinization treatment after the secondary treatment, and wash the sections with running water after the deparaffinization treatment. The deparaffinization process is the reverse operation of paraffin replacement.

The section placed on the fixing member by the secondary processing is not peeled off from the fixing member until the learning data set is generated by the stained image data generation device 1. Since it is not peeled off from the fixing member, the position and shape of the biological tissue in the HE-stained image acquired by the stained image data generator 1 and the biological tissue in the immunostained image acquired by the stained image data generator 1 in the image. Are almost the same. Therefore, machine learning using the learning data set generated by the stained image data generation device 1 can learn a machine learning model with high accuracy.

The first staining unit 104 HE-stains the section placed on the fixing member. The first dyeing section 104 encloses the dyed section with a cover glass. The first staining unit 104 may HE-stain the section by any method as long as the section can be HE-stained. For example, the first dyeing unit 104 may be a commercially available fully automatic HE dyeing apparatus. The sections may be HE-stained, for example, by the following procedure.

<Procedure for HE staining>
First, the sections are washed with distilled water. The washed sections are soaked in hematoxylin solution. After soaking in hematoxylin solution, the sections are rinsed with running water. The sections are colored by being washed with running water. The section may be washed with distilled water and then immersed in alcohol. The colored sections are soaked in eosin solution. Sections soaked in eosin solution are dehydrated multiple times with alcohol. The dehydrated section is immersed in xylene. Sections soaked in xylene are transparent. This is the end of the explanation of the HE staining procedure.

The imaging unit 105 includes a first scanner 501 and a second scanner 502. The first scanner 501 photographs the HE-stained sections by the first staining unit 104. The first scanner 501 is, for example, a digital slide scanner. The image data (that is, HE-stained image data) of the section image (that is, HE-stained image) taken by the first scanner 501 is recorded in the storage unit 102 by the control unit 101. Details of the second scanner 502 will be described later.

The bleaching section 106 bleachs the HE-stained section. The bleaching section 106 may bleach the HE-stained section by any method as long as it can be bleached. An example of the decolorization procedure is shown by taking the case where the section is enclosed by the cover glass as an example.

<Procedure for decolorization>
First, the section is immersed in xylene. By immersing in xylene, the encapsulant such as marinol is removed. The sections are then soaked in alcohol. Immersion in alcohol causes two chemical reactions: xylene substitution and eosin decolorization. The sections after xylene substitution and eosin decolorization are then immersed in diluted alcohol and hydrochloric acid. Hematoxylin is decolorized by immersion in diluted alcohol and hydrochloric acid. Then, the section after the hematoxylin is decolorized is washed with running water. This is the end of the explanation of an example of the decolorization procedure.

The second staining section 107 immunostains the section bleached by the bleaching section 106. Cancer cells are stained with a color different from that of other living tissues by immunostaining by the second staining unit 107. The second staining section 107 encloses the stained living tissue with a cover glass. The second dyeing section 107 is a dyeing method performed by a method other than the first dyeing section 104, and may be dyed by any method if possible. For example, the second dyeing unit 107 may be a commercially available fully automatic dyeing apparatus. The sections may be immunostained, for example, by the following procedure.

<Procedure of immunostaining>
Sections are enzymatically treated with Proteinase K. The enzyme-treated section is soaked in hydrogen peroxide. Immersion in oxidol removes endogenous peroxidase. If the endogenous peroxidase is removed, the section does not necessarily have to be immersed in hydrogen peroxide. On the section from which the endogenous peroxidase has been removed, a primary such as CK CAM5.2 (Cytokeratin CAM5.2 (Cytokeratin is a type of protein, CAM5.2 is an identification number of an antibody that reacts with a specific Cytokeratin)), etc. Antibodies are applied. The secondary antibody is applied to the biological tissue after application of the primary antibody. A color former such as DAB (33, 3'-diaminobenzidine tetrachloride) is applied to the section after application of the secondary antibody. The section is colored by the color former. After color development, the section is immersed in hematoxylin solution. After soaking in hematoxylin solution, the sections are colored with running water. Sections washed with running water are dehydrated multiple times with alcohol. The dehydrated section is immersed in xylene. Sections soaked in xylene are transparent.

If the section is not immersed in hematoxylin solution, the nucleus of the section is not stained. By immersing the section in hematoxylin solution, the nucleus and cancer cells are stained with different colors. When the sections are soaked in hematoxylin solution, the nuclei, cancer cells, and other living tissues are color-coded, respectively. Therefore, the positions and shapes of the nuclei, cancer cells, and other biological tissues become clearer than when the sections are not immersed in hematoxylin solution. This is the end of the description of the immunostaining procedure.

The second scanner 502 photographs the immunostained sections by the second staining unit 107. The image data (that is, the immunostained image data) of the section image (that is, the immunostained image) taken by the second scanner 502 is stored in the storage unit 102 in association with the image data of the HE-stained image taken by the first scanner 501. Recorded. The second scanner 502 is, for example, a digital slide scanner.

The communication unit 108 includes a communication interface for connecting the own device to the image processing device 2. The communication unit 108 communicates with the image processing device 2 via a wired or wireless device.

The image processing device 2 includes a control unit 201 including a processor such as a CPU connected by a bus and a memory, and executes a program. The image processing device 2 functions as a device including a control unit 201, a storage unit 202, and a communication unit 203 by executing a program.

The control unit 201 generates the corrected immunostaining image data by executing the corrected immunostaining image data generation process on the immunostaining image. The corrected immunostained image data is transmitted to the stained image data generation device 1 via the communication unit 203, and is recorded in the storage unit 102 in association with the image data of the HE stained image captured by the first scanner 501.

FIG. 3 is a diagram showing an example of the functional configuration of the control unit 201 in the embodiment.
The control unit 201 includes an image acquisition unit 211, an alignment unit 212, and an image correction unit 213. The image acquisition unit 211 acquires an HE stained image and an immunostained image from the stained image data generation device 1 via the communication unit 203. The alignment unit 212 executes the alignment process. The image correction unit 213 executes image conversion processing and noise removal image processing.

Returning to the description of FIG. The storage unit 202 is configured by using a storage device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 202 stores various information related to the operation of the image processing device 2. The storage unit 202 may store HE-stained image data, immunostained image data, and corrected immunostained image data. The communication unit 203 includes a communication interface for connecting the own device to the stained image data generation device 1. The communication unit 203 communicates with the dyed image data generation device 1 via a wired or wireless device.

FIG. 4 is an explanatory diagram illustrating an outline of a flow of data set generation processing executed by the data set generation system 100 of the embodiment.
The data set generation system 100 cuts a section from a block in which a sample is embedded. The data set generation system 100 places the section on a fixing member such as a slide glass. The data set generation system 100 performs HE staining on the sections placed on the fixing member, and the HE-stained sections are photographed by the first scanner 501. The data set generation system 100 decolorizes the sections that have been photographed by the first scanner 501 and then immunostains them. The immunostained section is imaged by a second scanner 502. The control unit 101 stores the image data of the image of the section taken by the first scanner 501 and the image data of the image of the section taken by the second scanner 502 in the storage unit 102 in association with each other. The image processing device 2 executes noise removal image processing on the image data of the image of the section captured by the second scanner 502. The image processing device 2 may perform an affine transformation on the image of the section acquired by the second scanner 502.

FIG. 5 is a flowchart showing an example of the flow of the data set generation process executed by the data set generation system 100 of the embodiment.
The section preparation unit 103 prepares a section (step S101). Next, the section preparation unit 103 places the produced section on a fixing member such as a slide glass (step S102). Next, the first staining unit 104 HE-stains the section placed on the fixing member (step S103). Next, the first scanner 501 captures the HE-stained section (step S104). As a result, an HE-stained image is acquired. Here, as a normal flow during immunostaining, the slide glass is preferably silane-coated glass, but by fixing to the silane-coated glass in advance before HE staining, the specimen remains stuck during the period from decolorization to re-staining. (The specimen often peels off under other conditions).
Next, the decolorizing section 106 decolorizes the section (step S105). Next, the second staining unit 107 immunostains the section (step S106). That is, the same sample is re-stained after bleaching. The second scanner 502 then captures the immunostained section (step S107). As a result, an immunostaining image is acquired. Next, the control unit 101 records the image data of the HE-stained image acquired in step S104 and the image data of the immunostained image acquired in step S107 in association with each other in the storage unit 102 (step S108).
In the above method, when the section is placed on a slide glass or the like, it is fixed more strongly than before. Therefore, when only the cell nucleus is restained with hematoxylin (H) during immunostaining, the alignment target (images). It functions effectively as a feature point matching, etc.) (Eosin (E), which stains the entire cell, is not used because it stains only the cell nucleus). That is, when aligning two images, "H staining + E staining" (HE staining) and "Cytokeratin + H staining" (immunostaining), the position of the H-stained cell nucleus is corrected more accurately. be able to. Here, in the immunostaining operation, nuclear staining with hematoxylin is performed after coloring with cytokeratin.
Therefore, according to the above-mentioned method in which the sections are more strongly fixed to the slide glass or the like than before, when the deviation between the images obtained by HE staining and immunostaining in S109 described later is minimized, HE staining is performed. By comparing the hematoxylin-stained area and the immunostained area in the image, it becomes easier to recognize how many pixels should be moved.

Next, the control unit 201 included in the image processing device 2 executes the alignment process on the immunostained image acquired in step S107 (step S109). Next, the control unit 201 included in the image processing device 2 executes an image conversion process on the immunostained image after the alignment process is executed (step S110). Next, the control unit 201 included in the image processing device 2 executes noise removal image processing on the immunostained image after the image conversion processing is executed (step S111). Next, the control unit 101 records the image data of the HE-stained image acquired in step S104 and the image data of the immunostained image after the processing of step S111 in association with each other in the storage unit 102 (step S112). .. In FIG. 5, a series of processes from step S109 to step S111 is an example of a corrected immunostaining image data generation process.

FIG. 6 is an explanatory diagram illustrating an example of machine learning processing contents using the learning data set generated by the data set generation system 100 of the embodiment. In other words, the same sample is decolorized, re-stained, and then used for machine learning.
In the machine learning using the learning data set of the embodiment, the parameters of the machine learning model are suitably adjusted by using the HE stained image data as the training data on the input side and the corresponding immunostained image data as the teacher data. The machine learning model may be a neural network. The neural network may be, for example, an encoder and a decoder. The neural network may be a convolutional neural network. The parameters of the machine learning model may be adjusted by an error backpropagation algorithm.
The machine learning model at the time when the end condition is satisfied is the trained model. The end condition may be any condition as long as it is a condition related to the end of learning. The end condition may be, for example, a condition that learning with a predetermined number of data sets has been executed, and the end condition may be, for example, a condition that the amount of change in parameters due to training is less than a predetermined magnitude. There may be.

FIG. 7 is an explanatory diagram illustrating the processing content of estimating the immunostained image by the apparatus using the trained model learned by machine learning using the learning data set in the embodiment. When the device that estimates the immunostained image using the trained model learned by executing the process shown in FIG. 6 acquires new HE-stained image data, it estimates the immunostained image based on the trained model. This is the end of the description of FIG. 7.

The data set generation system 100 configured in this way provides a stained image data generation device 1 that generates data indicating a lesion portion in a living tissue reflected in an HE stained image (that is, immunostained image data or corrected immunostained image data). Be prepared. Therefore, the data set generation system 100 configured in this way can reduce the burden on the pathologist who performs the work of marking the lesion tissue shown in the HE-stained image.

Further, the data set generation system 100 of the embodiment configured as described above includes a stained image data generation device 1 that generates a learning data set by staining a single biological tissue a plurality of times. Therefore, the data set generation system 100 configured in this way can suppress a decrease in learning accuracy due to the fact that the biological tissue captured in the HE-stained image and the biological tissue captured in the immunostained image are not the same.

In addition, a device that shows lesion tissue based on a trained model trained using the learning data set thus constructed can show the lesion portion by inexpensive HE staining without using expensive immunostaining. it can.

(Modification example)
The data set generation system 100 does not necessarily have to include the image processing device 2. In this case, the teacher data of the learning data set is not the corrected immunostaining image data but the immunostaining image data.

The first staining unit 104 may stain the biological tissue by any method, not limited to HE staining, as long as the biological tissue can be stained and decolorized after staining. The first staining unit 104 may stain the living tissue by a method such as Azan staining or Masson staining.

The second staining unit 107 does not necessarily have to stain the biological tissue by immunostaining as long as it can stain the biological tissue satisfying the condition condition with a color having a hue, lightness or saturation different from that of the biological tissue not satisfying the condition condition. Good. The second staining unit 107 may be stained by targeting a specific base sequence (gene) by, for example, a fluorescence in-situ hybridization method (FISH method).

The imaging unit 105 does not necessarily have to include two scanners, the first scanner 501 and the second scanner 502. The imaging unit 105 may include, for example, only the first scanner 501, and the first scanner 501 may take a picture of the living tissue after HE staining and the living tissue after immunostaining.

The dyed image data generation device 1 and the image processing device 2 do not necessarily have to be mounted in different housings. The dyed image data generation device 1 and the image processing device 2 may be configured in one housing. The biological tissue is conveyed between the section preparation unit 103, the first dyeing unit 104, the decolorization unit 106, and the second dyeing unit 107 by a transfer mechanism such as a belt conveyor.

Further, the learning data set generation method does not necessarily have to be executed by an all-automated device such as the data set generation system 100 shown in FIG. In the learning data set generation method, human work may be partially included.

Further, HE staining is an example of the first staining method. Immunostaining is an example of a second staining method. The control unit 101 is an example of a management unit. The HE-stained image is an example of the first image. The first image data is an example of HE-stained image data. The immunostained image is an example of the second image. The second image data is an example of immunostaining image data. The third image data is an example of the corrected immunostaining image data. The image processing device 2 is an example of an image processing unit.

In each of the above embodiments, the control unit 101 and the control unit 201 are assumed to be software function units, but may be hardware function units such as LSI.

According to at least one embodiment described above, the data set generation system 100 includes a first staining unit 104 that stains the living tissue by HE staining, and a decolorizing unit 106 that decolorizes the living tissue stained by HE staining. Among the biological tissues constituting the biological tissue after decolorization by the bleaching unit 106, the biological tissue satisfying the condition condition is subjected to immunostaining, which is a method of dyeing the biological tissue satisfying the condition condition with a color different from that of other biological tissues not satisfying the condition condition. A second staining unit 107 that stains the living tissue to be filled, an imaging unit 105 that photographs the living tissue after staining by the first staining unit 104, and the living tissue after staining by the second staining unit 107, and the first staining unit. The HE-stained image data, which is the image data of the HE-stained image which is the image of the living tissue after staining by 104, and the immunostaining which is the image data of the immuno-stained image which is the image of the living tissue after being stained by the second staining unit 107. It includes a control unit 101 that associates with image data and records it in the storage unit 102. Therefore, the data set generation system 100 configured in this way can reduce the burden of creating the data set. In addition, the data set generation system 100 configured in this way can suppress a decrease in learning accuracy due to the fact that the biological tissue captured in the HE-stained image and the biological tissue captured in the immunostained image are not the same.

All or part of the functions of the control unit 101 and the control unit 201 are realized by using hardware such as ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array). You may. The program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a flexible disk, a magneto-optical disk, a portable medium such as a ROM or a CD-ROM, or a storage device such as a hard disk built in a computer system. The program may be transmitted over a telecommunication line.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, as well as in the scope of the invention described in the claims and the equivalent scope thereof.

100 ... Data set generation system, 1 ... Stained image data generation device, 2 ... Image processing device, 101 ... Control unit, 102 ... Storage unit, 103 ... Section preparation unit, 104 ... First staining unit, 105 ... Imaging unit, 106 ... Decolorization unit, 107 ... Second dyeing unit, 108 ... Communication unit, 201 ... Control unit, 202 ... Storage unit, 203 ... Communication unit, 501 ... First scanner, 502 ... Second scanner

Claims (8)

The first staining part that stains the living tissue by the first staining method,
A decolorized portion that decolorizes the living tissue stained by the first staining method, and a decolorized portion.
A method of dyeing a biological tissue that satisfies a predetermined condition regarding the state of the biological tissue among the biological tissues constituting the biological tissue after decolorization by the decolorized portion in a color different from that of other biological tissues that do not satisfy the above conditions. A second staining section that stains living tissue satisfying the above conditions by the staining method of 2.
An imaging unit that photographs the biological tissue after staining with the first staining unit and the biological tissue after staining with the second staining unit.
The first image data which is the image data of the first image which is the image of the living tissue after staining by the 1st staining part, and the 2nd image which is an image of the living tissue after staining by the 2nd staining part. A management unit that records image data in the storage unit in association with the second image data,
A dataset generation system that includes. Based on the second image data, the image data of the image having substantially the same arrangement and shape as the arrangement and shape of the living tissue shown in the first image, and the living tissue satisfying the above conditions is different from other living tissues. An image processing unit that generates third image data, which is image data of an image indicated by hue, lightness, or saturation color.
With more
The management unit associates the first image data with the third image data and records them in the storage unit.
The data set generation system according to claim 1. The image processing unit executes an affine transformation on the second image so that the degree of similarity between the first image data and the second image data is the highest.
The data set generation system according to claim 2. The image processing unit executes a process of removing noise from the image indicated by the second image data after the execution of the affine transformation.
The data set generation system according to claim 3. The image processing unit executes a process of removing noise in the image indicated by the second image data.
The data set generation system according to claim 2. The first staining method is HE (hematoxylin / eosin) staining.
The data set generation system according to any one of claims 1 to 5. The second staining method is immunostaining.
The data set generation system according to any one of claims 1 to 6. The first staining step of staining the living tissue by the first staining method and
A decolorization step of decolorizing the living tissue stained by the first staining method, and
A second staining for dyeing a biological tissue that satisfies a predetermined condition regarding the state of the biological tissue among the biological tissues constituting the biological tissue after the execution of the decolorization step with a color different from that of other biological tissues that do not satisfy the above conditions. The second staining step of staining by the method and
An imaging step of photographing the living tissue after the execution of the first staining step and the living tissue after the execution of the second staining step.
The first image data which is the image data of the first image which is the image of the living body tissue after the execution of the first staining step, and the image of the living body tissue after the staining by the execution of the second staining step. A management step of associating the second image data, which is the image data of the second image, with the second image data and recording the data in the storage unit,
Data set generation method having.

Images