WO2019225506A1 - Biological tissue image processing system, and machine learning method - Google Patents
Biological tissue image processing system, and machine learning method Download PDFInfo
- Publication number
- WO2019225506A1 WO2019225506A1 PCT/JP2019/019741 JP2019019741W WO2019225506A1 WO 2019225506 A1 WO2019225506 A1 WO 2019225506A1 JP 2019019741 W JP2019019741 W JP 2019019741W WO 2019225506 A1 WO2019225506 A1 WO 2019225506A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- biological tissue
- estimator
- learning
- processing system
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N23/00—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00
- G01N23/22—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00 by measuring secondary emission from the material
- G01N23/225—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00 by measuring secondary emission from the material using electron or ion
- G01N23/2251—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00 by measuring secondary emission from the material using electron or ion using incident electron beams, e.g. scanning electron microscopy [SEM]
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N33/00—Investigating or analysing materials by specific methods not covered by groups G01N1/00 - G01N31/00
- G01N33/48—Biological material, e.g. blood, urine; Haemocytometers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
-
- C—CHEMISTRY; METALLURGY
- C12—BIOCHEMISTRY; BEER; SPIRITS; WINE; VINEGAR; MICROBIOLOGY; ENZYMOLOGY; MUTATION OR GENETIC ENGINEERING
- C12M—APPARATUS FOR ENZYMOLOGY OR MICROBIOLOGY; APPARATUS FOR CULTURING MICROORGANISMS FOR PRODUCING BIOMASS, FOR GROWING CELLS OR FOR OBTAINING FERMENTATION OR METABOLIC PRODUCTS, i.e. BIOREACTORS OR FERMENTERS
- C12M1/00—Apparatus for enzymology or microbiology
- C12M1/34—Measuring or testing with condition measuring or sensing means, e.g. colony counters
Definitions
- the present disclosure relates to a biological tissue image processing system and a machine learning method.
- 3D microscopy is known as a technique for analyzing or imaging the 3D structure of biological tissue.
- an electron microscope is generally used.
- three-dimensional microscopy using a scanning electron microscope includes Focused Ion Beam SEM (FIB-SEM) method, Serial Block-Face SEM (SBF-SEM) method, and serial section SEM (Serial Section (SEM) method (see Non-Patent Document 1) has been proposed.
- the continuous section SEM method is also called an array tomography method.
- a plurality of sample sections (ultra-thin pieces) continuous in the depth direction are cut out from a sample of a biological tissue, and they are arranged on a substrate.
- Each sample section on the substrate is sequentially observed with a scanning electron microscope, whereby a plurality of images are acquired. Based on the acquired images, the three-dimensional structure of a specific organ (cell, mitochondria, nucleus, etc.) contained in the biological tissue is analyzed or imaged. According to the continuous section SEM method, it is possible to observe a sample section that has already been observed again.
- a machine learning type estimator is used to detect or identify a target element (Target Component) such as a cell, a cell membrane, or an organelle in the cell. It is possible. In that case, it is desirable to train the estimator so that the estimation accuracy of the estimator is increased.
- observation conditions at the time of acquisition of the learning image should be matched with the observation conditions at the time of acquisition of the image for analysis (observation conditions at the time of analysis). Arise.
- the estimation target in the estimator has a significant thickness in the biological tissue, the above idea does not necessarily hold, and it may be better to give diversity to the observation conditions during learning. It is thought that there is.
- An object of the present disclosure is to improve the estimation accuracy of a target element in a biological tissue image processing system having a machine learning type estimator that estimates a target element in a biological tissue.
- an object of the present disclosure is to cause the estimator to perform learning so that the estimation accuracy of the attention element is increased in the machine learning process of the estimator that estimates the attention element in the biological tissue.
- a biological tissue image processing system includes an electron microscope that generates an image by observing an analysis target that is a biological tissue, and machine learning that applies processing for estimating an element of interest included in the analysis target to the image
- An estimator of the type, and the estimator is a learned estimator that has undergone learning using a learning original image set, and the learning original image set is subjected to biological tissue using an electron microscope under a first observation condition.
- the first observation condition is a condition set for an electron microscope when observing the analysis target sample, and the second observation condition is to perform observation over a depth range different from the first observation condition. Is that condition, it is characterized in.
- a machine learning method applies a process of estimating a target element included in an analysis target sample to an image generated by observing the analysis target sample that is a biological tissue using an electron microscope.
- a biological tissue image processing system including an estimator, a machine learning method for causing the estimator to learn, wherein the estimator learns as a biological tissue with an electron microscope under a first acceleration voltage.
- a step of providing a first original image for learning generated by observing a target sample and a first correct answer image for learning corresponding thereto, and higher than the first acceleration voltage for the estimator A process for providing a second original image for learning generated by observing the learning target sample with an electron microscope under a second acceleration voltage, and a second correct image for learning corresponding thereto.
- the above method is realized as a hardware function or a software function.
- a program for executing the function is installed in the information processing apparatus via a network or a portable storage medium.
- the concept of the information processing apparatus includes a personal computer, an electron microscope system, and the like.
- a biological tissue image processing system includes an electron microscope and a machine learning type estimator.
- the electron microscope is an apparatus that generates an image by observing a sample to be analyzed which is a biological tissue, and preferably it is a scanning electron microscope.
- the estimator applies a process of estimating the element of interest included in the sample to be analyzed to the image generated by the electron microscope, and the estimator is a learned estimation that has been learned by the original image set for learning. It is a vessel. More precisely, the object to be estimated is an image of the element of interest.
- the learning original image set the first original image generated by observing the learning target sample with the electron microscope under the first observation condition and the learning target sample with the microscope under the second observation condition are observed.
- the first observation condition is a condition set for the electron microscope when observing the analysis target sample, and the condition is also set for the electron microscope when observing the learning target sample.
- the second observation condition is a condition under which observation can be performed over a depth range different from the first observation condition, and the condition is a condition set for the electron microscope when observing the learning target sample.
- the estimation accuracy for the element of interest can be increased by using a learned estimator that has undergone a special learning process. For example, an image (for example, a deep image) that does not relatively occur under the first observation condition is likely to be correctly estimated as a part of the target tissue.
- the analysis target sample and the learning target sample are generally the same (same type) biological tissue, but they are usually physically different.
- the analysis target sample and the learning target sample are generally observed with the same electron microscope, but they may be observed with different electron microscopes.
- the element of interest is a specific tissue component, and in embodiments, the element of interest is a cell membrane.
- the cell itself, the cell lumen (cytoplasm) surrounded by the cell membrane, the organelle in the cell (Organelle), and the like may be considered as elements of interest.
- the machine learning type estimator is configured by CNN, for example, but other types of machine learning type estimators may be used.
- a plurality of images are acquired by the continuous section SEM method (array tomography method), but a plurality of images may be acquired by the FIB-SEM method, the SBF-SEM method, and other methods.
- the above configuration can be applied to two-dimensional microscopy as well as three-dimensional microscopy.
- the second observation condition is a condition under which observation can be performed over a greater depth range than the first observation condition.
- the acceleration voltage, irradiation current, and the like may be changed.
- the first observation condition is a condition for setting the first acceleration voltage as the acceleration voltage of the electron microscope
- the second observation condition is a second acceleration voltage higher than the first acceleration voltage as the acceleration voltage of the electron microscope. Is a condition for setting.
- the acceleration voltage is increased, information from a deeper position in the observation object can be easily obtained. In other words, an object existing deeper can be easily imaged.
- the estimator is trained using the first original image and the second original image, the deep image of the tissue of interest is detected even if the input image of the estimator is an image generated under the first observation condition. It becomes easy to be done. If the sample to be analyzed is observed under the first observation condition in the analysis process, damage to the sample to be analyzed can be prevented or reduced.
- the biological tissue image processing system includes a correct image creation unit that generates a second correct image based on the second original image. If the correct image creation unit for the second original image is provided, image processing suitable for the second original image can be easily applied to the image when creating the correct image.
- the correct image creation unit includes an enhancement unit that emphasizes the deep image of the element of interest in the second original image. According to this configuration, the deep part image that is a part of the target element (target element image) can be easily learned.
- the element of interest is a cell membrane
- the deep image of the element of interest is a deep image of the cell membrane. For example, when analyzing a three-dimensional structure of a cell, it is desired to estimate or detect even a cell membrane present inside a biological tissue section. The above configuration meets such a demand.
- the correct image creation unit is configured to perform segmentation for dividing the image that has been processed by the enhancement unit into a plurality of small regions, and a plurality of small regions corresponding to cell membranes in the segmented image.
- An annotation unit that performs labeling to generate a second correct image. If segmentation (small area division) is performed prior to labeling, the labeling work becomes easier or the labeling becomes accurate.
- the correct image creation unit includes a machine learning type second estimator. This configuration uses another estimator when creating a correct image for the first estimator. The element of interest in the second original image is estimated by the second estimator, and a second correct image is created based on the estimation result.
- the machine learning method includes a first original image for learning generated by observing a learning target sample as a biological tissue with an electron microscope under a first acceleration voltage with respect to an estimator, and A step of providing a first correct image for learning corresponding thereto, and learning generated by observing a learning target sample with an electron microscope under a second acceleration voltage higher than the first acceleration voltage with respect to the estimator And providing a second original image for learning and a second correct image for learning corresponding thereto.
- the first image pair including the first original image and the first correct image is input to the estimator as the first teacher data, and the second original image and the second correct image are used.
- the second image pair is input to the estimator as second teacher data.
- the first correct image is created manually from the first original image, or is created by correcting the estimation result of the estimator for the first original image.
- the second correct image is created from the second original image by the correct image creating unit. Both the first correct image and the second correct image may be generated by the correct image creating unit.
- FIG. 1 shows a biological tissue image processing system according to an embodiment.
- the illustrated biological tissue image processing system 10 is a system for analyzing and imaging a three-dimensional structure of a biological tissue. Using this biological tissue image processing system, for example, an image that three-dimensionally represents nerve cells in the brain of a human body or an animal is generated. Any tissue, organ, etc. in the organism can be the subject of analysis.
- the biological tissue image processing system 10 includes a sample pretreatment device 12, a continuous section creation device 14, a scanning electron microscope (SEM) 16, and a biological tissue image processing device 18. .
- SEM scanning electron microscope
- the sample pretreatment device 12 is a device that performs pretreatment on the tissue 20 extracted from a living body, or corresponds to various instruments for the pretreatment.
- the pretreatment include fixing treatment, dyeing treatment, conductive treatment, resin embedding treatment, and shaping treatment. All or some of them are implemented as necessary.
- the staining treatment osmium tetroxide, uranium acetate, lead citrate, or the like may be used. A staining process may be performed on each sample section described below. Some or all of the steps included in the pretreatment may be performed manually.
- the continuous section preparation device 14 is provided outside the SEM 16 or inside the SEM 16.
- a plurality of sample sections 24 arranged in the depth direction (Z direction) are cut out from the cubic sample after the pretreatment by the continuous section creating apparatus 14.
- an apparatus such as an ultramicrotome may be used.
- the work may be performed manually.
- a plurality of sample sections 24 constitute a sample section group 22.
- the plurality of sample sections 24 cut out are arranged on the substrate 28 in a predetermined arrangement.
- the substrate 28 is, for example, a glass substrate or a silicone substrate.
- the sample section array 22 ⁇ / b> A composed of two sample section rows is configured on the substrate 28, but this is merely an example.
- a sample unit 26 is configured by the substrate 28 and the sample section array 22A.
- each sample section 24 is, for example, on the order of nm or ⁇ m.
- a sample section 24 having a larger size may be produced.
- the thickness (size in the Z direction) of each sample section 24 is, for example, in the range of several nm to several hundred nm, and in the embodiment, the thickness is in the range of, for example, 30 to 70 nm. Any numerical value given in this specification is an example.
- the sample observed in the learning process of the estimator is a learning sample, that is, a learning target sample.
- a plurality of tissue sections are prepared from the learning target sample.
- the sample observed in the analysis process is the sample to be analyzed.
- a plurality of tissue sections are prepared from the sample to be analyzed.
- both the learning target sample and the analysis target sample are expressed as a tissue 20.
- Each sample slice cut out from the learning target sample and each sample slice cut out from the analysis target sample are both expressed as a sample slice 24.
- the learning target sample and the analysis target sample are usually separate, but they may be the same. For example, the analysis process and the learning process may be executed in parallel.
- the SEM 16 includes an electron gun, a deflector (scanner), an objective lens, a sample chamber, a detector 34, a control unit 204, and the like.
- a stage for holding the sample unit 26 and a moving mechanism for moving the stage are provided in the sample chamber.
- the control unit 204 controls the operation of the moving mechanism, that is, the movement of the stage.
- the electron beam 30 is irradiated to a specific sample section 24 selected from the sample section array 22A.
- scanning the irradiation position for example, raster scanning
- the reflected electrons 32 emitted from each irradiation position are detected by the detector 34. Thereby, an SEM image is formed. This is performed for each sample section 24.
- the control unit 204 has a function of setting an acceleration voltage for forming an electron beam. In general, when the acceleration voltage is increased, information from a deeper position in the sample section 24 is easily obtained.
- the X direction observation range and the Y direction observation range coincide with each other.
- the observation range electron beam two-dimensional scanning range
- secondary electrons or the like may be detected.
- the first acceleration voltage (low voltage) as the first observation condition and the second as the second observation condition are obtained in units of 24 sample sections.
- the acceleration voltage (high voltage) is alternately set. That is, the individual sample slices 24 cut out from the learning target sample are observed by scanning with the electron beam 30L formed under the first acceleration voltage (low voltage observation), whereby the low voltage image (first original image). 36L is generated.
- the individual specimen sections 24 are observed by scanning with the electron beam 30H formed under the second acceleration voltage, thereby generating a high voltage image (second original image) 36H (high voltage observation).
- the low voltage image stack 35L is configured by the plurality of low voltage images 36L
- the high voltage image stack 35H is configured by the plurality of high voltage images 36H.
- the first acceleration voltage is set as an observation condition under the control of the control unit 204. That is, the individual sample sections 24 cut out from the analysis target sample are observed by scanning with the electron beam 30L formed under the first acceleration voltage (low voltage observation), and thereby a low voltage image (analysis original image). 38L is generated. A plurality of low voltage images 38L constitute a low voltage image stack 37L.
- the first acceleration voltage is, for example, 1 keV or 2 keV
- the second acceleration voltage is, for example, 3 keV, 5 keV, or 7 keV. It is desirable to determine the first acceleration voltage and the second acceleration voltage in consideration of the sample, section thickness, target tissue, observation purpose, and other circumstances.
- the image stacks 35L, 35H, and 37L include a plurality of images 36L, 36H, and 38L corresponding to a plurality of depths in the Z direction (in other words, arranged in the Z direction in the data storage space).
- Each of the images 36L, 36H, and 38L is an original image or an input image when viewed from the biological tissue image processing apparatus 18 side.
- the images 36L, 36H, and 38L are electronic data, and the images 36L, 36H, and 38L are transmitted from the SEM 16 to the biological tissue image processing apparatus 18 via a network or a portable storage medium.
- the biological tissue image processing apparatus 18 is configured by a personal computer in the illustrated configuration example.
- the biological tissue image processing apparatus 18 may be incorporated in the SEM 16, or the biological tissue image processing apparatus 18 may be incorporated in a system computer that controls the SEM 16 or the like.
- the SEM 16 may be controlled by the biological tissue image processing apparatus 18.
- the biological tissue image processing apparatus 18 has a main body 40, a display 46 and an input device 48.
- the plurality of functions of the main body 40 will be described in detail later with reference to FIGS.
- FIG. 1 two typical functions (first image processing function and second image processing function) exhibited by the main body 40 are represented as blocks.
- the main body 40 includes a first image processing unit 200 including a machine learning type film estimator, and a second image processing unit 202 as a correct image creation unit.
- the display 46 is configured by an LCD, an organic EL display device, or the like.
- the input device 48 includes a keyboard operated by a user, a pointing device, and the like.
- FIG. 2 shows a first configuration example of the main body 40.
- the main body 40 has the first image processing unit 200, the second image processing unit 202, and the volume data processing unit 56 as described above.
- the entity of each configuration shown in FIG. 2 is software, that is, a program executed by a general-purpose processor such as a CPU or GPU, except for a portion corresponding to a user's work or action.
- a general-purpose processor such as a CPU or GPU
- part or all of the configuration may be configured by a dedicated processor or other hardware. All or part of the functions of the biological tissue image processing apparatus may be executed by one or a plurality of information processing devices existing on the network.
- the configuration that functions in the analysis process will be described in detail first, and then the configuration that functions in the learning process will be described in detail.
- the operation in the learning process is also referred to in describing the operation in the analysis process. I will do it.
- the first image processing unit 200 includes a machine learning type film estimator 42, a binarizer (image generator) 50, a correction unit 52, a labeling processing unit 54, and the like.
- the membrane estimator 42 functions as membrane estimation means, and applies a membrane estimation process to the input image 206, thereby outputting a membrane likelihood map 60.
- the membrane estimator 42 is configured by a CNN (Convolutional Neural Network) which is a machine learning type membrane estimator.
- the learning process is executed in advance prior to the actual operation of the CNN, that is, the analysis process so that the film is correctly estimated by the CNN.
- the learning process includes a primary learning process (initial learning process) and a secondary learning process.
- the plurality of image pairs include a plurality of first image pairs and a plurality of second image pairs.
- Each first image pair is composed of a first original image (low voltage image) 36L and a corresponding correct image 223.
- Each second image pair includes a second original image (high voltage image) 36H and a correct image 224 corresponding thereto.
- the correct image 223 is created, for example, by manual work on the first original image 36L.
- the correct image 223 may be created from the first original image 36L by an unsupervised machine learning device, a simple classifier (for example, SVM (Support Vector Vector)) or the like.
- the correct image 224 is created by the second image processing unit 202 based on the second original image, as will be described in detail later.
- the correct image 223 may be created from the first original image 36L by the second image processing unit 202.
- a plurality of image pairs are given as teacher data to the membrane estimator 42 as in the primary learning process.
- the teacher data includes a plurality of image pairs used in the primary learning process and a plurality of image pairs added in the secondary learning process.
- the plurality of added image pairs include a plurality of first image pairs and a plurality of second image pairs.
- Each first image pair includes a first original image 36L and a correct image 64A corresponding to the first original image 36L.
- Each second image pair includes a second original image 36H and a correct image 224 corresponding to the second original image 36H.
- the correct image 64A is created by the configuration from the film estimator 42 to the processing tool unit 44. Specifically, when the first original image 36L is input to the membrane estimator 42, the membrane likelihood map 60 is output from the membrane estimator 42 as an estimation result image.
- a correct image 64 ⁇ / b> A is created through generation of a mask image 62 based on the film likelihood map 60 and user (expert) correction of the mask image 62 using the processing tool unit 44. These processes will be described in detail later. It is also conceivable to use the mask image 62 as the correct image 62A.
- the CNN parameter group in the film estimator 42 is further improved by the secondary learning process. That is, the machine learning result is further accumulated in the film estimator 42.
- the secondary learning process for example, it is determined that the result of the estimation process for the input image 206 (the first original image 36L and the second original image 36H) is sufficiently similar to the corresponding correct images 223, 224, and 64A. If finished. Thereafter, re-learning of the film estimator 42 is executed by the same method as described above as necessary.
- the second image processing unit 202 functions as correct image creation means in both the primary learning process and the secondary learning process.
- an original image (low voltage image) 38L as the input image 206 is input to the membrane estimator 42.
- the database 57 stores a plurality of original images 36L, 36H, and 38L and a plurality of correct images 223, 224, and 64A.
- a plurality of film likelihood maps 60 may be stored there.
- the film estimator 42 and the database 57 may be integrated.
- U-net may be used, or a support vector machine, a random forest, or the like may be used.
- the binarizer 50 functions as an image generator. Specifically, the binarizer 50 is a module that generates a temporary membrane image 62 by binarization processing on the membrane likelihood map 60, as will be exemplified later with reference to FIG.
- the film likelihood map 60 includes a plurality of film likelihoods arranged two-dimensionally. Each film likelihood is a numerical value indicating the probability (probability) of being a film. The film likelihood is, for example, between 0 and 1.
- the film likelihood map 60 can also be understood as a film likelihood image.
- a threshold is set in the binarizer 50, and the binarizer 50 converts a film likelihood that is equal to or greater than the threshold to 1 and converts a film likelihood that is less than the threshold to 0.
- the image generated as a result is a temporary membrane image 62.
- the film image before correction is referred to as a temporary film image 62 in order to distinguish the film image before correction from the film image after correction.
- the image generation unit 61 is configured by both the film estimator 42 and the binarizer 50.
- the entire image generation unit 61 may be configured by CNN or the like. Even in such a case, the stepwise generation of the membrane likelihood map and the temporary membrane image can be considered. It is also possible to use the temporary membrane image 62 as the correct image 62A. Prior to binarization processing, processing such as noise removal and edge enhancement may be applied to the film likelihood map 60.
- the machining tool unit 44 functions as a work support unit or work support means.
- the processing tool unit 44 is provided across the first image processing unit 200 and the second image processing unit 202 in the illustrated configuration example.
- the processing tool unit 44 has a display processing function and an image processing function from the viewpoint of information processing. From the viewpoint of work, it has a correction function and a labeling function. In FIG. 2, these functions are expressed as a correction unit 52, a labeling processing unit 54, and a correction unit 214.
- the correction unit 52 displays, as a work target image, a mask image that is a work target for the user via a work window illustrated in FIG. 5 later, and accepts a user's correction instruction on the work target image. Is.
- the correction contents are reflected in the mask image that is the work target image.
- the work target image includes a discontinuous portion of the film
- a film pixel group is added to the discontinuous portion.
- the correction unit 52 is a module that supports the user's work or operation and manages each mask image.
- the input original image (input image) 206 and the generated membrane likelihood are generated with respect to the correction unit 52 (or the processing tool unit 44).
- the degree map 60 is also sequentially input.
- the original image 206 or the membrane likelihood map 60 corresponding to the work target image can be displayed together with or instead of the temporary membrane image as the work target image.
- the labeling processing unit 54 is a module for performing labeling (painting and labeling) on individual regions (cell lumens) included in the corrected membrane image (or the uncorrected membrane image). Labeling includes manual labeling by the user and automatic labeling. At the stage where the correction work and the labeling work are completed, the three-dimensional labeling data 66 in which the cell lumen is distinguished from the others is configured. This is sent to the volume data processing unit 56. The correction unit 214 will be described later.
- the volume data processing unit 56 includes an analysis unit 56A and a rendering unit 56B.
- the volume data processing unit 56 receives an original image stack 37L composed of a plurality of original images.
- the original image stack 37L constitutes volume data.
- the three-dimensional labeling data 66 is also input to the volume data processing unit 56. It is also a kind of volume data.
- the analysis unit 56A analyzes a target organ (for example, a nerve cell) based on the three-dimensional labeling data 66, for example. For example, the form, volume, length, etc. may be analyzed.
- the original image stack 37L is analyzed with reference to the three-dimensional labeling data.
- the rendering unit 56B forms a three-dimensional image (stereoscopic expression image) based on the three-dimensional labeling data 66. For example, an imaging portion may be extracted from the original image stack 37L based on the three-dimensional labeling data 66, and a rendering process may be applied to the extracted image portion.
- the second image processing unit 202 is a module that generates the second correct image 224 based on the second original image (high voltage image) 36H as the input image 208.
- the high voltage image as the second original image 36H includes more deep images of the cell membrane than the low voltage image as the first original image 36L. This is because when the acceleration voltage is increased, information from a deeper position inside the sample slice can be obtained (this will be described in detail later with reference to FIG. 8).
- a deep image of a cell membrane typically appears as a haze (or like a haze) in the vicinity of a shallow image (surface layer image) of the cell membrane. Even in a low-voltage image, a deep image appears as a haze, but the amount thereof is relatively small, and in a low-voltage image, a blurred image in which it is not clear whether the image is a deep image tends to occur.
- the membrane estimator may be required to input a low voltage image due to damage to the sample section, imaging time, and other reasons. Assuming that a low-voltage image is input, in order to facilitate detection of a deep image of the cell membrane as a part of the cell membrane, in the learning stage of the membrane estimator, the membrane estimator is used together with the low-voltage image as its teaching data. A high voltage image is given.
- the second image processing unit 202 includes a haze enhancement unit 210, a segmentation unit 212, and a correction unit (annotation unit) 214 in the illustrated configuration example.
- the haze enhancement unit 210 applies processing for enhancing the deep image of the cell membrane included in the input image 208. Specifically, contrast enhancement is performed as moya enhancement.
- the segmentation unit 212 applies segmentation to the image 218 after contrast enhancement. Segmentation is a process of dividing image content into a plurality of small areas. This segmentation is performed to make it easier to do the work in the next annotation.
- the correction unit 214 functions as correction means or annotation means. In the sense of a tool for user work, the correction unit 214 exhibits the same function as the correction unit 52 described above. Therefore, in FIG. 2, the correction unit 214 is positioned as a part of the processing tool unit 44. Specifically, the correction unit 214 displays the segmented image 220, accepts the user's membrane designation and membrane correction for the image 220, and as a result, the second correct image 224 as a membrane image showing only the cell membrane. Is to create. A plurality of small regions are displayed in the image 220, and a membrane label is given by the user to the small region corresponding to the cell membrane. The work corresponds to annotation and painting. In order to improve the workability, a work window similar to the above may be displayed. In the learning process (primary learning process and secondary learning process) of the film estimator 42, the second correct image 224 created as described above is given to the film estimator 42.
- the combined use of the film estimator 42 and the processing tool unit 44 can improve the quality of the image group to be analyzed or rendered, and easily and quickly generate the image group. Is possible.
- the membrane estimator 42 can learn using a high voltage image in addition to a low voltage image. Since this is done, the film estimation accuracy of the film estimator 42 can be increased. Actually, a low voltage image and a high voltage image representing the same part of the same sample section for learning are generated, and both of them are given to the film estimator 42 as a part of the teacher data.
- the possibility that the deep image of the cell membrane is regarded as a part of the cell membrane can be increased, so that the quality of the second correct image can be improved.
- a plurality of first original images and a plurality of second original images are sequentially input to the membrane estimator 42 in a predetermined order. Those images may be batch processed.
- FIG. 3 schematically shows a configuration example of the membrane estimator 42.
- the membrane estimator 42 has a number of layers, including an input layer 80, a convolution layer 82, a pooling layer 84, an output layer 86, and the like. They act according to the CNN parameter group 88.
- the CNN parameter group 88 includes a number of weighting factors, a number of bias values, and the like.
- the CNN parameter group 88 is initially configured by an initial value group 94. For example, the initial value group 94 is generated using random numbers or the like.
- the evaluation unit 90 and the update unit 92 function.
- the evaluation unit 90 calculates an evaluation value based on a plurality of image pairs (original images 36L and 36H (see FIG. 2) and corresponding correct images 223, 224 and 64A) constituting the teacher data. is there. Specifically, the evaluation value is calculated by sequentially giving the result 60A of the estimation processing for the original images 36L and 36H and the correct images 223, 224 and 64A corresponding to the original images 36L and 36H to the error function.
- the updating unit 92 updates the CNN parameter group 88 so that the evaluation value changes in a favorable direction. By repeating this, the CNN parameter group 88 is optimized as a whole. Actually, the end of the learning process is determined when the evaluation value reaches a certain value.
- the configuration shown in FIG. 3 is merely an example, and it is possible to use estimators having various structures as the film estimator 42.
- FIG. 4 shows the operation of the binarizer.
- a film likelihood map 60 is output from the film estimator.
- the film likelihood map 60 is composed of a plurality of film likelihoods 60a corresponding to a plurality of pixels, and each film likelihood 60a is a numerical value indicating the probability of being a film.
- the binarizer binarizes the membrane likelihood map 60, thereby generating a temporary membrane image 62 as a binarized image.
- each film likelihood 60a is compared with a threshold value.
- a membrane likelihood 60a that is equal to or greater than the threshold is converted into a pixel 62a having a value of 1, and a membrane likelihood 60a that is less than the threshold is converted into a pixel 62b having a value of 0.
- the pixel 62a is handled as a pixel (film pixel) constituting the film.
- the threshold value can be variably set by the user. The setting may be automated. For example, the threshold value may be changed while observing the temporary membrane image 62.
- FIG. 5 illustrates a work window displayed by the machining tool unit.
- the work window 100 includes a display image 102.
- the display image 102 is a composite image (composite image) composed of a mask image (work target image) corresponding to the depth selected by the user and the original image.
- An original image as a gray scale image constitutes a background image, and a temporary membrane image (work target image) as a color image (for example, a blue image) is superimposed and displayed on the background image.
- the illustrated display image 102 shows brain tissue, in which cross sections of a plurality of cells appear. At the same time, a cross section of an organelle (such as mitochondria) in the cell also appears.
- the display image 102 described above is displayed.
- the tab 105 is selected, only the original image as a gray scale image is displayed.
- the tab 106 is selected, only the film likelihood map (film likelihood image) as a gray scale image is displayed.
- a tab for displaying only a mask image as a work target image may be added.
- a membrane likelihood map may be used as a background image, and a temporary membrane image may be superimposed on the background image.
- the depth selection tool 108 is a display element (operation element) for selecting a specific depth (display depth) in the Z direction. It consists of a Z-axis symbol 108b representing the Z-axis and a marker 108a as a slider that slides along the Z-axis symbol 108b. It is possible to select the desired depth by moving the marker 108a. According to such a depth selection tool 108, it is possible to obtain an advantage that it is easy to intuitively recognize the depth to be selected and the depth change amount.
- the left end point of the Z-axis symbol 108b corresponds to zero depth, and the right end point of the Z-axis symbol 108b corresponds to the maximum depth. Depth selection tools having other forms may be employed.
- the depth input field 114 is a field for directly specifying the depth as a numerical value. The depth currently selected according to the position of the marker 108a may be displayed as a numerical value in the depth input field 114.
- the transparency adjustment tool 110 is a tool for adjusting the transparency (display weight) of the color temporary membrane image (work target image) displayed in a synthesized state in a state where the display image 102 is displayed. For example, when the marker 110a is moved to the left side, the display weight of the color mask image is reduced, the transparency is increased, and the original image is dominantly displayed. On the contrary, when the marker 110a is moved to the right side, the display weight of the color mask image increases, the transparency decreases, and the color mask image is displayed more clearly.
- the superimposition display tool 112 is an image (synthesized image, original image or film likelihood map) adjacent to the currently displayed image (synthesized image, original image or film likelihood map) on the shallow side in the depth direction.
- the operation is performed when an image (a composite image, an original image, or a film likelihood map) adjacent to the deep side in the depth direction is displayed in an overlapping manner.
- the marker 112a is moved to the left side, the display weight for the image adjacent to the shallow side increases.
- the marker 112a is moved to the right side, the display weight for the image adjacent to the deep side increases.
- Three or more images may be superimposed and displayed. Of course, if too many images are superimposed, the image content becomes too complex, so it is desirable to overlay a small number of images. Such superposition display makes it easy to obtain depth information.
- the button row 115 includes a plurality of virtual buttons 116, 118, 120, 121, 122, 126.
- the button 116 is a display element operated when performing image zoom (enlargement or reduction).
- the button 118 is a display element that is operated when the pen tool is used. When the button 118 is turned on, the shape of the cursor is changed to a pen shape, and a film pixel can be added using the cursor shape. It is also possible to change the pen size.
- the button 120 is a display element that is operated when using an eraser. When the button 120 is turned on, the shape of the cursor is changed to an eraser shape, and the film pixel can be deleted using the cursor shape. It is also possible to change the size of the eraser.
- the button 121 is a display element that is operated when painting. If any area is designated after the button 121 is turned on, the area is filled. It is also possible to select an arbitrary function from a plurality of functions prepared for painting (or labeling) by operating the button 121.
- the color palette 124 is displayed. For example, a color selected from the color palette is given to the painted area. Thereby, the area is colored by the selected color.
- Each color is associated with an object number. If the same color, that is, the same object number is given to a plurality of regions across the layers, a three-dimensional lumen region in a specific cell is defined by these regions.
- the button 126 is a button for black and white reversal. When it is operated, a portion displayed in black in the displayed image is displayed in white, and conversely, a portion displayed in white is displayed in black.
- FIG. 6 shows a temporary membrane image 180 before correction and a temporary membrane image 182 after correction.
- the temporary film image 180 before correction includes film break portions 184 and 188.
- the corrected temporary membrane image 182 as shown by reference numerals 186 and 190, the film interruption portion is repaired.
- the temporary membrane image 180 before correction includes non-membrane portions 192 and 196.
- those non-membrane portions have disappeared. According to the corrected mask image 182, it is possible to accurately perform labeling, and consequently, it is possible to improve the quality of the three-dimensional labeling data.
- FIG. 7 illustrates a three-dimensional splicing process included in the labeling process.
- a plurality of temporary membrane images D1 to D4 arranged in the Z direction are shown.
- a temporary membrane image D1 corresponding to the depth Zi is a work target image. This is a reference image in the processing described below.
- the representative point is specified for the region R1 included in the mask image D1 that is the reference image.
- a center point, a center of gravity point, etc. are specified as representative points.
- a perpendicular C passing through the representative point is defined. From the reference image, for example, N masked membrane images are referred to on the deep side, and each region through which the perpendicular C passes is specified in these images. And the same label is provided with respect to area
- the above processing may be applied to N temporary film images on the shallow side from the reference image. The result of automatic labeling is usually visually confirmed by the user.
- the region R1 (outer edge) included in the mask image D1 which is the reference image is projected onto the mask image D2, and the projection region R1a is defined.
- a region R2 that most overlaps the projection region R1a is specified on the mask image D2.
- the region R2 is projected onto the mask image D3, and the projection region R2a is defined.
- a region R3 that most overlaps the projection region R2a is specified on the mask image D3.
- a projection region R3a is defined on the mask image D4, and the region R4 is specified based on the projection region R3a.
- the same label is assigned to the region R1 and all of the regions R2, R3, and R4 identified from the region R1.
- the projection source area is sequentially updated when the layer is changed, but it may be fixed.
- the region R1 may be projected onto the mask images D2, D3, D4.
- the connection destination is searched from the reference image to one side in the Z direction, but the connection destination may be searched from the reference image to both sides in the Z direction.
- the search range may be selected by the user. In any case, the result of automatic labeling is usually visually confirmed by the user. In that case, the work window shown in FIG. 5 is used.
- the three-dimensional connection process described above is an example, and a three-dimensional connection process other than the above may be employed.
- one or a plurality of feature amounts for each region can be used.
- As the feature amount of the area an area, a form, a perimeter, a barycentric point, a luminance histogram, a texture, or the like may be used.
- a superposition area, a distance between representative points, or the like may be used as a feature amount between regions.
- FIG. 8 schematically shows exaggeratedly the change in the observation depth range and the change in the SEM image caused by the change in the applied voltage.
- Reference numerals 230, 232, and 234 indicate three different observation conditions.
- the observation condition 230 is a condition for setting the acceleration voltage to a low voltage.
- the observation condition 232 is a condition in which the acceleration voltage is a medium voltage.
- the observation condition 234 is a condition for increasing the acceleration voltage.
- the upper part (A) of FIG. 8 shows the relationship between the cross section of the sample piece 236 and the electron beams 244, 246, 248. Each cross section includes a film portion 238 extending parallel to the Z direction and a film portion 240 extending in a direction oblique to the Z direction.
- a SEM image is shown in the lower part (B) of FIG.
- the object to be observed by the electron beam 244 is exclusively the surface layer of the sample section 236. Therefore, on the SEM image, images 238a and 240a representing the surface layers of the two film portions 238 and 240 appear.
- the observation condition 232 is selected, the object to be observed by the electron beam 246 extends to the intermediate layer of the sample slice 236. Therefore, on the SEM image, an image (intermediate integrated image) 238b representing the intermediate layer of the film portion 238 appears, and a deep image 240c appears in the vicinity of the image 240b for the film portion 240.
- the object to be observed by the electron beam 248 extends to the entire sample section 236 in the illustrated example. Therefore, on the SEM image, an image (entire integrated image) 238c representing the entire film portion 238 appears, and a wide depth image 240e appears in the vicinity of the image 240d for the film portion 240.
- the membrane estimator In order to avoid or reduce damage to the sample, it is required to observe the sample at a low acceleration voltage.On the other hand, when the membrane is estimated by the membrane estimator, the depth image of the cell membrane is also correctly estimated as part of the cell membrane. Is required. It is necessary to firmly instruct the membrane estimator that the deep image of the cell membrane is a part of the cell membrane. That is, in the learning stage of the film estimator, it is necessary to give the film estimator an image including many deep images. From such a viewpoint, in the embodiment, in the learning process of the film estimator, a plurality of high voltage images are provided to the film estimator together with the plurality of low voltage images. Their ratio may be 1: 1, but the ratio can vary depending on the situation.
- contrast enhancement 252 as moya enhancement is applied to the second original image (high voltage image) 250.
- segmentation 256 is performed on the image 254 after contrast enhancement.
- many small regions are set in the image 258 after segmentation.
- a membrane image 262 is created by painting a plurality of small regions to which a membrane label is attached.
- the film image 262 may be corrected after painting.
- the image may be corrected in an earlier stage.
- the film image 262 created in this way is used as the second correct image.
- membrane image 262 shown by FIG. 10 a white part is a film
- both the low voltage image and the high voltage image are given to the film estimator in the learning stage of the film estimator.
- the following comparative example gives only a low voltage image to the film estimator in the learning stage of the film estimator.
- the input image 270 is shown on the left side of FIG. 11, and the correct image 272 corresponding to the input image 270 is shown on the right side of FIG.
- the black part is the film part.
- a film likelihood map 274 according to the comparative example is shown on the left side of FIG. 12.
- a membrane likelihood map 276 according to the embodiment is shown on the right side of FIG. In these film likelihood maps 274 and 276, the magnitude of the film likelihood is expressed in black.
- the film portions A1, B1, C1, and D1 that are thinly expressed on the film likelihood map 274 are the film portions A2 that are expressed deeply on the film likelihood map 276. It has changed to B2, C2, and D2.
- the film portions A1 and B1 (and the film portions A2 and B2) are considered to be portions including a deep image.
- the land error for the comparative example was 0.012, while the land error for the embodiment was 0.009.
- the land error is defined by 1- (Rand Index), where Rand Index is a known index indicating the degree of coincidence or similarity between the evaluation target image and the correct image.
- FIG. 13 shows a second configuration example of the main body.
- symbol is attached
- the main body 280 includes a first image processing unit 200 and a second image processing unit 281.
- the second image processing unit 281 includes a machine learning type film estimator 282, a binarizer 284, and a correction unit 286.
- the film estimator 282 is composed of CNN in the illustrated example.
- the film estimator 282 applies a film estimation process to the second original image (high voltage image) 36H as the input image 208, and outputs a film likelihood map.
- the film estimator 282 is a learned estimator that has undergone learning using a high-voltage image.
- the binarizer 284 generates a temporary membrane image from the membrane likelihood map.
- the correction unit 286 has the same function as that of the correction unit 52, accepts user correction on the mask image, and generates the second correct image 224 thereby.
- the second configuration example uses a membrane estimator 282 that is different from the membrane estimator 42 in generating the second correct image.
- two observation conditions are used, but three or more observation conditions may be used.
- CNN is used as the film estimator, other machine learning type film estimators may be used.
Landscapes
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Chemical & Material Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Immunology (AREA)
- Analytical Chemistry (AREA)
- Biochemistry (AREA)
- Pathology (AREA)
- Molecular Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Biomedical Technology (AREA)
- Hematology (AREA)
- Theoretical Computer Science (AREA)
- Urology & Nephrology (AREA)
- Food Science & Technology (AREA)
- Medicinal Chemistry (AREA)
- Image Analysis (AREA)
- Analysing Materials By The Use Of Radiation (AREA)
- Investigating Or Analysing Biological Materials (AREA)
- Apparatus Associated With Microorganisms And Enzymes (AREA)
Abstract
In an analysis process, a membrane estimator (42) subjects an original image (38L) serving as a low voltage image to a process for estimating a cell membrane, and outputs a membrane likelihood map (60). In a learning process, a first original image (36L) serving as a low voltage image, and a corresponding first correct answer image (223, 64A), and a second original image (36H) serving as a high voltage image, and a corresponding second correct answer image (224) are input into the membrane estimator (42).
Description
本開示は生物組織画像処理システム及び機械学習方法に関する。
The present disclosure relates to a biological tissue image processing system and a machine learning method.
生物組織の三次元構造を解析し又はそれをイメージングするための手法として、三次元顕微鏡法が知られている。三次元顕微鏡法においては、一般に、電子顕微鏡が用いられる。例えば、走査型電子顕微鏡(Scanning Electron Microscope)を用いた三次元顕微鏡法として、Focused Ion Beam SEM(FIB-SEM)法、Serial Block-Face SEM(SBF-SEM)法、及び、連続切片SEM(Serial Section SEM)法(非特許文献1を参照)が提案されている。連続切片SEM法は、アレイトモグラフィ(Array-tomography)法とも呼ばれている。
3D microscopy is known as a technique for analyzing or imaging the 3D structure of biological tissue. In three-dimensional microscopy, an electron microscope is generally used. For example, three-dimensional microscopy using a scanning electron microscope (Scanning Electron Microscope) includes Focused Ion Beam SEM (FIB-SEM) method, Serial Block-Face SEM (SBF-SEM) method, and serial section SEM (Serial Section (SEM) method (see Non-Patent Document 1) has been proposed. The continuous section SEM method is also called an array tomography method.
連続切片SEM法では、生物組織の試料から、深さ方向に連なる複数の試料切片(極薄片)が切り出され、それらが基板上に配列される。基板上の各試料切片が走査型電子顕微鏡により順次観察され、これにより複数の画像が取得される。取得された複数の画像に基づいて、生物組織に含まれる特定の器官(細胞、ミトコンドリア、核等)の三次元構造が解析され、あるいは、それがイメージングされる。連続切片SEM法によれば、既に観察した試料切片を再び観察することが可能である。
In the continuous section SEM method, a plurality of sample sections (ultra-thin pieces) continuous in the depth direction are cut out from a sample of a biological tissue, and they are arranged on a substrate. Each sample section on the substrate is sequentially observed with a scanning electron microscope, whereby a plurality of images are acquired. Based on the acquired images, the three-dimensional structure of a specific organ (cell, mitochondria, nucleus, etc.) contained in the biological tissue is analyzed or imaged. According to the continuous section SEM method, it is possible to observe a sample section that has already been observed again.
電子顕微鏡による生物組織の観察により得られた画像の処理において、細胞、細胞膜、細胞内の小器官等の注目要素(Target Component)を検出又は識別するために、機械学習型の推定器を利用することが考えられる。その場合、推定器での推定精度が高まるように、推定器を学習させることが望まれる。
In the processing of images obtained by observation of biological tissues with an electron microscope, a machine learning type estimator is used to detect or identify a target element (Target Component) such as a cell, a cell membrane, or an organelle in the cell. It is possible. In that case, it is desirable to train the estimator so that the estimation accuracy of the estimator is increased.
電子顕微鏡により生成された画像の内容は、加速電圧等の観察条件によって変化する。その観点から言えば、解析用画像の取得時における観察条件(解析時の観察条件)に対して、学習用画像の取得時における観察条件(学習時の観察条件)を合わせるべきであるという考え方が生じる。しかし、推定器における推定対象が生物組織中において深さ方向に有意な厚みを有している場合、上記の考え方は必ずしも成り立たず、学習時の観察条件に多様性をもたせた方がよい場合があるものと考えられる。
The content of the image generated by the electron microscope varies depending on observation conditions such as acceleration voltage. From that perspective, there is an idea that the observation conditions at the time of acquisition of the learning image (observation conditions at the time of learning) should be matched with the observation conditions at the time of acquisition of the image for analysis (observation conditions at the time of analysis). Arise. However, if the estimation target in the estimator has a significant thickness in the biological tissue, the above idea does not necessarily hold, and it may be better to give diversity to the observation conditions during learning. It is thought that there is.
本開示の目的は、生物組織中の注目要素を推定する機械学習型の推定器を有する生物組織画像処理システムにおいて、注目要素の推定精度を高めることにある。あるいは、本開示の目的は、生物組織中の注目要素を推定する推定器の機械学習過程において、注目要素の推定精度が高まるように推定器に学習を行わせることにある。
An object of the present disclosure is to improve the estimation accuracy of a target element in a biological tissue image processing system having a machine learning type estimator that estimates a target element in a biological tissue. Alternatively, an object of the present disclosure is to cause the estimator to perform learning so that the estimation accuracy of the attention element is increased in the machine learning process of the estimator that estimates the attention element in the biological tissue.
本開示に係る生物組織画像処理システムは、生物組織である解析対象の観察により画像を生成する電子顕微鏡と、前記画像に対して前記解析対象に含まれる注目要素を推定する処理を適用する機械学習型の推定器と、を含み、前記推定器は学習用元画像セットによる学習を経た学習済み推定器であり、前記学習用元画像セットには、第1観察条件の下で電子顕微鏡により生物組織である学習対象試料を観察することにより生成された第1元画像と、第2観察条件の下で顕微鏡により前記学習対象試料を観察することにより生成された第2元画像と、が含まれ、前記第1観察条件は、前記解析対象試料を観察する際に電子顕微鏡に対して設定される条件であり、前記第2観察条件は、前記第1観察条件とは異なる深さ範囲にわたって観察を行える条件である、ことを特徴とするものである。
A biological tissue image processing system according to the present disclosure includes an electron microscope that generates an image by observing an analysis target that is a biological tissue, and machine learning that applies processing for estimating an element of interest included in the analysis target to the image An estimator of the type, and the estimator is a learned estimator that has undergone learning using a learning original image set, and the learning original image set is subjected to biological tissue using an electron microscope under a first observation condition. A first original image generated by observing the learning target sample, and a second original image generated by observing the learning target sample with a microscope under a second observation condition, The first observation condition is a condition set for an electron microscope when observing the analysis target sample, and the second observation condition is to perform observation over a depth range different from the first observation condition. Is that condition, it is characterized in.
本開示に係る機械学習方法は、電子顕微鏡により生物組織である解析対象試料を観察することにより生成された画像に対して前記解析対象試料に含まれる注目要素を推定する処理を適用する機械学習型の推定器を含む生物組織画像処理システムにおいて、前記推定器に学習をさせるための機械学習方法であって、前記推定器に対して、第1加速電圧の下で電子顕微鏡により生物組織としての学習対象試料を観察することにより生成された学習用の第1元画像、及び、それに対応する学習用の第1正解画像を与える工程と、前記推定器に対して、前記第1加速電圧よりも高い第2加速電圧の下で電子顕微鏡により前記学習対象試料を観察することにより生成された学習用の第2元画像、及び、それに対応する学習用の第2正解画像を与える工程と、を含む、ことを特徴とするものである。
A machine learning method according to the present disclosure applies a process of estimating a target element included in an analysis target sample to an image generated by observing the analysis target sample that is a biological tissue using an electron microscope. In a biological tissue image processing system including an estimator, a machine learning method for causing the estimator to learn, wherein the estimator learns as a biological tissue with an electron microscope under a first acceleration voltage. A step of providing a first original image for learning generated by observing a target sample and a first correct answer image for learning corresponding thereto, and higher than the first acceleration voltage for the estimator A process for providing a second original image for learning generated by observing the learning target sample with an electron microscope under a second acceleration voltage, and a second correct image for learning corresponding thereto. When, including, it is characterized in.
上記方法は、ハードウエアの機能又はソフトウエアの機能として実現され、後者の場合、その機能を実行するプログラムが、ネットワーク又は可搬型記憶媒体を介して、情報処理装置へインストールされる。情報処理装置の概念には、パーソナルコンピュータ、電子顕微鏡システム等含まれる。
The above method is realized as a hardware function or a software function. In the latter case, a program for executing the function is installed in the information processing apparatus via a network or a portable storage medium. The concept of the information processing apparatus includes a personal computer, an electron microscope system, and the like.
(A)実施形態の概要
実施形態に係る生物組織画像処理システムは、電子顕微鏡、及び、機械学習型の推定器を含む。電子顕微鏡は、生物組織である解析対象試料の観察により画像を生成する装置であり、望ましくは、それは走査型電子顕微鏡である。推定器は、電子顕微鏡により生成された画像に対して解析対象試料に含まれる注目要素を推定する処理を適用するものであり、その推定器は、学習用元画像セットによる学習を経た学習済み推定器である。推定する対象は、より正確には、注目要素の像である。学習用元画像セットには、第1観察条件の下で電子顕微鏡により学習対象試料を観察することにより生成された第1元画像と、第2観察条件の下で顕微鏡により学習対象試料を観察することにより生成された第2元画像と、が含まれる。第1観察条件は、解析対象試料を観察する際に電子顕微鏡に対して設定される条件であり、その条件は、学習対象試料を観察する際にも電子顕微鏡に対して設定される。第2観察条件は、第1観察条件とは異なる深さ範囲にわたって観察を行える条件であり、その条件は、学習対象試料を観察する際に電子顕微鏡に対して設定される条件である。 (A) Outline of Embodiment A biological tissue image processing system according to an embodiment includes an electron microscope and a machine learning type estimator. The electron microscope is an apparatus that generates an image by observing a sample to be analyzed which is a biological tissue, and preferably it is a scanning electron microscope. The estimator applies a process of estimating the element of interest included in the sample to be analyzed to the image generated by the electron microscope, and the estimator is a learned estimation that has been learned by the original image set for learning. It is a vessel. More precisely, the object to be estimated is an image of the element of interest. In the learning original image set, the first original image generated by observing the learning target sample with the electron microscope under the first observation condition and the learning target sample with the microscope under the second observation condition are observed. 2nd original image produced | generated by this. The first observation condition is a condition set for the electron microscope when observing the analysis target sample, and the condition is also set for the electron microscope when observing the learning target sample. The second observation condition is a condition under which observation can be performed over a depth range different from the first observation condition, and the condition is a condition set for the electron microscope when observing the learning target sample.
実施形態に係る生物組織画像処理システムは、電子顕微鏡、及び、機械学習型の推定器を含む。電子顕微鏡は、生物組織である解析対象試料の観察により画像を生成する装置であり、望ましくは、それは走査型電子顕微鏡である。推定器は、電子顕微鏡により生成された画像に対して解析対象試料に含まれる注目要素を推定する処理を適用するものであり、その推定器は、学習用元画像セットによる学習を経た学習済み推定器である。推定する対象は、より正確には、注目要素の像である。学習用元画像セットには、第1観察条件の下で電子顕微鏡により学習対象試料を観察することにより生成された第1元画像と、第2観察条件の下で顕微鏡により学習対象試料を観察することにより生成された第2元画像と、が含まれる。第1観察条件は、解析対象試料を観察する際に電子顕微鏡に対して設定される条件であり、その条件は、学習対象試料を観察する際にも電子顕微鏡に対して設定される。第2観察条件は、第1観察条件とは異なる深さ範囲にわたって観察を行える条件であり、その条件は、学習対象試料を観察する際に電子顕微鏡に対して設定される条件である。 (A) Outline of Embodiment A biological tissue image processing system according to an embodiment includes an electron microscope and a machine learning type estimator. The electron microscope is an apparatus that generates an image by observing a sample to be analyzed which is a biological tissue, and preferably it is a scanning electron microscope. The estimator applies a process of estimating the element of interest included in the sample to be analyzed to the image generated by the electron microscope, and the estimator is a learned estimation that has been learned by the original image set for learning. It is a vessel. More precisely, the object to be estimated is an image of the element of interest. In the learning original image set, the first original image generated by observing the learning target sample with the electron microscope under the first observation condition and the learning target sample with the microscope under the second observation condition are observed. 2nd original image produced | generated by this. The first observation condition is a condition set for the electron microscope when observing the analysis target sample, and the condition is also set for the electron microscope when observing the learning target sample. The second observation condition is a condition under which observation can be performed over a depth range different from the first observation condition, and the condition is a condition set for the electron microscope when observing the learning target sample.
上記構成によれば、特別な学習過程を経た学習済み推定器の利用により、注目要素についての推定精度を高められる。例えば、第1観察条件では比較的にあまり生じない像(例えば深部像)が注目組織の一部として正しく推定され易くなる。
According to the above configuration, the estimation accuracy for the element of interest can be increased by using a learned estimator that has undergone a special learning process. For example, an image (for example, a deep image) that does not relatively occur under the first observation condition is likely to be correctly estimated as a part of the target tissue.
解析対象試料及び学習対象試料は、一般に、同一の(同種の)生物組織であるが、それらは通常、物理的には別のものである。解析対象試料及び学習対象試料は、一般に、同じ電子顕微鏡により観察されるが、それらが、異なる電子顕微鏡により観察されてもよい。注目要素は、特定の組織構成要素であり、実施形態において、注目要素は細胞膜である。細胞それ自体、細胞膜によって囲まれた細胞内腔(細胞質)、細胞内の小器官(オルガネラ(Organelle))等が注目要素とされてもよい。機械学習型の推定器は、例えばCNNで構成されるが、他のタイプの機械学習型の推定器が利用されてもよい。実施形態においては、連続切片SEM法(アレイトモグラフィ法)により複数の画像が取得されているが、FIB-SEM法、SBF-SEM法、その他の手法により複数の画像が取得されてもよい。上記構成は、三次元顕微鏡法の他、二次元顕微鏡法等にも適用され得る。
The analysis target sample and the learning target sample are generally the same (same type) biological tissue, but they are usually physically different. The analysis target sample and the learning target sample are generally observed with the same electron microscope, but they may be observed with different electron microscopes. The element of interest is a specific tissue component, and in embodiments, the element of interest is a cell membrane. The cell itself, the cell lumen (cytoplasm) surrounded by the cell membrane, the organelle in the cell (Organelle), and the like may be considered as elements of interest. The machine learning type estimator is configured by CNN, for example, but other types of machine learning type estimators may be used. In the embodiment, a plurality of images are acquired by the continuous section SEM method (array tomography method), but a plurality of images may be acquired by the FIB-SEM method, the SBF-SEM method, and other methods. The above configuration can be applied to two-dimensional microscopy as well as three-dimensional microscopy.
実施形態において、第2観察条件は、第1観察条件よりも大きな深さ範囲にわたって観察を行える条件である。観察条件の切り替えに際して、加速電圧、照射電流その他が変更されてもよい。
In the embodiment, the second observation condition is a condition under which observation can be performed over a greater depth range than the first observation condition. When switching the observation conditions, the acceleration voltage, irradiation current, and the like may be changed.
実施形態において、第1観察条件は、電子顕微鏡の加速電圧として第1加速電圧を設定する条件であり、第2観察条件は、電子顕微鏡の加速電圧として第1加速電圧よりも高い第2加速電圧を設定する条件である。加速電圧を高くすると、観察対象物におけるより深いところからの情報が得られ易くなる。換言すれば、より深いところに存在している物体が画像化され易くなる。第1元画像及び第2元画像を利用して推定器に学習を行わせれば、推定器の入力画像が第1観察条件下で生成された画像であっても、注目組織の深部像が検出され易くなる。解析過程において第1観察条件下で解析対象試料の観察を行えば、解析対象試料の損傷等を防止又は軽減できる。
In the embodiment, the first observation condition is a condition for setting the first acceleration voltage as the acceleration voltage of the electron microscope, and the second observation condition is a second acceleration voltage higher than the first acceleration voltage as the acceleration voltage of the electron microscope. Is a condition for setting. When the acceleration voltage is increased, information from a deeper position in the observation object can be easily obtained. In other words, an object existing deeper can be easily imaged. If the estimator is trained using the first original image and the second original image, the deep image of the tissue of interest is detected even if the input image of the estimator is an image generated under the first observation condition. It becomes easy to be done. If the sample to be analyzed is observed under the first observation condition in the analysis process, damage to the sample to be analyzed can be prevented or reduced.
実施形態において、生物組織画像処理システムは、第2元画像に基づいて第2正解画像を生成する正解画像作成部を含む。第2元画像用の正解画像作成部を設けておけば、正解画像の作成に際して、第2元画像に適した画像処理をその画像に対して適用し易くなる。実施形態において、正解画像作成部は、第2元画像において注目要素の深部像を強調する強調部を含む。この構成によれば、注目要素(注目要素像)の一部である深部像が学習対象となり易くなる。実施形態において、注目要素は細胞膜であり、注目要素の深部像は細胞膜の深部像である。例えば、細胞の三次元構造を解析する場合、生物組織切片の内部に存在する細胞膜までを推定又は検出することが望まれる。上記構成はそのような要請に応えるものである。
In the embodiment, the biological tissue image processing system includes a correct image creation unit that generates a second correct image based on the second original image. If the correct image creation unit for the second original image is provided, image processing suitable for the second original image can be easily applied to the image when creating the correct image. In the embodiment, the correct image creation unit includes an enhancement unit that emphasizes the deep image of the element of interest in the second original image. According to this configuration, the deep part image that is a part of the target element (target element image) can be easily learned. In the embodiment, the element of interest is a cell membrane, and the deep image of the element of interest is a deep image of the cell membrane. For example, when analyzing a three-dimensional structure of a cell, it is desired to estimate or detect even a cell membrane present inside a biological tissue section. The above configuration meets such a demand.
実施形態において、正解画像作成部は、強調部による処理を経た画像を複数の小領域に分割するセグメンテーションを実行するセグメンテーション部と、セグメンテーション後の画像において、細胞膜に相当する複数の小領域に対してラベリングを行って第2正解画像を生成するアノテーション部と、を含む。ラベリングに先立ってセグメンテーション(小領域分割)を行えば、ラベリングの作業が楽になり、あるいは、ラベリングが正確となる。
In the embodiment, the correct image creation unit is configured to perform segmentation for dividing the image that has been processed by the enhancement unit into a plurality of small regions, and a plurality of small regions corresponding to cell membranes in the segmented image. An annotation unit that performs labeling to generate a second correct image. If segmentation (small area division) is performed prior to labeling, the labeling work becomes easier or the labeling becomes accurate.
実施形態において、正解画像作成部は機械学習型の第2推定器を含む。この構成は第1推定器のための正解画像の作成に際して別の推定器を利用するものである。第2推定器によって第2元画像中の注目要素を推定し、その推定結果に基づいて第2正解画像が作成される。
In the embodiment, the correct image creation unit includes a machine learning type second estimator. This configuration uses another estimator when creating a correct image for the first estimator. The element of interest in the second original image is estimated by the second estimator, and a second correct image is created based on the estimation result.
実施形態に係る機械学習方法は、推定器に対して、第1加速電圧の下で電子顕微鏡により生物組織としての学習対象試料を観察することにより生成された学習用の第1元画像、及び、それに対応する学習用の第1正解画像を与える工程と、推定器に対して、第1加速電圧よりも高い第2加速電圧の下で電子顕微鏡により学習対象試料を観察することにより生成された学習用の第2元画像、及び、それに対応する学習用の第2正解画像を与える工程と、を含む。
The machine learning method according to the embodiment includes a first original image for learning generated by observing a learning target sample as a biological tissue with an electron microscope under a first acceleration voltage with respect to an estimator, and A step of providing a first correct image for learning corresponding thereto, and learning generated by observing a learning target sample with an electron microscope under a second acceleration voltage higher than the first acceleration voltage with respect to the estimator And providing a second original image for learning and a second correct image for learning corresponding thereto.
実施形態においては、推定器の機械学習過程では、第1元画像及び第1正解画像からなる第1画像ペアが第1教師データとして推定器に入力され、第2元画像及び第2正解画像からなる第2画像ペアが第2教師データとして推定器に入力される。第1正解画像は、第1元画像から手作業により作成され、又は、第1元画像に対する推定器の推定結果を修正することにより作成される。第2正解画像は、実施形態において、正解画像作成部によって第2元画像から作成される。正解画像作成部によって、第1正解画像及び第2正解画像の両方が作成されてもよい。
In the embodiment, in the machine learning process of the estimator, the first image pair including the first original image and the first correct image is input to the estimator as the first teacher data, and the second original image and the second correct image are used. The second image pair is input to the estimator as second teacher data. The first correct image is created manually from the first original image, or is created by correcting the estimation result of the estimator for the first original image. In the embodiment, the second correct image is created from the second original image by the correct image creating unit. Both the first correct image and the second correct image may be generated by the correct image creating unit.
(B)実施形態の詳細
図1には、実施形態に係る生物組織画像処理システムが示されている。図示された生物組織画像処理システム10は、生物組織について、三次元構造の解析やイメージングを行うためのシステムである。この生物組織画像処理システムを利用して、例えば、人体又は動物の脳内の神経細胞を三次元的に表現した画像が生成される。生物中の任意の組織、器官、その他が解析の対象になり得る。 (B) Details of Embodiment FIG. 1 shows a biological tissue image processing system according to an embodiment. The illustrated biological tissueimage processing system 10 is a system for analyzing and imaging a three-dimensional structure of a biological tissue. Using this biological tissue image processing system, for example, an image that three-dimensionally represents nerve cells in the brain of a human body or an animal is generated. Any tissue, organ, etc. in the organism can be the subject of analysis.
図1には、実施形態に係る生物組織画像処理システムが示されている。図示された生物組織画像処理システム10は、生物組織について、三次元構造の解析やイメージングを行うためのシステムである。この生物組織画像処理システムを利用して、例えば、人体又は動物の脳内の神経細胞を三次元的に表現した画像が生成される。生物中の任意の組織、器官、その他が解析の対象になり得る。 (B) Details of Embodiment FIG. 1 shows a biological tissue image processing system according to an embodiment. The illustrated biological tissue
図1に示す構成例において、生物組織画像処理システム10は、試料前処理装置12、連続切片作成装置14、走査型電子顕微鏡(SEM)16、及び、生物組織画像処理装置18によって構成されている。
In the configuration example shown in FIG. 1, the biological tissue image processing system 10 includes a sample pretreatment device 12, a continuous section creation device 14, a scanning electron microscope (SEM) 16, and a biological tissue image processing device 18. .
試料前処理装置12は、生体から取り出された組織20に対して前処理を行う装置であり、又は、その前処理のための各種の器具に相当する。前処理として、固定処理、染色処理、導電処理、樹脂包埋処理、整形処理等が挙げられる。それらの全部又は一部が、必要に応じて、実施される。染色処理においては四酸化オスミウム(osmium tetroxide)、酢酸ウラン(uranium acetate)、クエン酸鉛(lead citrate)等が用いられてもよい。染色処理が以下に説明する各試料切片に対して行われてもよい。前処理に含まれる一部又は全部の工程が手作業によって行われてもよい。
The sample pretreatment device 12 is a device that performs pretreatment on the tissue 20 extracted from a living body, or corresponds to various instruments for the pretreatment. Examples of the pretreatment include fixing treatment, dyeing treatment, conductive treatment, resin embedding treatment, and shaping treatment. All or some of them are implemented as necessary. In the staining treatment, osmium tetroxide, uranium acetate, lead citrate, or the like may be used. A staining process may be performed on each sample section described below. Some or all of the steps included in the pretreatment may be performed manually.
連続切片作成装置14は、SEM16の外部に設けられ、あるいは、SEM16の内部に設けられる。連続切片作成装置14により、前処理後のキュービック状の試料から、深さ方向(Z方向)に並ぶ複数の試料切片24が切り出される。その際には、ウルトラミクロトーム(Ultramicrotome)等の装置が利用されてもよい。その作業が手作業で行われてもよい。複数の試料切片24により、試料切片群22が構成される。実際には、切り出された複数の試料切片24は、基板28上に所定の配列で配置される。基板28は、例えば、ガラス基板、シリコーン基板である。図1においては、基板28上に、2つの試料切片列からなる試料切片アレイ22Aが構成されているが、それは例示に過ぎない。基板28及び試料切片アレイ22Aにより、試料ユニット26が構成される。
The continuous section preparation device 14 is provided outside the SEM 16 or inside the SEM 16. A plurality of sample sections 24 arranged in the depth direction (Z direction) are cut out from the cubic sample after the pretreatment by the continuous section creating apparatus 14. In this case, an apparatus such as an ultramicrotome may be used. The work may be performed manually. A plurality of sample sections 24 constitute a sample section group 22. Actually, the plurality of sample sections 24 cut out are arranged on the substrate 28 in a predetermined arrangement. The substrate 28 is, for example, a glass substrate or a silicone substrate. In FIG. 1, the sample section array 22 </ b> A composed of two sample section rows is configured on the substrate 28, but this is merely an example. A sample unit 26 is configured by the substrate 28 and the sample section array 22A.
ちなみに、個々の試料切片24における縦及び横のサイズは、例えば、nmオーダー又はμmオーダーである。それ以上のサイズ(例えばmmオーダーのサイズ)を有する試料切片24が作製されてもよい。個々の試料切片24の厚み(Z方向のサイズ)は、例えば、数nm~数百nmの範囲内にあり、実施形態においては、その厚みは例えば30~70nmの範囲内にある。本願明細書において挙げる数値はいずれも例示である。
Incidentally, the vertical and horizontal sizes of the individual sample sections 24 are, for example, on the order of nm or μm. A sample section 24 having a larger size (for example, a size on the order of mm) may be produced. The thickness (size in the Z direction) of each sample section 24 is, for example, in the range of several nm to several hundred nm, and in the embodiment, the thickness is in the range of, for example, 30 to 70 nm. Any numerical value given in this specification is an example.
推定器の学習過程で観察される試料は学習用試料つまり学習対象試料である。学習対象試料から複数の組織切片が作製される。解析過程で観察される試料は解析対象試料である。解析対象試料から複数の組織切片が作製される。図1においては、学習対象試料と解析対象試料の両者がいずれも組織20として表現されている。また、学習対象試料から切り出された各試料切片及び解析対象試料から切り出された各試料切片がいずれも試料切片24として表現されている。なお、学習対象試料と解析対象試料は通常、別々のものであるが、それらが同一のものであってもよい。例えば、解析過程と学習過程とが並列的に実行されてもよい。
The sample observed in the learning process of the estimator is a learning sample, that is, a learning target sample. A plurality of tissue sections are prepared from the learning target sample. The sample observed in the analysis process is the sample to be analyzed. A plurality of tissue sections are prepared from the sample to be analyzed. In FIG. 1, both the learning target sample and the analysis target sample are expressed as a tissue 20. Each sample slice cut out from the learning target sample and each sample slice cut out from the analysis target sample are both expressed as a sample slice 24. Note that the learning target sample and the analysis target sample are usually separate, but they may be the same. For example, the analysis process and the learning process may be executed in parallel.
SEM16は、電子銃、偏向器(走査器)、対物レンズ、試料室、検出器34、制御部204等を有している。試料室内には、試料ユニット26を保持するステージ、及び、そのステージを移動させる移動機構、が設けられている。制御部204により、移動機構の動作つまりステージの移動が制御される。具体的には、試料切片アレイ22Aの中から選択された特定の試料切片24に対して電子ビーム30が照射される。照射位置を走査(例えばラスタースキャン)させながら、各照射位置から放出される反射電子32が検出器34で検出される。これによりSEM画像が形成される。これが試料切片24ごとに実行される。制御部204は、電子ビームを形成するための加速電圧を設定する機能を有する。一般に、加速電圧を上げると、試料切片24の内部におけるより深い位置からの情報が得られ易くなる。
The SEM 16 includes an electron gun, a deflector (scanner), an objective lens, a sample chamber, a detector 34, a control unit 204, and the like. In the sample chamber, a stage for holding the sample unit 26 and a moving mechanism for moving the stage are provided. The control unit 204 controls the operation of the moving mechanism, that is, the movement of the stage. Specifically, the electron beam 30 is irradiated to a specific sample section 24 selected from the sample section array 22A. While scanning the irradiation position (for example, raster scanning), the reflected electrons 32 emitted from each irradiation position are detected by the detector 34. Thereby, an SEM image is formed. This is performed for each sample section 24. The control unit 204 has a function of setting an acceleration voltage for forming an electron beam. In general, when the acceleration voltage is increased, information from a deeper position in the sample section 24 is easily obtained.
なお、Z方向に直交する方向をX方向と定義し、Z方向及びX方向に直交する方向をY方向と定義した場合、望ましくは、X方向観察範囲及びY方向観察範囲が互いに一致するように、各試料切片24における観察範囲(電子ビーム二次元走査範囲)が定められる。反射電子32ではなく、二次電子等が検出されてもよい。
In addition, when the direction orthogonal to the Z direction is defined as the X direction and the direction orthogonal to the Z direction and the X direction is defined as the Y direction, it is desirable that the X direction observation range and the Y direction observation range coincide with each other. The observation range (electron beam two-dimensional scanning range) in each sample section 24 is determined. Instead of the reflected electrons 32, secondary electrons or the like may be detected.
実施形態においては、推定器の学習過程では、制御部204の制御により、試料切片24単位で、第1観察条件としての第1加速電圧(低電圧)、及び、第2観察条件としての第2加速電圧(高電圧)が交互に設定される。すなわち、学習対象試料から切り出された個々の試料切片24が第1加速電圧の下で形成された電子ビーム30Lの走査により観察され(低電圧観察)、これにより低電圧画像(第1元画像)36Lが生成される。また、個々の試料切片24が第2加速電圧の下で形成された電子ビーム30Hの走査により観察され、これにより高電圧画像(第2元画像)36Hが生成される(高電圧観察)。通常、1つの試料切片24に対して低電圧観察及び低電圧観察が連続して実行され、これが試料切片24ごとに繰り返される。その結果、複数の低電圧画像36Lにより低電圧画像スタック35Lが構成され、また、複数の高電圧画像36Hにより高電圧画像スタック35Hが構成される。
In the embodiment, in the learning process of the estimator, under the control of the control unit 204, the first acceleration voltage (low voltage) as the first observation condition and the second as the second observation condition are obtained in units of 24 sample sections. The acceleration voltage (high voltage) is alternately set. That is, the individual sample slices 24 cut out from the learning target sample are observed by scanning with the electron beam 30L formed under the first acceleration voltage (low voltage observation), whereby the low voltage image (first original image). 36L is generated. In addition, the individual specimen sections 24 are observed by scanning with the electron beam 30H formed under the second acceleration voltage, thereby generating a high voltage image (second original image) 36H (high voltage observation). Usually, low-voltage observation and low-voltage observation are continuously performed on one sample section 24, and this is repeated for each sample section 24. As a result, the low voltage image stack 35L is configured by the plurality of low voltage images 36L, and the high voltage image stack 35H is configured by the plurality of high voltage images 36H.
一方、解析過程では、制御部204の制御により、観察条件として第1加速電圧だけが設定される。すなわち、解析対象試料から切り出された個々の試料切片24が第1加速電圧の下で形成された電子ビーム30Lの走査により観察され(低電圧観察)、これにより低電圧画像(解析用元画像)38Lが生成される。複数の低電圧画像38Lにより低電圧画像スタック37Lが構成される。第1加速電圧は、例えば、1keV又は2keVであり、第2加速電圧は、例えば、3keV、5keV又は7keVである。試料、切片の厚み、注目組織、観察目的その他の事情を考慮して第1加速電圧及び第2加速電圧を定めるのが望ましい。
On the other hand, in the analysis process, only the first acceleration voltage is set as an observation condition under the control of the control unit 204. That is, the individual sample sections 24 cut out from the analysis target sample are observed by scanning with the electron beam 30L formed under the first acceleration voltage (low voltage observation), and thereby a low voltage image (analysis original image). 38L is generated. A plurality of low voltage images 38L constitute a low voltage image stack 37L. The first acceleration voltage is, for example, 1 keV or 2 keV, and the second acceleration voltage is, for example, 3 keV, 5 keV, or 7 keV. It is desirable to determine the first acceleration voltage and the second acceleration voltage in consideration of the sample, section thickness, target tissue, observation purpose, and other circumstances.
画像スタック35L,35H,37Lは、Z方向における複数の深さに対応する(換言すればデータ記憶空間内でZ方向に並ぶ)複数の画像36L,36H,38Lにより構成される。各画像36L,36H,38Lは、生物組織画像処理装置18側から見て、元画像又は入力画像である。各画像36L,36H,38Lは電子データであり、各画像36L,36H,38LがSEM16から生物組織画像処理装置18へ、ネットワーク又は可搬型記憶媒体を介して、伝送される。
The image stacks 35L, 35H, and 37L include a plurality of images 36L, 36H, and 38L corresponding to a plurality of depths in the Z direction (in other words, arranged in the Z direction in the data storage space). Each of the images 36L, 36H, and 38L is an original image or an input image when viewed from the biological tissue image processing apparatus 18 side. The images 36L, 36H, and 38L are electronic data, and the images 36L, 36H, and 38L are transmitted from the SEM 16 to the biological tissue image processing apparatus 18 via a network or a portable storage medium.
生物組織画像処理装置18は、図示の構成例において、パーソナルコンピュータによって構成されている。生物組織画像処理装置18がSEM16内に組み込まれてもよく、生物組織画像処理装置18がSEM16等を制御するシステムコンピュータ内に組み込まれてもよい。生物組織画像処理装置18によりSEM16が制御されてもよい。
The biological tissue image processing apparatus 18 is configured by a personal computer in the illustrated configuration example. The biological tissue image processing apparatus 18 may be incorporated in the SEM 16, or the biological tissue image processing apparatus 18 may be incorporated in a system computer that controls the SEM 16 or the like. The SEM 16 may be controlled by the biological tissue image processing apparatus 18.
生物組織画像処理装置18は、本体40、表示器46及び入力器48を有している。本体40が有する複数の機能については、後に、図2以降の各図に基づいて詳しく説明する。図1においては、本体40が発揮する代表的な2つの機能(第1画像処理機能及び第2画像処理機能)がそれぞれブロックとして表現されている。具体的には、本体40は、機械学習型の膜推定器を含む第1画像処理部200、及び、正解画像作成部としての第2画像処理部202を有する。表示器46は、LCD、有機EL表示デバイス等によって構成される。入力器48は、ユーザーによって操作されるキーボード、ポインティングデバイス等によって構成される。
The biological tissue image processing apparatus 18 has a main body 40, a display 46 and an input device 48. The plurality of functions of the main body 40 will be described in detail later with reference to FIGS. In FIG. 1, two typical functions (first image processing function and second image processing function) exhibited by the main body 40 are represented as blocks. Specifically, the main body 40 includes a first image processing unit 200 including a machine learning type film estimator, and a second image processing unit 202 as a correct image creation unit. The display 46 is configured by an LCD, an organic EL display device, or the like. The input device 48 includes a keyboard operated by a user, a pointing device, and the like.
図2には、本体40についての第1構成例が示されている。本体40は、上記のように、第1画像処理部200、第2画像処理部202、及び、ボリュームデータ処理部56を有する。図2に示される各構成の実体は、ユーザーの作業又は行為に相当する部分を除いて、CPU、GPU等の汎用プロセッサによって実行されるソフトウエアつまりプログラムである。もっとも、それらの構成の一部又は全部が専用プロセッサ又は他のハードウエアによって構成されてもよい。生物組織画像処理装置が有する機能の全部又は一部がネットワーク上に存在する1又は複数の情報処理デバイスにより実行されてもよい。
FIG. 2 shows a first configuration example of the main body 40. The main body 40 has the first image processing unit 200, the second image processing unit 202, and the volume data processing unit 56 as described above. The entity of each configuration shown in FIG. 2 is software, that is, a program executed by a general-purpose processor such as a CPU or GPU, except for a portion corresponding to a user's work or action. However, part or all of the configuration may be configured by a dedicated processor or other hardware. All or part of the functions of the biological tissue image processing apparatus may be executed by one or a plurality of information processing devices existing on the network.
以下、まず、解析過程で機能する構成を中心に詳述し、その後、学習過程で機能する構成について詳述する。もっとも、第1画像処理部200は、解析過程及び学習過程の両過程において機能する構成を有するので、そのような構成については、解析過程での動作の説明に際して、学習過程での動作についても言及することにする。
Hereinafter, the configuration that functions in the analysis process will be described in detail first, and then the configuration that functions in the learning process will be described in detail. However, since the first image processing unit 200 has a configuration that functions in both the analysis process and the learning process, the operation in the learning process is also referred to in describing the operation in the analysis process. I will do it.
第1画像処理部200は、機械学習型の膜推定器42、二値化器(画像生成器)50、修正部52及びラベリング処理部54、等を有する。
The first image processing unit 200 includes a machine learning type film estimator 42, a binarizer (image generator) 50, a correction unit 52, a labeling processing unit 54, and the like.
膜推定器42は、膜推定手段として機能するものであり、入力画像206に対して膜推定処理を適用し、これにより膜尤度マップ60を出力する。図示の構成例において、膜推定器42は、機械学習型膜推定器であるCNN(Convolutional Neural Network)により構成されている。その具体的な構成例については後に図3を用いて説明する。CNNによって膜が正しく推定されるように、CNNの実際の稼働つまり解析過程に先立って、学習過程が事前に実行される。その学習過程には、一次学習過程(初期学習過程)と、二次学習過程と、が含まれる。
The membrane estimator 42 functions as membrane estimation means, and applies a membrane estimation process to the input image 206, thereby outputting a membrane likelihood map 60. In the illustrated configuration example, the membrane estimator 42 is configured by a CNN (Convolutional Neural Network) which is a machine learning type membrane estimator. A specific configuration example will be described later with reference to FIG. The learning process is executed in advance prior to the actual operation of the CNN, that is, the analysis process so that the film is correctly estimated by the CNN. The learning process includes a primary learning process (initial learning process) and a secondary learning process.
一次学習過程では、教師データを構成する複数の画像ペアが膜推定器42に与えられ、これにより膜推定器42内のCNNパラメータ群が優良化(最適化)される。すなわち、膜推定器42内に機械学習結果が蓄積される。ここで、複数の画像ペアには、複数の第1画像ペアと、複数の第2画像ペアと、が含まれる。個々の第1画像ペアは、第1元画像(低電圧画像)36Lとそれに対応する正解画像223とにより構成される。個々の第2画像ペアは、第2元画像(高電圧画像)36Hとそれに対応する正解画像224とにより構成される。正解画像223は、例えば、第1元画像36Lに対する手作業により作成される。教師なし機械学習器、簡易な識別器(例えばSVM(Support Vector Machine))等によって、第1元画像36Lから正解画像223が作成されてもよい。一方、正解画像224は、後に詳述するように、第2元画像に基づいて、第2画像処理部202により作成される。なお、第2画像処理部202により、第1元画像36Lから正解画像223が作成されてもよい。
In the primary learning process, a plurality of image pairs constituting the teacher data are given to the membrane estimator 42, whereby the CNN parameter group in the membrane estimator 42 is improved (optimized). That is, the machine learning result is accumulated in the film estimator 42. Here, the plurality of image pairs include a plurality of first image pairs and a plurality of second image pairs. Each first image pair is composed of a first original image (low voltage image) 36L and a corresponding correct image 223. Each second image pair includes a second original image (high voltage image) 36H and a correct image 224 corresponding thereto. The correct image 223 is created, for example, by manual work on the first original image 36L. The correct image 223 may be created from the first original image 36L by an unsupervised machine learning device, a simple classifier (for example, SVM (Support Vector Vector)) or the like. On the other hand, the correct image 224 is created by the second image processing unit 202 based on the second original image, as will be described in detail later. The correct image 223 may be created from the first original image 36L by the second image processing unit 202.
続く二次学習過程では、一次学習過程を経て膜推定器42がある程度働くことを前提として、膜推定器42に対して、一次学習過程と同様に、教師データとして複数の画像ペアが与えられる。実施形態において、その教師データは、一次学習過程で利用された複数の画像ペアと二次学習過程で追加される複数の画像ペアとにより構成される。追加される複数の画像ペアには、複数の第1画像ペアと、複数の第2画像ペアと、が含まれる。個々の第1画像ペアは、第1元画像36Lとそれに対応する正解画像64Aとにより構成される。個々の第2画像ペアは、第2元画像36Hとそれに対応する正解画像224とにより構成される。
In the subsequent secondary learning process, on the premise that the membrane estimator 42 works to some extent through the primary learning process, a plurality of image pairs are given as teacher data to the membrane estimator 42 as in the primary learning process. In the embodiment, the teacher data includes a plurality of image pairs used in the primary learning process and a plurality of image pairs added in the secondary learning process. The plurality of added image pairs include a plurality of first image pairs and a plurality of second image pairs. Each first image pair includes a first original image 36L and a correct image 64A corresponding to the first original image 36L. Each second image pair includes a second original image 36H and a correct image 224 corresponding to the second original image 36H.
正解画像64Aは、膜推定器42から加工ツールユニット44までの構成により作成される。具体的には、第1元画像36Lを膜推定器42に入力すると、膜推定器42から推定結果画像として膜尤度マップ60が出力される。膜尤度マップ60に基づく仮膜画像62の生成、及び、加工ツールユニット44を利用した仮膜画像62に対するユーザー(専門家)修正を経て、正解画像64Aが作成される。それらの処理については後に詳述する。仮膜画像62を正解画像62Aとして利用することも考えられる。
The correct image 64A is created by the configuration from the film estimator 42 to the processing tool unit 44. Specifically, when the first original image 36L is input to the membrane estimator 42, the membrane likelihood map 60 is output from the membrane estimator 42 as an estimation result image. A correct image 64 </ b> A is created through generation of a mask image 62 based on the film likelihood map 60 and user (expert) correction of the mask image 62 using the processing tool unit 44. These processes will be described in detail later. It is also conceivable to use the mask image 62 as the correct image 62A.
二次学習過程により、膜推定器42内のCNNパラメータ群が更に優良化される。つまり、膜推定器42内に機械学習結果が更に蓄積される。二次学習過程は、例えば、入力画像206(第1元画像36L,第2元画像36H)に対する推定処理の結果が、それに対応する正解画像223,224,64Aに十分に類似したと判断された場合に終了する。その後、必要に応じて、上記同様の手法により、膜推定器42の再学習が実行される。実施形態では、第2画像処理部202が一次学習過程及び二次学習過程のいずれにおいても正解画像作成手段として機能している。なお、解析過程においては、入力画像206としての元画像(低電圧画像)38Lが膜推定器42に入力される。
The CNN parameter group in the film estimator 42 is further improved by the secondary learning process. That is, the machine learning result is further accumulated in the film estimator 42. In the secondary learning process, for example, it is determined that the result of the estimation process for the input image 206 (the first original image 36L and the second original image 36H) is sufficiently similar to the corresponding correct images 223, 224, and 64A. If finished. Thereafter, re-learning of the film estimator 42 is executed by the same method as described above as necessary. In the embodiment, the second image processing unit 202 functions as correct image creation means in both the primary learning process and the secondary learning process. In the analysis process, an original image (low voltage image) 38L as the input image 206 is input to the membrane estimator 42.
データベース57には、複数の元画像36L,36H,38L、及び、複数の正解画像223,224,64Aが格納される。そこに複数の膜尤度マップ60が格納されてもよい。膜推定器42とデータベース57とが一体化されてもよい。機械学習型の膜推定器として、U-netが利用されてもよく、また、サポートベクターマシーン、ランダムフォレスト等が利用されてもよい。
The database 57 stores a plurality of original images 36L, 36H, and 38L and a plurality of correct images 223, 224, and 64A. A plurality of film likelihood maps 60 may be stored there. The film estimator 42 and the database 57 may be integrated. As a machine learning type film estimator, U-net may be used, or a support vector machine, a random forest, or the like may be used.
二値化器50は、画像生成器として機能するものである。具体的には、二値化器50は、後に図4を用いて例示するように、膜尤度マップ60に対する二値化処理により、仮膜画像62を生成するモジュールである。膜尤度マップ60は、二次元配列された複数の膜尤度からなる。個々の膜尤度は、膜である確からしさ(確率)を示す数値である。膜尤度は例えば0から1の間をとる。膜尤度マップ60を膜尤度画像として捉えることもできる。実施形態において、二値化器50には閾値が設定されており、二値化器50は閾値以上の膜尤度を1に変換し、閾値未満の膜尤度を0に変換する。その結果として生成される画像が仮膜画像62である。実施形態においては、便宜上、修正前の膜画像を修正後の膜画像から区別するために、修正前の膜画像を仮膜画像62と呼んでいる。
The binarizer 50 functions as an image generator. Specifically, the binarizer 50 is a module that generates a temporary membrane image 62 by binarization processing on the membrane likelihood map 60, as will be exemplified later with reference to FIG. The film likelihood map 60 includes a plurality of film likelihoods arranged two-dimensionally. Each film likelihood is a numerical value indicating the probability (probability) of being a film. The film likelihood is, for example, between 0 and 1. The film likelihood map 60 can also be understood as a film likelihood image. In the embodiment, a threshold is set in the binarizer 50, and the binarizer 50 converts a film likelihood that is equal to or greater than the threshold to 1 and converts a film likelihood that is less than the threshold to 0. The image generated as a result is a temporary membrane image 62. In the embodiment, for the sake of convenience, the film image before correction is referred to as a temporary film image 62 in order to distinguish the film image before correction from the film image after correction.
なお、膜推定器42及び二値化器50の両者により画像生成部61が構成される。画像生成部61の全体がCNN等により構成されてもよい。その場合でも、膜尤度マップ及び仮膜画像の段階的な生成を観念できる。仮膜画像62を正解画像62Aとして用いることも可能である。二値化処理前に、膜尤度マップ60に対して、ノイズ除去、エッジ強調等の処理が適用されてもよい。
In addition, the image generation unit 61 is configured by both the film estimator 42 and the binarizer 50. The entire image generation unit 61 may be configured by CNN or the like. Even in such a case, the stepwise generation of the membrane likelihood map and the temporary membrane image can be considered. It is also possible to use the temporary membrane image 62 as the correct image 62A. Prior to binarization processing, processing such as noise removal and edge enhancement may be applied to the film likelihood map 60.
加工ツールユニット44は、作業支援部又は作業支援手段として機能するものである。加工ツールユニット44は、図示の構成例において、第1画像処理部200と第2画像処理部202に跨って設けられている。加工ツールユニット44は、情報処理の観点から見て、表示処理機能及び画像処理機能を有している。作業の観点から見て、修正機能及びラベリング機能を有しており、図2においては、それらの機能が修正部52、ラベリング処理部54及び修正部214として表現されている。
The machining tool unit 44 functions as a work support unit or work support means. The processing tool unit 44 is provided across the first image processing unit 200 and the second image processing unit 202 in the illustrated configuration example. The processing tool unit 44 has a display processing function and an image processing function from the viewpoint of information processing. From the viewpoint of work, it has a correction function and a labeling function. In FIG. 2, these functions are expressed as a correction unit 52, a labeling processing unit 54, and a correction unit 214.
修正部52は、後に図5において例示する作業ウインドウを介して、ユーザーに対して作業対象となった仮膜画像を作業対象画像として表示し、また、作業対象画像上におけるユーザーの修正指示を受け付けるものである。修正内容は作業対象画像となった仮膜画像に反映される。作業対象画像内に例えば膜の途切れ部分が含まれる場合、その途切れ部分に対して膜画素群が追加される。作業対象画像内に例えば膜以外の部分が含まれ、その部分が膜として誤認されている場合、その部分を構成する膜画素群が削除される。そのような追加及び削除に際し、ユーザーの作業又は操作を支援し各仮膜画像を管理するモジュールが修正部52である。
The correction unit 52 displays, as a work target image, a mask image that is a work target for the user via a work window illustrated in FIG. 5 later, and accepts a user's correction instruction on the work target image. Is. The correction contents are reflected in the mask image that is the work target image. For example, when the work target image includes a discontinuous portion of the film, a film pixel group is added to the discontinuous portion. For example, when a part other than the film is included in the work target image and the part is mistaken as a film, the film pixel group constituting the part is deleted. In such addition and deletion, the correction unit 52 is a module that supports the user's work or operation and manages each mask image.
図示の構成例においては、修正部52(又は加工ツールユニット44)に対して、生成された仮膜画像62の他に、入力された元画像(入力画像)206、及び、生成された膜尤度マップ60も順次入力されている。これにより、作業対象画像としての仮膜画像と共に、又はそれに代えて、作業対象画像に対応する元画像206又は膜尤度マップ60を表示することが可能である。
In the illustrated configuration example, in addition to the generated temporary membrane image 62, the input original image (input image) 206 and the generated membrane likelihood are generated with respect to the correction unit 52 (or the processing tool unit 44). The degree map 60 is also sequentially input. Thus, the original image 206 or the membrane likelihood map 60 corresponding to the work target image can be displayed together with or instead of the temporary membrane image as the work target image.
ラベリング処理部54は、修正後の膜画像(又は修正未了の膜画像)に含まれる個々の領域(細胞内腔)に対して、ラベリング(ペイント及びラベル付け)を行うためのモジュールである。ラベリングには、ユーザーによるマニュアルでのラベリング、及び、自動的なラベリングがある。修正作業及びラベリング作業が完了した段階で、細胞内腔とそれ以外とが区別された三次元ラベリングデータ66が構成される。それがボリュームデータ処理部56へ送られる。修正部214については後述する。
The labeling processing unit 54 is a module for performing labeling (painting and labeling) on individual regions (cell lumens) included in the corrected membrane image (or the uncorrected membrane image). Labeling includes manual labeling by the user and automatic labeling. At the stage where the correction work and the labeling work are completed, the three-dimensional labeling data 66 in which the cell lumen is distinguished from the others is configured. This is sent to the volume data processing unit 56. The correction unit 214 will be described later.
ボリュームデータ処理部56は、解析部56A及びレンダリング部56Bを有している。ボリュームデータ処理部56には、図示の構成例において、複数の元画像からなる元画像スタック37Lが入力されている。元画像スタック37Lはボリュームデータを構成するものである。ボリュームデータ処理部56には、上記のように、三次元ラベリングデータ66も入力されている。それもボリュームデータの一種である。
The volume data processing unit 56 includes an analysis unit 56A and a rendering unit 56B. In the illustrated configuration example, the volume data processing unit 56 receives an original image stack 37L composed of a plurality of original images. The original image stack 37L constitutes volume data. As described above, the three-dimensional labeling data 66 is also input to the volume data processing unit 56. It is also a kind of volume data.
解析部56Aは、例えば、三次元ラベリングデータ66に基づいて、対象器官(例えば神経細胞)を解析する。例えば、形態、体積、長さ等が解析されてもよい。その際、三次元ラベリングデータを参照しながら、元画像スタック37Lが解析される。レンダリング部56Bは、三次元ラベリングデータ66に基づいて、三次元画像(立体的表現画像)を形成するものである。例えば、三次元ラベリングデータ66に基づいて、元画像スタック37Lの中から画像化部分が抽出され、それに対してレンダリング処理が適用されてもよい。
The analysis unit 56A analyzes a target organ (for example, a nerve cell) based on the three-dimensional labeling data 66, for example. For example, the form, volume, length, etc. may be analyzed. At that time, the original image stack 37L is analyzed with reference to the three-dimensional labeling data. The rendering unit 56B forms a three-dimensional image (stereoscopic expression image) based on the three-dimensional labeling data 66. For example, an imaging portion may be extracted from the original image stack 37L based on the three-dimensional labeling data 66, and a rendering process may be applied to the extracted image portion.
次に、第2画像処理部202について説明する。第2画像処理部202は、入力画像208としての第2元画像(高電圧画像)36Hに基づいて第2正解画像224を生成するモジュールである。
Next, the second image processing unit 202 will be described. The second image processing unit 202 is a module that generates the second correct image 224 based on the second original image (high voltage image) 36H as the input image 208.
第2元画像36Hとしての高電圧画像には、第1元画像36Lとしての低電圧画像よりも、細胞膜の深部像がより多く含まれる。加速電圧を上げると、試料切片内部におけるより深いところからの情報が得られるからである(これについては後に図8を用いて具体的に説明する)。高電圧画像において、細胞膜の深部像は、典型的には、細胞膜の浅部像(表層像)の近傍にモヤとして(又はモヤのように)現れる。低電圧画像においても、深部像がモヤとして現れるが、その量は比較的に少なく、また、低電圧画像においては、深部像か否かはっきりしないようなぼやけた像が生じ易い。
The high voltage image as the second original image 36H includes more deep images of the cell membrane than the low voltage image as the first original image 36L. This is because when the acceleration voltage is increased, information from a deeper position inside the sample slice can be obtained (this will be described in detail later with reference to FIG. 8). In a high voltage image, a deep image of a cell membrane typically appears as a haze (or like a haze) in the vicinity of a shallow image (surface layer image) of the cell membrane. Even in a low-voltage image, a deep image appears as a haze, but the amount thereof is relatively small, and in a low-voltage image, a blurred image in which it is not clear whether the image is a deep image tends to occur.
細胞内部の三次元ラベリングに際しては、細胞膜の浅部像の他、細胞膜の深部像についても、それが細胞膜の一部として検出されることが望まれる。例えば、細胞膜の浅部像から深部像までを順々に撮影し、各深さに対応する形状を決定すべきである。しかし、試料切片の損傷、撮影時間その他の理由から、膜推定器に対しては低電圧画像の入力が求められることがある。低電圧画像の入力を前提として、細胞膜の深部像が細胞膜の一部として検出され易くなるように、膜推定器の学習段階において、膜推定器に対して、その教師データとして、低電圧画像と共に、高電圧画像が与えられている。
In the three-dimensional labeling inside the cell, it is desired that not only the shallow image of the cell membrane but also the deep image of the cell membrane is detected as a part of the cell membrane. For example, a shallow image and a deep image of the cell membrane should be taken in order, and a shape corresponding to each depth should be determined. However, the membrane estimator may be required to input a low voltage image due to damage to the sample section, imaging time, and other reasons. Assuming that a low-voltage image is input, in order to facilitate detection of a deep image of the cell membrane as a part of the cell membrane, in the learning stage of the membrane estimator, the membrane estimator is used together with the low-voltage image as its teaching data. A high voltage image is given.
第2画像処理部202は、図示の構成例において、モヤ強調部210、セグメンテーション部212、及び、修正部(アノテーション部)214を有する。
The second image processing unit 202 includes a haze enhancement unit 210, a segmentation unit 212, and a correction unit (annotation unit) 214 in the illustrated configuration example.
モヤ強調部210は、入力画像208に対して、それに含まれる細胞膜の深部像を強調する処理を適用するものである。具体的には、モヤ強調としてコントラスト強調が実行されている。
The haze enhancement unit 210 applies processing for enhancing the deep image of the cell membrane included in the input image 208. Specifically, contrast enhancement is performed as moya enhancement.
セグメンテーション部212は、コントラスト強調後の画像218に対してセグメンテーションを適用するものである。セグメンテーションは、画像内容を複数の小領域に区画する処理である。次のアノテーションにおいてその作業を行いやすいように、このセグメンテーションが実行される。
The segmentation unit 212 applies segmentation to the image 218 after contrast enhancement. Segmentation is a process of dividing image content into a plurality of small areas. This segmentation is performed to make it easier to do the work in the next annotation.
修正部214は、修正手段又はアノテーション手段として機能するものである。ユーザー作業のためのツールという意味において、修正部214は上記の修正部52と同じ機能を発揮する。よって、図2においては、修正部214は、加工ツールユニット44の一部として位置付けられている。具体的には、修正部214は、セグメンテーション後の画像220を表示し、画像220に対するユーザーの膜指定及び膜修正を受け付け、その結果として、細胞膜だけが表された膜画像として第2正解画像224を作成するものである。画像220内には複数の小領域が表示されており、細胞膜に相当する小領域に対してユーザーによって膜ラベルが与えられる。その作業はアノテーション及びペイントに相当するものである。その作業性を高めるために、上記同様の作業ウインドウが表示されてもよい。膜推定器42の学習過程(一次学習過程、二次学習過程)において、上記のようにして作成された第2正解画像224が膜推定器42に与えられる。
The correction unit 214 functions as correction means or annotation means. In the sense of a tool for user work, the correction unit 214 exhibits the same function as the correction unit 52 described above. Therefore, in FIG. 2, the correction unit 214 is positioned as a part of the processing tool unit 44. Specifically, the correction unit 214 displays the segmented image 220, accepts the user's membrane designation and membrane correction for the image 220, and as a result, the second correct image 224 as a membrane image showing only the cell membrane. Is to create. A plurality of small regions are displayed in the image 220, and a membrane label is given by the user to the small region corresponding to the cell membrane. The work corresponds to annotation and painting. In order to improve the workability, a work window similar to the above may be displayed. In the learning process (primary learning process and secondary learning process) of the film estimator 42, the second correct image 224 created as described above is given to the film estimator 42.
上記構成によれば、正解画像64Aの作製に際して、膜推定器42による推定結果を利用することができるので、且つ、修正部52を利用することができるので、それらを利用できない場合に比べて、正解画像64Aの作製負担が大幅に削減される。二次学習過程後の解析過程においては、膜推定器42及び加工ツールユニット44の併用により、解析対象又はレンダリング対象となる画像群の品質を高められ、また、その画像群を簡便かつ迅速に生成することが可能である。
According to the above configuration, since the estimation result by the membrane estimator 42 can be used in the production of the correct image 64A, and the correction unit 52 can be used, compared to the case where they cannot be used, The burden of preparing the correct image 64A is greatly reduced. In the analysis process after the secondary learning process, the combined use of the film estimator 42 and the processing tool unit 44 can improve the quality of the image group to be analyzed or rendered, and easily and quickly generate the image group. Is possible.
また、上記構成によれば、解析過程において膜推定器42に対して低電圧画像を入力することを前提として、低電圧画像の他、高電圧画像をも利用して膜推定器42の学習が行われているので、膜推定器42の膜推定精度を高めることが可能である。実際には、学習用の同じ試料切片の同一部位を表す低電圧画像及び高電圧画像が生成され、それらの両方が教師データの一部として膜推定器42へ与えられている。
Further, according to the above configuration, on the premise that a low voltage image is input to the membrane estimator 42 in the analysis process, the membrane estimator 42 can learn using a high voltage image in addition to a low voltage image. Since this is done, the film estimation accuracy of the film estimator 42 can be increased. Actually, a low voltage image and a high voltage image representing the same part of the same sample section for learning are generated, and both of them are given to the film estimator 42 as a part of the teacher data.
更に、上記構成によれば、第2正解画像の作成に際して、細胞膜の深部像が細胞膜の一部とみなされる可能性を高められるので、第2正解画像の品質を向上できる。
Furthermore, according to the above configuration, when creating the second correct image, the possibility that the deep image of the cell membrane is regarded as a part of the cell membrane can be increased, so that the quality of the second correct image can be improved.
なお、一次学習過程及び二次学習過程では、複数の第1元画像及び複数の第2元画像が所定順序で膜推定器42に順次入力される。それらの画像がバッチ処理されてもよい。
In the primary learning process and the secondary learning process, a plurality of first original images and a plurality of second original images are sequentially input to the membrane estimator 42 in a predetermined order. Those images may be batch processed.
図3には、膜推定器42の構成例が模式的に示されている。膜推定器42は、多数の層を有しており、それには、入力層80、畳み込み層82、プーリング層84、出力層86等が含まれる。それらはCNNパラメータ群88に従って作用する。CNNパラメータ群88には、多数の重み係数、多数のバイアス値、その他が含まれる。CNNパラメータ群88は最初に初期値群94によって構成される。例えば、乱数等を利用して初期値群94が生成される。
FIG. 3 schematically shows a configuration example of the membrane estimator 42. The membrane estimator 42 has a number of layers, including an input layer 80, a convolution layer 82, a pooling layer 84, an output layer 86, and the like. They act according to the CNN parameter group 88. The CNN parameter group 88 includes a number of weighting factors, a number of bias values, and the like. The CNN parameter group 88 is initially configured by an initial value group 94. For example, the initial value group 94 is generated using random numbers or the like.
学習過程では、評価部90及び更新部92が機能する。例えば、評価部90は、教師データを構成する複数の画像ペア(元画像36L,36H(図2を参照)とそれに対応する正解画像223,224,64A)に基づいて評価値を計算するものである。具体的には、元画像36L,36Hに対する推定処理の結果60Aと元画像36L,36Hに対応する正解画像223,224,64Aとを誤差関数に順次与えることにより評価値が演算される。更新部92は、その評価値が良い方向に変化するように、CNNパラメータ群88を更新する。それを繰り返すことによって、CNNパラメータ群88が全体的に最適化される。実際には、評価値が一定値に到達した時点で、学習過程の終了が判断される。図3に示した構成は単なる例示に過ぎず、膜推定器42として、多様な構造を有する推定器を利用することが可能である。
In the learning process, the evaluation unit 90 and the update unit 92 function. For example, the evaluation unit 90 calculates an evaluation value based on a plurality of image pairs ( original images 36L and 36H (see FIG. 2) and corresponding correct images 223, 224 and 64A) constituting the teacher data. is there. Specifically, the evaluation value is calculated by sequentially giving the result 60A of the estimation processing for the original images 36L and 36H and the correct images 223, 224 and 64A corresponding to the original images 36L and 36H to the error function. The updating unit 92 updates the CNN parameter group 88 so that the evaluation value changes in a favorable direction. By repeating this, the CNN parameter group 88 is optimized as a whole. Actually, the end of the learning process is determined when the evaluation value reaches a certain value. The configuration shown in FIG. 3 is merely an example, and it is possible to use estimators having various structures as the film estimator 42.
図4には、二値化器の作用が示されている。膜推定器から膜尤度マップ60が出力される。膜尤度マップ60は、複数の画素に相当する複数の膜尤度60aからなるものであり、個々の膜尤度60aは膜であることの確からしさを示す数値である。符号50Aで示すように、二値化器は膜尤度マップ60を二値化し、これにより二値化画像としての仮膜画像62を生成する。二値化に際しては、個々の膜尤度60aが閾値と比較される。例えば、閾値以上の膜尤度60aが値1を有する画素62aに変換され、閾値未満の膜尤度60aが値0を有する画素62bに変換される。画素62aが膜を構成する画素(膜画素)として取り扱われる。閾値はユーザーにより可変設定し得る。その設定が自動化されてもよい。例えば、仮膜画像62を観察しながら閾値が可変されてもよい。
FIG. 4 shows the operation of the binarizer. A film likelihood map 60 is output from the film estimator. The film likelihood map 60 is composed of a plurality of film likelihoods 60a corresponding to a plurality of pixels, and each film likelihood 60a is a numerical value indicating the probability of being a film. As indicated by reference numeral 50A, the binarizer binarizes the membrane likelihood map 60, thereby generating a temporary membrane image 62 as a binarized image. In binarization, each film likelihood 60a is compared with a threshold value. For example, a membrane likelihood 60a that is equal to or greater than the threshold is converted into a pixel 62a having a value of 1, and a membrane likelihood 60a that is less than the threshold is converted into a pixel 62b having a value of 0. The pixel 62a is handled as a pixel (film pixel) constituting the film. The threshold value can be variably set by the user. The setting may be automated. For example, the threshold value may be changed while observing the temporary membrane image 62.
図5には、加工ツールユニットにより表示される作業ウインドウが例示されている。図示の例において、作業ウインドウ100は、表示画像102を含んでいる。表示画像102は、ユーザーによって選択された深さに対応する仮膜画像(作業対象画像)及び元画像からなる合成画像(複合画像)である。グレイスケール画像としての元画像が背景画像を構成しており、それに対して、カラー画像(例えば青色画像)としての仮膜画像(作業対象画像)が重畳表示されている。なお、図示された表示画像102は脳組織を示しており、そこには、複数の細胞の断面が現れている。同時に、細胞内の小器官(ミトコンドリア等)の断面も現れている。
FIG. 5 illustrates a work window displayed by the machining tool unit. In the illustrated example, the work window 100 includes a display image 102. The display image 102 is a composite image (composite image) composed of a mask image (work target image) corresponding to the depth selected by the user and the original image. An original image as a gray scale image constitutes a background image, and a temporary membrane image (work target image) as a color image (for example, a blue image) is superimposed and displayed on the background image. Note that the illustrated display image 102 shows brain tissue, in which cross sections of a plurality of cells appear. At the same time, a cross section of an organelle (such as mitochondria) in the cell also appears.
タブ104が選択された場合、上記で説明した表示画像102が表示される。タブ105が選択された場合、グレイスケール画像としての元画像だけが表示される。タブ106が選択された場合、グレイスケール画像としての膜尤度マップ(膜尤度画像)だけが表示される。膜尤度マップの観察により、閾値が適切に設定されていること等を確認できる。また、元画像の単独表示によれば、膜の細部を詳細観察することが容易となる。作業対象画像としての仮膜画像だけを表示するタブを追加してもよい。膜尤度マップを背景画像とし、それに対して仮膜画像が重畳表示されてもよい。
When the tab 104 is selected, the display image 102 described above is displayed. When the tab 105 is selected, only the original image as a gray scale image is displayed. When the tab 106 is selected, only the film likelihood map (film likelihood image) as a gray scale image is displayed. By observing the film likelihood map, it can be confirmed that the threshold is appropriately set. Moreover, according to the original display of the original image, it becomes easy to observe the details of the film in detail. A tab for displaying only a mask image as a work target image may be added. A membrane likelihood map may be used as a background image, and a temporary membrane image may be superimposed on the background image.
深さ選択ツール108は、Z方向において、特定の深さ(表示深さ)を選択するための表示要素(操作要素)である。それは、Z軸を表すZ軸シンボル108bと、Z軸シンボル108bに沿ってスライド運動するスライダとしてのマーカー108aと、からなる。マーカー108aを移動させて、所望の深さを選択することが可能である。このような深さ選択ツール108によれば、選択する深さや深さ変化量を直感的に認識し易いという利点を得られる。なお、Z軸シンボル108bの左端点が深さゼロに相当し、Z軸シンボル108bの右端点が最大深さに相当している。他の形態を有する深さ選択ツールを採用してもよい。深さ入力欄114は、深さを数値として直接的に指定するための欄である。深さ入力欄114に、マーカー108aの位置によって現在選択されている深さが数値として表示されてもよい。
The depth selection tool 108 is a display element (operation element) for selecting a specific depth (display depth) in the Z direction. It consists of a Z-axis symbol 108b representing the Z-axis and a marker 108a as a slider that slides along the Z-axis symbol 108b. It is possible to select the desired depth by moving the marker 108a. According to such a depth selection tool 108, it is possible to obtain an advantage that it is easy to intuitively recognize the depth to be selected and the depth change amount. The left end point of the Z-axis symbol 108b corresponds to zero depth, and the right end point of the Z-axis symbol 108b corresponds to the maximum depth. Depth selection tools having other forms may be employed. The depth input field 114 is a field for directly specifying the depth as a numerical value. The depth currently selected according to the position of the marker 108a may be displayed as a numerical value in the depth input field 114.
透過度調整ツール110は、表示画像102が表示されている状態において、合成表示されているカラーの仮膜画像(作業対象画像)の透過度(表示重み)を調整するためのツールである。例えば、マーカー110aを左側へ移動させると、カラーの仮膜画像の表示重みが小さくなり、その透明度が上がって、元画像が支配的に表示されるようになる。逆に、マーカー110aを右側へ移動させると、カラーの仮膜画像の表示重みが大きくなり、その透明度が下がって、カラーの仮膜画像がよりはっきりと表示されるようになる。
The transparency adjustment tool 110 is a tool for adjusting the transparency (display weight) of the color temporary membrane image (work target image) displayed in a synthesized state in a state where the display image 102 is displayed. For example, when the marker 110a is moved to the left side, the display weight of the color mask image is reduced, the transparency is increased, and the original image is dominantly displayed. On the contrary, when the marker 110a is moved to the right side, the display weight of the color mask image increases, the transparency decreases, and the color mask image is displayed more clearly.
重合表示ツール112は、現在表示されている画像(合成画像、元画像又は膜尤度マップ)に対して、深さ方向に浅い側に隣接する画像(合成画像、元画像又は膜尤度マップ)又は深さ方向に深い側に隣接する画像(合成画像、元画像又は膜尤度マップ)を重ねて表示する場合において操作されるものである。マーカー112aを左側へ移動させると、浅い側に隣接する画像に対する表示重みが大きくなり、逆に、マーカー112aを右側へ移動させると、深い側に隣接する画像に対する表示重みが大きくなる。3つ以上の画像を重ね合わせて表示してもよい。もっとも、あまり多くの画像を重ね合わせると、画像内容が複雑になり過ぎてしまうので、少数の画像を重ね合わせるのが望ましい。このような重合表示によれば、奥行き情報を得られ易くなる。
The superimposition display tool 112 is an image (synthesized image, original image or film likelihood map) adjacent to the currently displayed image (synthesized image, original image or film likelihood map) on the shallow side in the depth direction. Alternatively, the operation is performed when an image (a composite image, an original image, or a film likelihood map) adjacent to the deep side in the depth direction is displayed in an overlapping manner. When the marker 112a is moved to the left side, the display weight for the image adjacent to the shallow side increases. Conversely, when the marker 112a is moved to the right side, the display weight for the image adjacent to the deep side increases. Three or more images may be superimposed and displayed. Of course, if too many images are superimposed, the image content becomes too complex, so it is desirable to overlay a small number of images. Such superposition display makes it easy to obtain depth information.
ボタン列115は、仮想的な複数のボタン116,118,120,121,122,126によって構成される。ボタン116は、画像ズーム(拡大又は縮小)を行う場合に操作される表示要素である。ボタン118は、ペンツールを利用する場合に操作される表示要素である。ボタン118をオンにすると、カーソルの形態がペン形に代わり、それを用いて膜画素の追加を行える。ペンのサイズを変更することも可能である。ボタン120は、消しゴムを利用する場合に操作される表示要素である。ボタン120をオンにすると、カーソルの形態が消しゴム形に代わり、それを用いて膜画素の削除を行える。消しゴムのサイズを変更することも可能である。
The button row 115 includes a plurality of virtual buttons 116, 118, 120, 121, 122, 126. The button 116 is a display element operated when performing image zoom (enlargement or reduction). The button 118 is a display element that is operated when the pen tool is used. When the button 118 is turned on, the shape of the cursor is changed to a pen shape, and a film pixel can be added using the cursor shape. It is also possible to change the pen size. The button 120 is a display element that is operated when using an eraser. When the button 120 is turned on, the shape of the cursor is changed to an eraser shape, and the film pixel can be deleted using the cursor shape. It is also possible to change the size of the eraser.
ボタン121は、ペイントを行う場合に操作される表示要素である。ボタン121をオンした上で、いずれかの領域を指定すれば、その領域が塗り潰される。また、ボタン121の操作によって、ペイント(又はラベリング)のために用意されている複数の機能の中から任意の機能を選択することも可能である。オブジェクト番号(ラベル)付けボタン122の操作により、カラーパレット124が表示される。例えば、ペイント処理済み領域に対して、カラーパレットの中から選択されたカラーが与えられる。これにより、その領域が、選択されたカラーによって着色される。個々の色がそれぞれオブジェクト番号に対応付けられている。レイヤ間にわたって複数の領域に対して同じカラーつまり同じオブジェクト番号を付与すれば、それらの領域によって特定の細胞内の三次元内腔領域が画定される。
The button 121 is a display element that is operated when painting. If any area is designated after the button 121 is turned on, the area is filled. It is also possible to select an arbitrary function from a plurality of functions prepared for painting (or labeling) by operating the button 121. By operating the object number (label) button 122, the color palette 124 is displayed. For example, a color selected from the color palette is given to the painted area. Thereby, the area is colored by the selected color. Each color is associated with an object number. If the same color, that is, the same object number is given to a plurality of regions across the layers, a three-dimensional lumen region in a specific cell is defined by these regions.
ボタン126は、白黒反転用のボタンである。それを操作すると、表示された画像において、黒く表示されていた部分が白く表示され、逆に白く表示されていた部分が黒く表示される。
The button 126 is a button for black and white reversal. When it is operated, a portion displayed in black in the displayed image is displayed in white, and conversely, a portion displayed in white is displayed in black.
上記の他、ラベリング後の複数の仮膜画像に基づいて三次元像を生成及び表示するためのボタンを設けるのが望ましい。図5に示した作業ウインドウ100の内容は例示に過ぎず、ユーザー作業においてその作業性が良好になるように、その内容が適宜定められるのが望ましい。
In addition to the above, it is desirable to provide a button for generating and displaying a three-dimensional image based on a plurality of temporary membrane images after labeling. The contents of the work window 100 shown in FIG. 5 are merely examples, and it is desirable that the contents be appropriately determined so that the workability is improved in the user work.
図6には、修正前の仮膜画像180と修正後の仮膜画像182とが示されている。修正前の仮膜画像180には、膜途切れ部分184,188が含まれる。修正後の仮膜画像182においては、符号186及び符号190で示されているように、膜途切れ部分が補修されている。また、修正前の仮膜画像180には、非膜部分192,196が含まれている。修正後の仮膜画像182においては、符号194及び符号198で示されているように、それらの非膜部分が消失している。修正後の仮膜画像182によれば、ラベリングを的確に行うことが可能となり、ひいては、三次元ラベリングデータの品質を高めることが可能となる。
FIG. 6 shows a temporary membrane image 180 before correction and a temporary membrane image 182 after correction. The temporary film image 180 before correction includes film break portions 184 and 188. In the corrected temporary membrane image 182, as shown by reference numerals 186 and 190, the film interruption portion is repaired. In addition, the temporary membrane image 180 before correction includes non-membrane portions 192 and 196. In the corrected temporary membrane image 182, as indicated by reference numerals 194 and 198, those non-membrane portions have disappeared. According to the corrected mask image 182, it is possible to accurately perform labeling, and consequently, it is possible to improve the quality of the three-dimensional labeling data.
図7には、ラベリング処理に含まれる三次元繋ぎ処理が例示されている。図7においては、Z方向に並ぶ複数の仮膜画像D1~D4が示されている。深さZiに対応する仮膜画像D1が作業対象画像である。それが以下に説明する処理での基準画像とされる。
FIG. 7 illustrates a three-dimensional splicing process included in the labeling process. In FIG. 7, a plurality of temporary membrane images D1 to D4 arranged in the Z direction are shown. A temporary membrane image D1 corresponding to the depth Zi is a work target image. This is a reference image in the processing described below.
三次元繋ぎ処理の第1例では、基準画像である仮膜画像D1に含まれる領域R1について代表点が特定される。例えば、代表点として中心点、重心点等が特定される。次いで、その代表点を通過する垂線Cが定義される。基準画像から、例えば、深い側にN枚の仮膜画像が参照され、それらの画像において垂線Cが通過している各領域が特定される。そして、垂線Cが通過している領域R1,R2,R3,R4,・・・に対して、同じラベルが付与される。以上の処理が基準画像から浅い側のN枚の仮膜画像に対して適用されてもよい。自動的なラベリングの結果は、通常、ユーザーによって目視確認される。
In the first example of the three-dimensional joining process, the representative point is specified for the region R1 included in the mask image D1 that is the reference image. For example, a center point, a center of gravity point, etc. are specified as representative points. Next, a perpendicular C passing through the representative point is defined. From the reference image, for example, N masked membrane images are referred to on the deep side, and each region through which the perpendicular C passes is specified in these images. And the same label is provided with respect to area | region R1, R2, R3, R4, ... through which the perpendicular C passes. The above processing may be applied to N temporary film images on the shallow side from the reference image. The result of automatic labeling is usually visually confirmed by the user.
三次元繋ぎ処理の第2例では、基準画像である仮膜画像D1に含まれる領域R1(の外縁)が仮膜画像D2上に投影され、投影領域R1aが定義される。仮膜画像D2上において、投影領域R1aに対して最も重なり合う領域R2が特定される。続いて、領域R2が仮膜画像D3上に投影され、投影領域R2aが定義される。仮膜画像D3上において、投影領域R2aに対して最も重なり合う領域R3が特定される。同様に、仮膜画像D4上において、投影領域R3aが定義され、それに基づいて領域R4が特定される。領域R1、及び、それを出発点として特定された領域R2,R3,R4の全部に対して、同じラベルが付与される。
In the second example of the three-dimensional splicing process, the region R1 (outer edge) included in the mask image D1 which is the reference image is projected onto the mask image D2, and the projection region R1a is defined. A region R2 that most overlaps the projection region R1a is specified on the mask image D2. Subsequently, the region R2 is projected onto the mask image D3, and the projection region R2a is defined. A region R3 that most overlaps the projection region R2a is specified on the mask image D3. Similarly, a projection region R3a is defined on the mask image D4, and the region R4 is specified based on the projection region R3a. The same label is assigned to the region R1 and all of the regions R2, R3, and R4 identified from the region R1.
上記の処理では、投影元の領域がレイヤ変更時に逐次更新されていたが、それを固定するようにしてもよい。例えば、領域R1を仮膜画像D2,D3,D4へ投影するようにしてもよい。また、上記の処理では、基準画像からZ方向の一方側に繋がり先が探索されていたが、基準画像からZ方向の両側に繋がり先が探索されてもよい。その探索範囲がユーザー選択されてもよい。いずれにしても、自動的なラベリングの結果が、通常、ユーザーによって目視確認される。その場合には、図5に示した作業ウインドウが利用される。
In the above processing, the projection source area is sequentially updated when the layer is changed, but it may be fixed. For example, the region R1 may be projected onto the mask images D2, D3, D4. In the above processing, the connection destination is searched from the reference image to one side in the Z direction, but the connection destination may be searched from the reference image to both sides in the Z direction. The search range may be selected by the user. In any case, the result of automatic labeling is usually visually confirmed by the user. In that case, the work window shown in FIG. 5 is used.
以上説明した三次元繋ぎ処理は例示であり、上記以外の三次元繋ぎ処理が採用されてもよい。領域間における繋がり関係の特定に際しては、領域ごとの1又は複数の特徴量が利用され得る。領域の特徴量として、面積、形態、周囲長、重心点、輝度ヒストグラム、テクスチャ、等が利用されてもよい。あるいは、領域間の特徴量として、重合面積、代表点間距離、等が利用されてもよい。
The three-dimensional connection process described above is an example, and a three-dimensional connection process other than the above may be employed. In specifying the connection relationship between regions, one or a plurality of feature amounts for each region can be used. As the feature amount of the area, an area, a form, a perimeter, a barycentric point, a luminance histogram, a texture, or the like may be used. Alternatively, a superposition area, a distance between representative points, or the like may be used as a feature amount between regions.
図8には、印加電圧の変化に伴って生じる観察深さ範囲の変化及びSEM画像の変化が誇張して模式的に示されている。符号230,232,234は、互いに異なる3つの観察条件を示している。観察条件230は加速電圧を低い電圧とする条件である。観察条件232は加速電圧を中程度の電圧とする条件である。観察条件234は加速電圧を高い電圧とする条件である。図8の上段(A)には、試料切片236の断面と電子ビーム244,246,248との関係が示されている。各断面には、Z方向に平行に伸長した膜部分238と、Z方向に対して斜めの方向に伸長した膜部分240と、が含まれる。図8の下段(B)にはSEM画像が示されている。
FIG. 8 schematically shows exaggeratedly the change in the observation depth range and the change in the SEM image caused by the change in the applied voltage. Reference numerals 230, 232, and 234 indicate three different observation conditions. The observation condition 230 is a condition for setting the acceleration voltage to a low voltage. The observation condition 232 is a condition in which the acceleration voltage is a medium voltage. The observation condition 234 is a condition for increasing the acceleration voltage. The upper part (A) of FIG. 8 shows the relationship between the cross section of the sample piece 236 and the electron beams 244, 246, 248. Each cross section includes a film portion 238 extending parallel to the Z direction and a film portion 240 extending in a direction oblique to the Z direction. A SEM image is shown in the lower part (B) of FIG.
図示の例において、観察条件230が選択された場合、電子ビーム244による観察の対象は、もっぱら試料切片236の表層となる。このため、SEM画像上においては、2つの膜部分238,240の表層を表す像238a,240aが現れる。観察条件232が選択された場合、電子ビーム246による観察の対象は、試料切片236の中間層にまで及ぶ。このため、SEM画像上においては、膜部分238の中間層までを表す像(中間積算像)238bが現れ、膜部分240については像240bの近傍に深部像240cが現れる。観察条件234が選択された場合、電子ビーム248による観察の対象は、図示の例では、試料切片236の全体に及んでいる。このため、SEM画像上においては、膜部分238の全体を表す像(全体積算像)238cが現れ、膜部分240については像240dの近傍に幅の広い深部像240eが現れる。
In the illustrated example, when the observation condition 230 is selected, the object to be observed by the electron beam 244 is exclusively the surface layer of the sample section 236. Therefore, on the SEM image, images 238a and 240a representing the surface layers of the two film portions 238 and 240 appear. When the observation condition 232 is selected, the object to be observed by the electron beam 246 extends to the intermediate layer of the sample slice 236. Therefore, on the SEM image, an image (intermediate integrated image) 238b representing the intermediate layer of the film portion 238 appears, and a deep image 240c appears in the vicinity of the image 240b for the film portion 240. When the observation condition 234 is selected, the object to be observed by the electron beam 248 extends to the entire sample section 236 in the illustrated example. Therefore, on the SEM image, an image (entire integrated image) 238c representing the entire film portion 238 appears, and a wide depth image 240e appears in the vicinity of the image 240d for the film portion 240.
試料の損傷の回避又は軽減のためには低い加速電圧で試料を観察することが求められるが、その一方、膜推定器による膜推定に際しては、細胞膜の深部像も細胞膜の一部として正しく推定されることが求められる。膜推定器に対して細胞膜の深部像が細胞膜の一部であることをしっかり教え込むことが必要となる。すなわち、膜推定器の学習段階において、深部像を多く含む画像を膜推定器に与える必要がある。そのような観点から、実施形態では、膜推定器の学習過程において、複数の低電圧画像と共に複数の高電圧画像を膜推定器に与えるようにしている。それらの比率は1対1であってもよいが、その比率は状況に応じて変更し得る。
In order to avoid or reduce damage to the sample, it is required to observe the sample at a low acceleration voltage.On the other hand, when the membrane is estimated by the membrane estimator, the depth image of the cell membrane is also correctly estimated as part of the cell membrane. Is required. It is necessary to firmly instruct the membrane estimator that the deep image of the cell membrane is a part of the cell membrane. That is, in the learning stage of the film estimator, it is necessary to give the film estimator an image including many deep images. From such a viewpoint, in the embodiment, in the learning process of the film estimator, a plurality of high voltage images are provided to the film estimator together with the plurality of low voltage images. Their ratio may be 1: 1, but the ratio can vary depending on the situation.
図9及び図10を用いて、第2画像処理部の作用について具体例を用いて説明する。第2画像処理部においては、最初に、図9に示されているように、第2元画像(高電圧画像)250に対して、モヤ強調としてのコントラスト強調252が適用される。続いて、コントラスト強調後の画像254に対してセグメンテーション256が実行される。図10に示されているように、セグメンテーション後の画像258には多数の小領域が設定されている。その画像258上において、細胞膜に相当する複数の小領域に対して膜ラベルがユーザーによって付与される。膜ラベルが付与された複数の小領域を塗り潰すことにより、膜画像262が作成される。塗り潰し後において膜画像262が修正されてもよい。それ以前の段階において画像が修正されてもよい。このようにして作成された膜画像262が第2正解画像として利用される。なお、図10に示されている膜画像262においては白い部分が膜部分である。
The operation of the second image processing unit will be described with reference to FIGS. 9 and 10 using a specific example. In the second image processing unit, first, as shown in FIG. 9, contrast enhancement 252 as moya enhancement is applied to the second original image (high voltage image) 250. Subsequently, segmentation 256 is performed on the image 254 after contrast enhancement. As shown in FIG. 10, many small regions are set in the image 258 after segmentation. On the image 258, a membrane label is given by the user to a plurality of small regions corresponding to the cell membrane. A membrane image 262 is created by painting a plurality of small regions to which a membrane label is attached. The film image 262 may be corrected after painting. The image may be corrected in an earlier stage. The film image 262 created in this way is used as the second correct image. In addition, in the film | membrane image 262 shown by FIG. 10, a white part is a film | membrane part.
図11及び図12を用いて、実施形態の効果について説明する。既に説明したように、実施形態では、膜推定器の学習段階において、膜推定器に対して低電圧画像及び高電圧画像の両方が与えられる。これに対し、以下の比較例は、膜推定器の学習段階において、膜推定器に対して低電圧画像だけを与えたものである。
The effect of the embodiment will be described with reference to FIGS. 11 and 12. As already explained, in the embodiment, both the low voltage image and the high voltage image are given to the film estimator in the learning stage of the film estimator. On the other hand, the following comparative example gives only a low voltage image to the film estimator in the learning stage of the film estimator.
図11の左側には、入力画像270が示されており、図11の右側には、入力画像270に対応する正解画像272が示されている。なお、正解画像272では、黒い部分が膜部分である。
The input image 270 is shown on the left side of FIG. 11, and the correct image 272 corresponding to the input image 270 is shown on the right side of FIG. In the correct image 272, the black part is the film part.
図12の左側には、比較例に係る膜尤度マップ274が示されている。図11の右側には、実施形態に係る膜尤度マップ276が示されている。それらの膜尤度マップ274,276においては、膜尤度の大きさが黒色の濃さで表現されている。2つの膜尤度マップ274,276を対比すると、膜尤度マップ274上で薄く表現された膜部分A1,B1,C1,D1が、膜尤度マップ276上で濃く表現された膜部分A2,B2,C2,D2に変化している。ここで、膜部分A1,B1(及び膜部分A2,B2)は、深部像を含む部分であると思われる。以上のように、高電圧画像を用いて膜推定器に学習させることにより、様々な向きを有している細胞膜の推定精度を高められる。
On the left side of FIG. 12, a film likelihood map 274 according to the comparative example is shown. A membrane likelihood map 276 according to the embodiment is shown on the right side of FIG. In these film likelihood maps 274 and 276, the magnitude of the film likelihood is expressed in black. When the two film likelihood maps 274 and 276 are compared, the film portions A1, B1, C1, and D1 that are thinly expressed on the film likelihood map 274 are the film portions A2 that are expressed deeply on the film likelihood map 276. It has changed to B2, C2, and D2. Here, the film portions A1 and B1 (and the film portions A2 and B2) are considered to be portions including a deep image. As described above, by causing the membrane estimator to learn using a high-voltage image, it is possible to improve the estimation accuracy of cell membranes having various orientations.
比較例についてのランド(Rand)誤差は0.012であったが、実施形態についてのランド誤差は0.009であった。ランド誤差は、1-(Rand Index)で定義されるものであり、ここでRand Indexは評価対象画像と正解画像との間の一致度又は類似度を示す公知の指標である。
The land error for the comparative example was 0.012, while the land error for the embodiment was 0.009. The land error is defined by 1- (Rand Index), where Rand Index is a known index indicating the degree of coincidence or similarity between the evaluation target image and the correct image.
図13には、本体の第2構成例が示されている。なお、図2に示した構成と同一の構成には同一符号を付し、その説明を省略する。
FIG. 13 shows a second configuration example of the main body. In addition, the same code | symbol is attached | subjected to the structure same as the structure shown in FIG. 2, and the description is abbreviate | omitted.
本体280は第1画像処理部200及び第2画像処理部281を含む。第2画像処理部281は、機械学習型の膜推定器282、二値化器284及び修正部286を含む。膜推定器282は、図示の例において、CNNで構成されている。膜推定器282は、入力画像208として第2元画像(高電圧画像)36Hに対して膜推定処理を適用し、膜尤度マップを出力するものである。膜推定器282は、高電圧画像を利用した学習を経た学習済み推定器である。二値化器284は二値化器50と同様に膜尤度マップから仮膜画像を生成するものである。修正部286は、修正部52と同様の機能を有し、仮膜画像に対するユーザー修正を受け付け、これにより第2正解画像224を生成するものである。第2構成例は、第2正解画像の生成において膜推定器42とは別の膜推定器282を利用するものである。
The main body 280 includes a first image processing unit 200 and a second image processing unit 281. The second image processing unit 281 includes a machine learning type film estimator 282, a binarizer 284, and a correction unit 286. The film estimator 282 is composed of CNN in the illustrated example. The film estimator 282 applies a film estimation process to the second original image (high voltage image) 36H as the input image 208, and outputs a film likelihood map. The film estimator 282 is a learned estimator that has undergone learning using a high-voltage image. Similarly to the binarizer 50, the binarizer 284 generates a temporary membrane image from the membrane likelihood map. The correction unit 286 has the same function as that of the correction unit 52, accepts user correction on the mask image, and generates the second correct image 224 thereby. The second configuration example uses a membrane estimator 282 that is different from the membrane estimator 42 in generating the second correct image.
上記実施形態では、2つの観察条件が利用されていたが、3つ以上の観察条件が利用されてもよい。上記膜推定器としてCNNが利用されていたが、他の機械学習型膜推定器が利用されてもよい。
In the above embodiment, two observation conditions are used, but three or more observation conditions may be used. Although CNN is used as the film estimator, other machine learning type film estimators may be used.
Claims (10)
- 生物組織である解析対象試料の観察により画像を生成する電子顕微鏡と、
前記画像に対して前記解析対象試料に含まれる注目要素を推定する処理を適用する機械学習型の推定器と、
を含み、
前記推定器は学習用元画像セットによる学習を経た学習済み推定器であり、
前記学習用元画像セットには、第1観察条件の下で電子顕微鏡により生物組織である学習対象試料を観察することにより生成された第1元画像と、第2観察条件の下で電子顕微鏡により前記学習対象試料を観察することにより生成された第2元画像と、が含まれ、
前記第1観察条件は、前記解析対象試料を観察する際に電子顕微鏡に対して設定される条件であり、前記第2観察条件は、前記第1観察条件とは異なる深さ範囲にわたって観察を行える条件である、
ことを特徴とする生物組織画像処理システム。 An electron microscope that generates an image by observing a sample to be analyzed that is a biological tissue;
A machine learning type estimator for applying a process of estimating an element of interest included in the analysis target sample to the image;
Including
The estimator is a learned estimator that has undergone learning using a learning original image set,
The learning original image set includes a first original image generated by observing a learning target sample that is a biological tissue with an electron microscope under a first observation condition, and an electron microscope under a second observation condition. A second original image generated by observing the learning target sample, and
The first observation condition is a condition set for an electron microscope when observing the analysis target sample, and the second observation condition can be observed over a depth range different from the first observation condition. Condition,
A biological tissue image processing system characterized by that. - 請求項1記載の生物組織画像処理システムにおいて、
前記第2観察条件は、前記第1観察条件よりも大きな深さ範囲にわたって観察を行える条件である、
ことを特徴とする生物組織画像処理システム。 The biological tissue image processing system according to claim 1,
The second observation condition is a condition that enables observation over a depth range larger than the first observation condition.
A biological tissue image processing system characterized by that. - 請求項2記載の生物組織画像処理システムにおいて、
前記第1観察条件は、電子顕微鏡の加速電圧として第1加速電圧を設定する条件であり、
前記第2観察条件は、電子顕微鏡の加速電圧として前記第1加速電圧よりも高い第2加速電圧を設定する条件である、
ことを特徴とする生物組織画像処理システム。 The biological tissue image processing system according to claim 2,
The first observation condition is a condition for setting the first acceleration voltage as the acceleration voltage of the electron microscope,
The second observation condition is a condition for setting a second acceleration voltage higher than the first acceleration voltage as an acceleration voltage of an electron microscope.
A biological tissue image processing system characterized by that. - 請求項3記載の生物組織画像処理システムにおいて、
前記第2元画像に基づいて第2正解画像を生成する正解画像作成部を含む、
ことを特徴とする生物組織画像処理システム。 The biological tissue image processing system according to claim 3,
A correct image creation unit that generates a second correct image based on the second original image;
A biological tissue image processing system characterized by that. - 請求項4記載の生物組織画像処理システムにおいて、
前記正解画像作成部は、前記第2元画像において前記注目要素の深部像を強調する強調部を含む、
ことを特徴とする生物組織画像処理システム。 The biological tissue image processing system according to claim 4, wherein
The correct image creation unit includes an enhancement unit that enhances a deep image of the element of interest in the second original image.
A biological tissue image processing system characterized by that. - 請求項5記載の生物組織画像処理システムにおいて、
前記注目要素は細胞膜であり、
前記注目要素の深部像は前記細胞膜の深部像である、
ことを特徴とする生物組織画像処理システム。 The biological tissue image processing system according to claim 5, wherein
The element of interest is a cell membrane,
The deep image of the element of interest is a deep image of the cell membrane,
A biological tissue image processing system characterized by that. - 請求項6記載の装置において、
前記正解画像作成部は、
前記強調部による処理を経た画像を複数の小領域に分割するセグメンテーションを実行するセグメンテーション部と、
前記セグメンテーション後の画像において、前記細胞膜に相当する複数の小領域に対してラベリングを行って前記第2正解画像を生成するアノテーション部と、
を含む、ことを特徴とする生物組織画像処理システム。 The apparatus of claim 6.
The correct image creation unit
A segmentation unit that performs segmentation to divide an image that has undergone processing by the enhancement unit into a plurality of small regions;
In the segmented image, an annotation unit that generates a second correct image by labeling a plurality of small regions corresponding to the cell membrane;
A biological tissue image processing system comprising: - 請求項4記載の生物組織画像処理システムにおいて、
前記機械学習型の推定器は第1推定器であり、
前記正解画像作成部は機械学習型の第2推定器を含む、
ことを特徴とする生物組織画像処理システム。 The biological tissue image processing system according to claim 4, wherein
The machine learning type estimator is a first estimator;
The correct image creation unit includes a machine learning type second estimator,
A biological tissue image processing system characterized by that. - 電子顕微鏡により生物組織である解析対象試料を観察することにより生成された画像に対して前記解析対象試料に含まれる注目要素を推定する処理を適用する機械学習型の推定器を含む生物組織画像処理システムにおいて、前記推定器に学習をさせるための機械学習方法であって、
前記推定器に対して、第1加速電圧の下で電子顕微鏡により生物組織としての学習対象試料を観察することにより生成された学習用の第1元画像、及び、それに対応する学習用の第1正解画像を与える工程と、
前記推定器に対して、前記第1加速電圧よりも高い第2加速電圧の下で電子顕微鏡により前記学習対象試料を観察することにより生成された学習用の第2元画像、及び、それに対応する学習用の第2正解画像を与える工程と、
を含む、ことを特徴とする機械学習方法。 Biological tissue image processing including a machine learning type estimator that applies processing for estimating an element of interest included in the analysis target sample to an image generated by observing the analysis target sample as a biological tissue with an electron microscope A machine learning method for causing the estimator to learn in a system,
For the estimator, a first original image for learning generated by observing a learning target sample as a biological tissue with an electron microscope under a first acceleration voltage, and a first learning image corresponding thereto. Providing a correct image;
A second original image for learning generated by observing the learning object sample with an electron microscope under a second acceleration voltage higher than the first acceleration voltage with respect to the estimator, and corresponding thereto Providing a second correct image for learning;
A machine learning method comprising: - 電子顕微鏡により生物組織である解析対象試料の観察により生成された画像に対して前記解析対象試料に含まれる注目要素を推定する処理を適用する機械学習型の推定器を含む生物組織画像処理システムにおいて、前記推定器に学習をさせるための機械学習方法を実行するためのプログラムであって、
前記推定器に対して、第1加速電圧の下で電子顕微鏡により生物組織である学習対象試料を観察することにより生成された学習用の第1元画像、及び、それに対応する学習用の第1正解画像を与える機能と、
前記推定器に対して、前記第1加速電圧よりも高い第2加速電圧の下で顕微鏡により前記学習対象試料を観察することにより生成された学習用の第2元画像、及び、それに対応する学習用の第2正解画像を与える機能と、
を含む、ことを特徴とするプログラム。 In a biological tissue image processing system including a machine learning type estimator that applies a process of estimating an element of interest included in an analysis target sample to an image generated by observing the analysis target sample as a biological tissue with an electron microscope , A program for executing a machine learning method for causing the estimator to learn,
A first original image for learning generated by observing a learning target sample that is a biological tissue with an electron microscope under a first acceleration voltage, and a first learning image corresponding to the estimator. The ability to give correct images,
A second original image for learning generated by observing the learning target sample with a microscope under a second acceleration voltage higher than the first acceleration voltage with respect to the estimator, and learning corresponding thereto A function for providing a second correct image for
The program characterized by including.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2018-099388 | 2018-05-24 | ||
JP2018099388A JP7181002B2 (en) | 2018-05-24 | 2018-05-24 | BIOLOGICAL TISSUE IMAGE PROCESSING SYSTEM AND MACHINE LEARNING METHOD |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2019225506A1 true WO2019225506A1 (en) | 2019-11-28 |
Family
ID=68616942
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2019/019741 WO2019225506A1 (en) | 2018-05-24 | 2019-05-17 | Biological tissue image processing system, and machine learning method |
Country Status (2)
Country | Link |
---|---|
JP (1) | JP7181002B2 (en) |
WO (1) | WO2019225506A1 (en) |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH05290786A (en) * | 1992-04-10 | 1993-11-05 | Hitachi Ltd | Method and device for displaying image of sample to be scanned and sample to be used therefor |
JP2004212355A (en) * | 2003-01-09 | 2004-07-29 | Hitachi Ltd | Biological electron microscope and method of sample observation |
JP2007263932A (en) * | 2006-03-30 | 2007-10-11 | Kose Corp | Method for acquiring inside image of cosmetic-applied film |
US20100183217A1 (en) * | 2007-04-24 | 2010-07-22 | Seung H Sebastian | Method and apparatus for image processing |
JP2015149169A (en) * | 2014-02-06 | 2015-08-20 | 株式会社日立ハイテクノロジーズ | Charged particle beam device, image generation method, and observation system |
WO2017221592A1 (en) * | 2016-06-23 | 2017-12-28 | コニカミノルタ株式会社 | Image processing device, image processing method, and image processing program |
-
2018
- 2018-05-24 JP JP2018099388A patent/JP7181002B2/en active Active
-
2019
- 2019-05-17 WO PCT/JP2019/019741 patent/WO2019225506A1/en active Application Filing
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH05290786A (en) * | 1992-04-10 | 1993-11-05 | Hitachi Ltd | Method and device for displaying image of sample to be scanned and sample to be used therefor |
JP2004212355A (en) * | 2003-01-09 | 2004-07-29 | Hitachi Ltd | Biological electron microscope and method of sample observation |
JP2007263932A (en) * | 2006-03-30 | 2007-10-11 | Kose Corp | Method for acquiring inside image of cosmetic-applied film |
US20100183217A1 (en) * | 2007-04-24 | 2010-07-22 | Seung H Sebastian | Method and apparatus for image processing |
JP2015149169A (en) * | 2014-02-06 | 2015-08-20 | 株式会社日立ハイテクノロジーズ | Charged particle beam device, image generation method, and observation system |
WO2017221592A1 (en) * | 2016-06-23 | 2017-12-28 | コニカミノルタ株式会社 | Image processing device, image processing method, and image processing program |
Non-Patent Citations (3)
Title |
---|
LIGONG HAN ET AL.: "Learning Generative Models of Tissue Organization with Supervised GANs", 2018 IEEE WINTER CONFERENCE ON APPLICATIONS OF COMPUTER VISION, 12 March 2018 (2018-03-12), pages 682 - 690, XP033337732, DOI: 10.1109/WACV.2018.00080 * |
OSUMI MASAKO: "Low-voltage scanning electronic microscopy - application of biological specimens", ELECTRONIC MICROSCOPE, vol. 31, no. 1, pages 45 - 50 * |
UCHIHASHI KENSHI ET AL.: "Neuronal cell memebrane segmentation from electron microscopy images using a hostile generation model", THE 31ST ANNUAL CONFERENCE OF THE JAPANESE SOCIETY FOR ARTIFICAIL INTELLIGENCE, 2017, pages 1 - 4 * |
Also Published As
Publication number | Publication date |
---|---|
JP2019203803A (en) | 2019-11-28 |
JP7181002B2 (en) | 2022-11-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11854201B2 (en) | Biological tissue image processing system, and machine learning method | |
Klumpe et al. | A modular platform for automated cryo-FIB workflows | |
CN109584156A (en) | Micro- sequence image splicing method and device | |
US11594051B2 (en) | Microscope system and projection unit | |
CN111445478A (en) | Intracranial aneurysm region automatic detection system and detection method for CTA image | |
CN113012172A (en) | AS-UNet-based medical image segmentation method and system | |
Bartesaghi et al. | An energy-based three-dimensional segmentation approach for the quantitative interpretation of electron tomograms | |
CN113744195B (en) | Automatic hRPE cell microtubule detection method based on deep learning | |
Pollastri et al. | Improving skin lesion segmentation with generative adversarial networks | |
Krawczyk et al. | Bones detection in the pelvic area on the basis of YOLO neural network | |
CN114764189A (en) | Microscope system and method for evaluating image processing results | |
TW201709150A (en) | Spatial multiplexing of histological stains | |
CN115546605A (en) | Training method and device based on image labeling and segmentation model | |
CN112633248A (en) | Deep learning all-in-focus microscopic image acquisition method | |
US8837795B2 (en) | Microscopy of several samples using optical microscopy and particle beam microscopy | |
Fortun et al. | Reconstruction from multiple particles for 3D isotropic resolution in fluorescence microscopy | |
Rangan et al. | Deep reconstructing generative networks for visualizing dynamic biomolecules inside cells | |
CN116797463B (en) | Feature point pair extraction method and image stitching method | |
WO2019225506A1 (en) | Biological tissue image processing system, and machine learning method | |
JP7181000B2 (en) | BIOLOGICAL TISSUE IMAGE PROCESSING APPARATUS AND METHOD | |
DE102021114287A1 (en) | Microscopy system and method for generating stylized contrast images | |
JP7181003B2 (en) | BIOLOGICAL TISSUE IMAGE PROCESSING APPARATUS AND METHOD | |
CN110189283A (en) | Remote sensing images DSM fusion method based on semantic segmentation figure | |
CN113066108B (en) | Anti-occlusion visual target tracking method and device based on ECO algorithm | |
JP7224484B2 (en) | Application method, observation method, application condition estimation system, section preparation device, and server |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19807182 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19807182 Country of ref document: EP Kind code of ref document: A1 |