JP2019203803A - Biological tissue image processing system and machine learning method - Google Patents

Abstract

To enhance estimation accuracy of a cell membrane in a biological tissue image processing system to estimate a cell membrane in a biological tissue.SOLUTION: In an analysis process, a membrane estimator 42 applies a process to estimate a cell membrane to an original image 38L as a low voltage image and outputs a membrane likelihood map 60. In a learning process, a first original image 36L as a low voltage image and first correct images 223, 64A corresponding to the same are input into the membrane estimator 42 and a second original image 36H as a high voltage image and a second correct image 224 corresponding to the same aare also input thereinto. A second image processing part 202 is used for creating the second correct image from the second original image 36H.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a biological tissue image processing system and a machine learning method.

  As a technique for analyzing or imaging a three-dimensional structure of a biological tissue, three-dimensional microscopy is known. In three-dimensional microscopy, an electron microscope is generally used. For example, three-dimensional microscopy using Scanning Electron Microscope, Focused Ion Beam SEM (FIB-SEM) method, Serial Block-Face SEM (SBF-SEM) method, and serial section SEM (Serial Section SEM method (see Non-Patent Document 1) has been proposed. The continuous section SEM method is also called an array tomography method.

  In the continuous section SEM method, a plurality of sample sections (ultra-thin pieces) continuous in the depth direction are cut out from a sample of a biological tissue and arranged on a substrate. Each sample section on the substrate is sequentially observed with a scanning electron microscope, whereby a plurality of images are acquired. Based on the acquired images, the three-dimensional structure of a specific organ (cell, mitochondria, nucleus, etc.) contained in the biological tissue is analyzed or imaged. According to the continuous section SEM method, it is possible to observe a sample section that has already been observed again.

Koka et al. “Application of 3D structure analysis of serial section SEM method and Golgi apparatus” microscope, Vol. 49, No. 3, 2014.

  In the processing of images obtained by observing biological tissues with an electron microscope, a machine learning type estimator is used to detect or identify a target component (Target Component) such as a cell, a cell membrane, or an organelle in the cell. It is possible. In that case, it is desirable to train the estimator so that the estimation accuracy of the estimator is increased.

  The content of the image generated by the electron microscope varies depending on observation conditions such as acceleration voltage. From that perspective, there is an idea that the observation conditions at the time of acquisition of the learning image (observation conditions at the time of learning) should be matched with the observation conditions at the time of acquisition of the image for analysis (observation conditions at the time of analysis). Arise. However, if the estimation target in the estimator has a significant thickness in the biological tissue, the above idea does not necessarily hold, and it may be better to give diversity to the observation conditions during learning. It is thought that there is.

  An object of the present invention is to increase the estimation accuracy of a target element in a biological tissue image processing system having a machine learning type estimator that estimates a target element in a biological tissue. Alternatively, an object of the present invention is to cause an estimator to perform learning so as to increase the estimation accuracy of an element of interest in the machine learning process of the estimator that estimates the element of interest in a biological tissue.

  The biological tissue image processing system according to the present invention is a machine learning that applies an electron microscope that generates an image by observing an analysis target that is a biological tissue, and a process that estimates a target element included in the analysis target with respect to the image. An estimator of the type, and the estimator is a learned estimator that has undergone learning using a learning original image set, and the learning original image set is subjected to biological tissue using an electron microscope under a first observation condition. A first original image generated by observing the learning target sample, and a second original image generated by observing the learning target sample with a microscope under a second observation condition, The first observation condition is a condition set for an electron microscope when observing the analysis target sample, and the second observation condition can be observed over a depth range different from the first observation condition. A condition, it is characterized in.

  The machine learning method according to the present invention applies a process of estimating a target element included in the analysis target sample to an image generated by observing the analysis target sample that is a biological tissue with an electron microscope. In a biological tissue image processing system including an estimator, a machine learning method for causing the estimator to learn, wherein the estimator learns as a biological tissue with an electron microscope under a first acceleration voltage. A step of providing a first original image for learning generated by observing a target sample and a first correct answer image for learning corresponding thereto, and higher than the first acceleration voltage for the estimator A step of providing a second original image for learning generated by observing the learning object sample with an electron microscope under a second acceleration voltage, and a second correct image for learning corresponding thereto. Includes, that is characterized in.

  The above method is realized as a hardware function or a software function. In the latter case, a program for executing the function is installed in the information processing apparatus via a network or a portable storage medium. The concept of the information processing apparatus includes a personal computer, an electron microscope system, and the like.

  ADVANTAGE OF THE INVENTION According to this invention, in the biological tissue image processing system which has a machine learning type estimator which estimates the attention element in biological tissue, the estimation precision of a attention element can be improved. Alternatively, according to the present invention, in the machine learning process of the estimator that estimates the element of interest in the biological tissue, it is possible to cause the estimator to perform learning so that the estimation accuracy of the element of interest increases.

It is a conceptual diagram which shows the biological tissue image processing system which concerns on embodiment. It is a block diagram which shows the 1st structural example of the main body of a biological tissue image processing apparatus. It is a block diagram which shows the structural example of a machine learning type | mold film | membrane estimator. It is a figure which shows the production | generation of a temporary membrane image based on a membrane likelihood map. It is a figure which shows an example of a work window. It is a figure which shows the film | membrane image before correction | amendment, and the film | membrane image after correction | amendment. It is a figure which shows an example of a labeling process. It is a figure which shows the change of the observation depth range etc. accompanying the change of an applied voltage. It is a figure for demonstrating contrast emphasis. It is a figure for demonstrating segmentation and correction by manual labor. It is a figure which shows an original image and a correct image. It is a figure which shows the film | membrane likelihood map which concerns on a comparative example, and the film | membrane likelihood map which concerns on embodiment. It is a block diagram which shows the 2nd structural example of a main body.

(A) Outline of Embodiment A biological tissue image processing system according to an embodiment includes an electron microscope and a machine learning type estimator. The electron microscope is an apparatus that generates an image by observing a sample to be analyzed which is a biological tissue, and preferably it is a scanning electron microscope. The estimator applies processing to estimate the element of interest (target element image) included in the sample to be analyzed on the image generated by the electron microscope, and the estimator performs learning using the original image set for learning. It is a learned estimator that passed through. In the learning original image set, the first original image generated by observing the learning target sample with the electron microscope under the first observation condition and the learning target sample with the microscope under the second observation condition are observed. 2nd original image produced | generated by this. The first observation condition is a condition set for the electron microscope when observing the analysis target sample, and the condition is also set for the electron microscope when observing the learning target sample. The second observation condition is a condition under which observation can be performed over a depth range different from the first observation condition, and the condition is a condition set for the electron microscope when observing the learning target sample.

  According to the above configuration, the estimation accuracy for the element of interest can be increased by using the learned estimator that has undergone a special learning process. For example, an image (for example, a deep image) that does not relatively occur under the first observation condition is likely to be correctly estimated as a part of the target tissue.

  The sample to be analyzed and the sample to be learned are generally the same (same type) biological tissue, but they are usually physically different. The analysis target sample and the learning target sample are generally observed with the same electron microscope, but they may be observed with different electron microscopes. The element of interest is a specific tissue component, and in embodiments, the element of interest is a cell membrane. The cell itself, the cell lumen (cytoplasm) surrounded by the cell membrane, the organelle (organelle) in the cell, and the like may be considered as elements of interest. The machine learning type estimator is configured by CNN, for example, but other types of machine learning type estimators may be used. In the embodiment, a plurality of images are acquired by the continuous section SEM method (array tomography method), but a plurality of images may be acquired by the FIB-SEM method, the SBF-SEM method, and other methods. The above configuration can be applied to two-dimensional microscopy as well as three-dimensional microscopy.

  In the embodiment, the second observation condition is a condition under which observation can be performed over a depth range larger than the first observation condition. When switching the observation conditions, the acceleration voltage, irradiation current, and the like may be changed. In the embodiment, the first observation condition is a condition for setting the first acceleration voltage as the acceleration voltage of the electron microscope, and the second observation condition is a second acceleration voltage higher than the first acceleration voltage as the acceleration voltage of the electron microscope. Is a condition for setting. When the acceleration voltage is increased, information from a deeper position in the observation object can be easily obtained. In other words, an object existing deeper can be easily imaged. If the estimator is trained using the first original image and the second original image, the deep image of the tissue of interest is detected even if the input image of the estimator is an image generated under the first observation condition. It becomes easy to be done. If the sample to be analyzed is observed under the first observation condition in the analysis process, damage to the sample to be analyzed can be prevented or reduced.

  In the embodiment, the biological tissue image processing system includes a correct image creation unit that generates a second correct image based on the second original image. If the correct image creation unit for the second original image is provided, image processing suitable for the second original image can be easily applied to the image when creating the correct image. In the embodiment, the correct image creation unit includes an enhancement unit that emphasizes the deep image of the element of interest in the second original image. According to this configuration, the deep part image that is a part of the target element (target element image) can be easily learned. In the embodiment, the element of interest is a cell membrane, and the deep image of the element of interest is a deep image of the cell membrane. For example, when analyzing a three-dimensional structure of a cell, it is desired to estimate or detect even a cell membrane present inside a biological tissue section. The above configuration meets such a demand.

  In the embodiment, the correct image creation unit includes a segmentation unit that performs segmentation for dividing an image that has been processed by the enhancement unit into a plurality of small regions, and a plurality of small regions that correspond to cell membranes in the segmented image. An annotation unit that performs labeling to generate a second correct image. If segmentation (small area division) is performed prior to labeling, the labeling work becomes easier or the labeling becomes accurate.

  In the embodiment, the correct image creation unit includes a machine learning type second estimator. This configuration uses another estimator when creating a correct image for the first estimator. The element of interest in the second original image is estimated by the second estimator, and a second correct image is created based on the estimation result.

  The machine learning method according to the embodiment includes a first original image for learning generated by observing a learning target sample as a biological tissue with an electron microscope under a first acceleration voltage with respect to an estimator, and A step of providing a first correct image for learning corresponding thereto, and learning generated by observing a learning target sample with an electron microscope under a second acceleration voltage higher than the first acceleration voltage with respect to the estimator And providing a second original image for learning and a second correct image for learning corresponding thereto.

  In the embodiment, in the machine learning process of the estimator, the first image pair including the first original image and the first correct image is input to the estimator as the first teacher data, and the second original image and the second correct image are used. The second image pair is input to the estimator as second teacher data. The first correct image is created manually from the first original image, or is created by correcting the estimation result of the estimator for the first original image. In the embodiment, the second correct image is created from the second original image by the correct image creating unit. Both the first correct image and the second correct image may be generated by the correct image creating unit.

(B) Details of Embodiment FIG. 1 shows a biological tissue image processing system according to an embodiment. The illustrated biological tissue image processing system 10 is a system for analyzing and imaging a three-dimensional structure of a biological tissue. Using this biological tissue image processing system, for example, an image that three-dimensionally represents nerve cells in the brain of a human body or an animal is generated. Any tissue, organ, etc. in the organism can be the subject of analysis.

  In the configuration example shown in FIG. 1, the biological tissue image processing system 10 includes a sample pretreatment device 12, a continuous section creation device 14, a scanning electron microscope (SEM) 16, and a biological tissue image processing device 18. .

  The sample pretreatment device 12 is a device that performs pretreatment on the tissue 20 extracted from a living body, or corresponds to various instruments for the pretreatment. Examples of the pretreatment include fixing treatment, dyeing treatment, conductive treatment, resin embedding treatment, and shaping treatment. All or some of them are implemented as necessary. In the staining treatment, osmium tetroxide, uranium acetate, lead citrate, or the like may be used. A staining process may be performed on each sample section described below. Some or all of the steps included in the pretreatment may be performed manually.

  The continuous section preparation device 14 is provided outside the SEM 16 or inside the SEM 16. A plurality of sample sections 24 arranged in the depth direction (Z direction) are cut out from the cubic sample after the pretreatment by the continuous section creating apparatus 14. In that case, an apparatus such as an ultramicrotome may be used. The work may be performed manually. A plurality of sample sections 24 constitute a sample section group 22. Actually, the plurality of sample sections 24 cut out are arranged on the substrate 28 in a predetermined arrangement. The substrate 28 is, for example, a glass substrate or a silicone substrate. In FIG. 1, the sample section array 22 </ b> A composed of two sample section rows is configured on the substrate 28, but this is merely an example. A sample unit 26 is configured by the substrate 28 and the sample section array 22A.

  Incidentally, the vertical and horizontal sizes of the individual sample sections 24 are, for example, on the order of nm or μm. A sample section 24 having a larger size (for example, a size on the order of mm) may be produced. The thickness (size in the Z direction) of each sample section 24 is, for example, several nm to several hundred nm, and in the embodiment, the thickness is, for example, in the range of 30 to 70 nm. Any numerical value given in this specification is an example.

  The sample observed in the learning process of the estimator is a learning sample, that is, a learning target sample. A plurality of tissue sections are prepared from the learning target sample. The sample observed in the analysis process is the sample to be analyzed. A plurality of tissue sections are prepared from the sample to be analyzed. In FIG. 1, both the learning target sample and the analysis target sample are expressed as a tissue 20. Each sample slice cut out from the learning target sample and each sample slice cut out from the analysis target sample are both expressed as a sample slice 24. Note that the learning target sample and the analysis target sample are usually separate, but they may be the same. For example, the analysis process and the learning process may be executed in parallel.

  The SEM 16 includes an electron gun, a deflector (scanner), an objective lens, a sample chamber, a detector 34, a control unit 204, and the like. In the sample chamber, a stage for holding the sample unit 26 and a moving mechanism for moving the stage are provided. The control unit 204 controls the operation of the moving mechanism, that is, the movement of the stage. Specifically, the electron beam 30 is irradiated to a specific sample section 24 selected from the sample section array 22A. While scanning the irradiation position (for example, raster scanning), the reflected electrons 32 emitted from each irradiation position are detected by the detector 34. Thereby, an SEM image is formed. This is performed for each sample section 24. The control unit 204 has a function of setting an acceleration voltage for forming an electron beam. In general, when the acceleration voltage is increased, information from a deeper position in the sample section 24 is easily obtained.

  In addition, when the direction orthogonal to the Z direction is defined as the X direction and the direction orthogonal to the Z direction and the X direction is defined as the Y direction, it is desirable that the X direction observation range and the Y direction observation range coincide with each other. The observation range (electron beam two-dimensional scanning range) in each sample section 24 is determined. Instead of the reflected electrons 32, secondary electrons or the like may be detected.

  In the embodiment, in the learning process of the estimator, under the control of the control unit 204, the first accelerating voltage (low voltage) as the first observation condition and the second as the second observation condition are obtained in units of 24 sample sections. The acceleration voltage (high voltage) is alternately set. That is, the individual sample slices 24 cut out from the learning target sample are observed by scanning with the electron beam 30L formed under the first acceleration voltage (low voltage observation), whereby the low voltage image (first original image). 36L is generated. In addition, the individual specimen sections 24 are observed by scanning with the electron beam 30H formed under the second acceleration voltage, thereby generating a high voltage image (second original image) 36H (high voltage observation). Usually, low-voltage observation and low-voltage observation are continuously performed on one sample section 24, and this is repeated for each sample section 24. As a result, the low voltage image stack 35L is configured by the plurality of low voltage images 36L, and the high voltage image stack 35H is configured by the plurality of high voltage images 36H.

  On the other hand, in the analysis process, only the first acceleration voltage is set as an observation condition under the control of the control unit 204. That is, the individual sample sections 24 cut out from the analysis target sample are observed by scanning with the electron beam 30L formed under the first acceleration voltage (low voltage observation), and thereby a low voltage image (analysis original image). 38L is generated. A plurality of low voltage images 38L constitute a low voltage image stack 37L. The first acceleration voltage is, for example, 1 keV or 2 keV, and the second acceleration voltage is, for example, 3 keV, 5 keV, or 7 keV. It is desirable to determine the first acceleration voltage and the second acceleration voltage in consideration of the sample, section thickness, target tissue, observation purpose, and other circumstances.

  The image stacks 35L, 35H, and 37L include a plurality of images 36L, 36H, and 38L corresponding to a plurality of depths in the Z direction (in other words, arranged in the Z direction in the data storage space). Each of the images 36L, 36H, and 38L is an original image or an input image when viewed from the biological tissue image processing apparatus 18 side. The images 36L, 36H, and 38L are electronic data, and the images 36L, 36H, and 38L are transmitted from the SEM 16 to the biological tissue image processing apparatus 18 via a network or a portable storage medium.

  The biological tissue image processing apparatus 18 is configured by a personal computer in the illustrated configuration example. The biological tissue image processing apparatus 18 may be incorporated in the SEM 16, or the biological tissue image processing apparatus 18 may be incorporated in a system computer that controls the SEM 16 or the like. The SEM 16 may be controlled by the biological tissue image processing apparatus 18.

  The biological tissue image processing apparatus 18 includes a main body 40, a display 46 and an input device 48. The plurality of functions of the main body 40 will be described in detail later with reference to FIGS. In FIG. 1, two typical functions (first image processing function and second image processing function) exhibited by the main body 40 are represented as blocks. Specifically, the main body 40 includes a first image processing unit 200 including a machine learning type film estimator, and a second image processing unit 202 as a correct image creation unit. The display 46 is configured by an LCD, an organic EL display device, or the like. The input device 48 includes a keyboard operated by a user, a pointing device, and the like.

  FIG. 2 shows a first configuration example of the main body 40. The main body 40 has the first image processing unit 200, the second image processing unit 202, and the volume data processing unit 56 as described above. The entity of each configuration shown in FIG. 2 is software, that is, a program executed by a general-purpose processor such as a CPU or GPU, except for a portion corresponding to a user's work or action. However, part or all of the configuration may be configured by a dedicated processor or other hardware. All or part of the functions of the biological tissue image processing apparatus may be executed by one or a plurality of information processing devices existing on the network.

  Hereinafter, first, the configuration that functions in the analysis process will be described in detail, and then the configuration that functions in the learning process will be described in detail. However, since the first image processing unit 200 has a configuration that functions in both the analysis process and the learning process, the operation in the learning process is also referred to in describing the operation in the analysis process. I will do it.

  The first image processing unit 200 includes a machine learning type film estimator 42, a binarizer (image generator) 50, a correction unit 52, a labeling processing unit 54, and the like.

  The film estimator 42 functions as a film estimation unit, and applies a film estimation process to the input image 206, thereby outputting a film likelihood map 60. In the illustrated configuration example, the membrane estimator 42 is configured by a CNN (Convolutional Neural Network) which is a machine learning type membrane estimator. A specific configuration example will be described later with reference to FIG. The learning process is executed in advance prior to the actual operation of the CNN, that is, the analysis process so that the film is correctly estimated by the CNN. The learning process includes a primary learning process (initial learning process) and a secondary learning process.

  In the primary learning process, a plurality of image pairs constituting the teacher data are given to the membrane estimator 42, whereby the CNN parameter group in the membrane estimator 42 is improved (optimized). That is, the machine learning result is accumulated in the film estimator 42. Here, the plurality of image pairs include a plurality of first image pairs and a plurality of second image pairs. Each first image pair is composed of a first original image (low voltage image) 36L and a corresponding correct image 223. Each second image pair includes a second original image (high voltage image) 36H and a correct image 224 corresponding thereto. The correct image 223 is created, for example, by manual work on the first original image 36L. The correct image 223 may be created from the first original image 36L by an unsupervised machine learning device, a simple classifier (for example, SVM (Support Vector Machine)), or the like. On the other hand, the correct image 224 is created by the second image processing unit 202 based on the second original image, as will be described in detail later. The correct image 223 may be created from the first original image 36L by the second image processing unit 202.

  In the subsequent secondary learning process, on the premise that the membrane estimator 42 works to some extent through the primary learning process, a plurality of image pairs are given to the membrane estimator 42 as teacher data as in the primary learning process. In the embodiment, the teacher data includes a plurality of image pairs used in the primary learning process and a plurality of image pairs added in the secondary learning process. The plurality of added image pairs include a plurality of first image pairs and a plurality of second image pairs. Each first image pair includes a first original image 36L and a correct image 64A corresponding to the first original image 36L. Each second image pair includes a second original image 36H and a correct image 224 corresponding to the second original image 36H.

  The correct image 64A is created by the configuration from the film estimator 42 to the processing tool unit 44. Specifically, when the first original image 36L is input to the membrane estimator 42, the membrane likelihood map 60 is output from the membrane estimator 42 as an estimation result image. A correct image 64 </ b> A is created through generation of a mask image 62 based on the film likelihood map 60 and user (expert) correction of the mask image 62 using the processing tool unit 44. These processes will be described in detail later. It is also conceivable to use the mask image 62 as the correct image 62A.

  The CNN parameter group in the film estimator 42 is further improved by the secondary learning process. That is, the machine learning result is further accumulated in the film estimator 42. In the secondary learning process, for example, it is determined that the result of the estimation process for the input image 206 (the first original image 36L and the second original image 36H) is sufficiently similar to the corresponding correct images 223, 224, and 64A. If finished. Thereafter, re-learning of the film estimator 42 is executed by the same method as described above as necessary. In the embodiment, the second image processing unit 202 functions as correct image creation means in both the primary learning process and the secondary learning process. In the analysis process, an original image (low voltage image) 38L as the input image 206 is input to the membrane estimator 42.

  The database 57 stores a plurality of original images 36L, 36H, and 38L and a plurality of correct images 223, 224, and 64A. A plurality of film likelihood maps 60 may be stored there. The film estimator 42 and the database 57 may be integrated. As a machine learning type film estimator, U-net may be used, or a support vector machine, a random forest, or the like may be used.

  The binarizer 50 functions as an image generator. Specifically, the binarizer 50 is a module that generates a temporary membrane image 62 by binarization processing on the membrane likelihood map 60, as will be exemplified later with reference to FIG. The film likelihood map 60 includes a plurality of film likelihoods arranged two-dimensionally. Each film likelihood is a numerical value indicating the probability (probability) of being a film. The film likelihood is, for example, between 0 and 1. The film likelihood map 60 can also be understood as a film likelihood image. In the embodiment, a threshold is set in the binarizer 50, and a film likelihood that is equal to or higher than the threshold is converted to 1, and a film likelihood that is less than the threshold is converted to 0. The image generated as a result is a temporary membrane image 62. In the embodiment, for the sake of convenience, the film image before correction is referred to as a temporary film image 62 in order to distinguish the film image before correction from the film image after correction.

  Note that an image generating unit 61 is configured by both the film estimator 42 and the binarizer 50. The entire image generation unit 61 may be configured by CNN or the like. Even in such a case, the stepwise generation of the membrane likelihood map and the temporary membrane image can be considered. It is also possible to use the temporary membrane image 62 as the correct image 62A. Prior to binarization processing, processing such as noise removal and edge enhancement may be applied to the film likelihood map 60.

  The processing tool unit 44 functions as a work support unit or work support means. The processing tool unit 44 is provided across the first image processing unit 200 and the second image processing unit 202 in the illustrated configuration example. The processing tool unit 44 has a display processing function and an image processing function from the viewpoint of information processing. From the viewpoint of work, it has a correction function and a labeling function. In FIG. 2, these functions are expressed as a correction unit 52, a labeling processing unit 54, and a correction unit 214.

  The correction unit 52 displays, as a work target image, a mask image that is a work target for the user via a work window illustrated in FIG. 5 later, and accepts a user's correction instruction on the work target image. Is. The correction contents are reflected in the mask image that is the work target image. For example, when the work target image includes a discontinuous portion of the film, a film pixel group is added to the discontinuous portion. For example, when a part other than the film is included in the work target image and the part is mistaken as a film, the film pixel group constituting the part is deleted. In such addition and deletion, the correction unit 52 is a module that supports the user's work or operation and manages each mask image.

  In the illustrated configuration example, in addition to the generated temporary membrane image 62, the input original image (input image) 206 and the generated membrane likelihood are generated with respect to the correction unit 52 (or the processing tool unit 44). The degree map 60 is also sequentially input. Thus, the original image 206 or the membrane likelihood map 60 corresponding to the work target image can be displayed together with or instead of the temporary membrane image as the work target image.

  The labeling processing unit 54 is a module for performing labeling (painting and labeling) on individual regions (cell lumens) included in the corrected membrane image (or uncorrected membrane image). Labeling includes manual labeling by the user and automatic labeling. At the stage where the correction work and the labeling work are completed, the three-dimensional labeling data 66 in which the cell lumen is distinguished from the others is configured. This is sent to the volume data processing unit 56. The correction unit 214 will be described later.

  The volume data processing unit 56 includes an analysis unit 56A and a rendering unit 56B. In the illustrated configuration example, the volume data processing unit 56 receives an original image stack 37L composed of a plurality of original images. The original image stack 37L constitutes volume data. As described above, the three-dimensional labeling data 66 is also input to the volume data processing unit 56. It is also a kind of volume data.

  For example, the analysis unit 56A analyzes a target organ (for example, a nerve cell) based on the three-dimensional labeling data 66. For example, the form, volume, length, etc. may be analyzed. At that time, the original image stack 37L is analyzed with reference to the three-dimensional labeling data. The rendering unit 56B forms a three-dimensional image (stereoscopic expression image) based on the three-dimensional labeling data 66. For example, an imaging portion may be extracted from the original image stack 37L based on the three-dimensional labeling data 66, and a rendering process may be applied to the extracted image portion.

  Next, the second image processing unit 202 will be described. The second image processing unit 202 is a module that generates the second correct image 224 based on the second original image (high voltage image) 36H as the input image 208.

  The high voltage image as the second original image 36H includes more deep images of the cell membrane than the low voltage image as the first original image 36L. This is because when the acceleration voltage is increased, information from a deeper position inside the sample slice can be obtained (this will be described in detail later with reference to FIG. 8). In a high voltage image, a deep image of a cell membrane typically appears as a haze (or like a haze) in the vicinity of a shallow image (surface layer image) of the cell membrane. Even in a low-voltage image, a deep image appears as a haze, but the amount thereof is relatively small, and in a low-voltage image, a blurred image in which it is not clear whether the image is a deep image tends to occur.

  In the three-dimensional labeling inside the cell, it is desired that not only the shallow image of the cell membrane but also the deep image of the cell membrane is detected as a part of the cell membrane. For example, a shallow image and a deep image of the cell membrane should be taken in order, and a shape corresponding to each depth should be determined. However, the membrane estimator may be required to input a low voltage image due to damage to the sample section, imaging time, and other reasons. Assuming that a low-voltage image is input, in order to facilitate detection of a deep image of the cell membrane as a part of the cell membrane, in the learning stage of the membrane estimator, the membrane estimator is used together with the low-voltage image as its teaching data. A high voltage image is given.

  The second image processing unit 202 includes a haze enhancement unit 210, a segmentation unit 212, and a correction unit (annotation unit) 214 in the illustrated configuration example.

  The haze enhancement unit 210 applies processing for enhancing the deep image of the cell membrane included in the input image 208. Specifically, contrast enhancement is performed as moya enhancement.

  The segmentation unit 212 applies segmentation to the image 218 after contrast enhancement. Segmentation is a process of dividing image content into a plurality of small areas. This segmentation is performed to make it easier to do the work in the next annotation.

  The correction unit 214 functions as correction means or annotation means. In the sense of a tool for user work, the correction unit 214 exhibits the same function as the correction unit 52 described above. Therefore, in FIG. 2, the correction unit 214 is positioned as a part of the processing tool unit 44. Specifically, the correction unit 214 displays the segmented image 220, accepts the user's membrane designation and membrane correction for the image 220, and as a result, the second correct image 224 as a membrane image showing only the cell membrane. Is to create. A plurality of small regions are displayed in the image 220, and a membrane label is given by the user to the small region corresponding to the cell membrane. The work corresponds to annotation and painting. In order to improve the workability, a work window similar to the above may be displayed. In the learning process (primary learning process and secondary learning process) of the film estimator 42, the second correct image 224 created as described above is given to the film estimator 42.

  According to the above configuration, since the estimation result by the membrane estimator 42 can be used in the production of the correct image 64A, and the correction unit 52 can be used, compared to the case where they cannot be used, The burden of preparing the correct image 64A is greatly reduced. In the analysis process after the secondary learning process, the combined use of the film estimator 42 and the processing tool unit 44 can improve the quality of the image group to be analyzed or rendered, and easily and quickly generate the image group. Is possible.

  Further, according to the above configuration, on the premise that a low voltage image is input to the membrane estimator 42 in the analysis process, the membrane estimator 42 can learn using a high voltage image in addition to a low voltage image. Since this is done, the film estimation accuracy of the film estimator 42 can be increased. Actually, a low voltage image and a high voltage image representing the same part of the same sample section for learning are generated, and both of them are given to the film estimator 42 as a part of the teacher data.

  Furthermore, according to the above configuration, when creating the second correct image, the possibility that the deep image of the cell membrane is regarded as a part of the cell membrane can be increased, so that the quality of the second correct image can be improved.

  In the primary learning process and the secondary learning process, a plurality of first original images and a plurality of second original images are sequentially input to the film estimator 42 in a predetermined order. Those images may be batch processed.

  FIG. 3 schematically shows a configuration example of the film estimator 42. The membrane estimator 42 has a number of layers, including an input layer 80, a convolution layer 82, a pooling layer 84, an output layer 86, and the like. They act according to the CNN parameter group 88. The CNN parameter group 88 includes a number of weighting factors, a number of bias values, and the like. The CNN parameter group 88 is initially configured by an initial value group 94. For example, the initial value group 94 is generated using random numbers or the like.

  In the learning process, the evaluation unit 90 and the update unit 92 function. For example, the evaluation unit 90 calculates an evaluation value based on a plurality of image pairs (original images 36L and 36H and corresponding correct images 223, 224 and 64A) constituting the teacher data. Specifically, the evaluation value is calculated by sequentially giving the result 60A of the estimation processing for the original images 36L and 36H and the correct images 223, 224 and 64A corresponding to the original images 36L and 36H to the error function. The updating unit 92 updates the CNN parameter group 88 so that the evaluation value changes in a favorable direction. By repeating this, the CNN parameter group 88 is optimized as a whole. Actually, the end of the learning process is determined when the evaluation value reaches a certain value. The configuration shown in FIG. 3 is merely an example, and it is possible to use estimators having various structures as the film estimator 42.

  FIG. 4 shows the operation of the binarizer. A film likelihood map 60 is output from the film estimator. The film likelihood map 60 is composed of a plurality of film likelihoods 60a corresponding to a plurality of pixels, and each film likelihood 60a is a numerical value indicating the probability of being a film. As indicated by reference numeral 50A, the binarizer binarizes the membrane likelihood map 60, thereby generating a temporary membrane image 62 as a binarized image. In binarization, each film likelihood 60a is compared with a threshold value. For example, a membrane likelihood 60a that is equal to or greater than the threshold is converted into a pixel 62a having a value of 1, and a membrane likelihood 60a that is less than the threshold is converted into a pixel 62b having a value of 0. The pixel 62a is handled as a pixel (film pixel) constituting the film. The threshold value can be variably set by the user. The setting may be automated. For example, the threshold value may be changed while observing the temporary membrane image 62.

  FIG. 5 illustrates a work window displayed by the machining tool unit. In the illustrated example, the work window 100 includes a display image 102. The display image 102 is a composite image (composite image) composed of a mask image (work target image) corresponding to the depth selected by the user and the original image. An original image as a gray scale image constitutes a background image, and a temporary membrane image (work target image) as a color image (for example, a blue image) is superimposed and displayed on the background image. Note that the illustrated display image 102 shows brain tissue, in which cross sections of a plurality of cells appear. At the same time, a cross section of an organelle (such as mitochondria) in the cell also appears.

  When the tab 104 is selected, the display image 102 described above is displayed. When the tab 105 is selected, only the original image as a gray scale image is displayed. When the tab 106 is selected, only the film likelihood map (film likelihood image) as a gray scale image is displayed. By observing the film likelihood map, it can be confirmed that the threshold is appropriately set. Moreover, according to the original display of the original image, it becomes easy to observe the details of the film in detail. A tab for displaying only a mask image as a work target image may be added. A membrane likelihood map may be used as a background image, and a temporary membrane image may be superimposed on the background image.

  The depth selection tool 108 is a display element (operation element) for selecting a specific depth (display depth) in the Z direction. It consists of a Z-axis symbol 108b representing the Z-axis and a marker 108a as a slider that slides along the Z-axis symbol 108b. It is possible to select the desired depth by moving the marker 108a. According to such a depth selection tool 108, it is possible to obtain an advantage that it is easy to intuitively recognize the depth to be selected and the depth change amount. The left end point of the Z-axis symbol 108b corresponds to zero depth, and the right end point of the Z-axis symbol 108b corresponds to the maximum depth. Depth selection tools having other forms may be employed. The depth input field 114 is a field for directly specifying the depth as a numerical value. The depth currently selected according to the position of the marker 108a may be displayed as a numerical value in the depth input field 114.

  The transparency adjustment tool 110 is a tool for adjusting the transparency (display weight) of a color temporary membrane image (work target image) that is displayed in a composite state while the display image 102 is displayed. For example, when the marker 110a is moved to the left side, the display weight of the color mask image is reduced, the transparency is increased, and the original image is dominantly displayed. On the contrary, when the marker 110a is moved to the right side, the display weight of the color mask image increases, the transparency decreases, and the color mask image is displayed more clearly.

  The superimposition display tool 112 is an image (synthesized image, original image or film likelihood map) adjacent to the currently displayed image (synthesized image, original image or film likelihood map) on the shallow side in the depth direction. Alternatively, the operation is performed when an image (a composite image, an original image, or a film likelihood map) adjacent to the deep side in the depth direction is displayed in an overlapping manner. When the marker 112a is moved to the left side, the display weight for the image adjacent to the shallow side increases. Conversely, when the marker 112a is moved to the right side, the display weight for the image adjacent to the deep side increases. Three or more images may be superimposed and displayed. Of course, if too many images are superimposed, the image content becomes too complex, so it is desirable to overlay a small number of images. Such superposition display makes it easy to obtain depth information.

  The button row 115 includes a plurality of virtual buttons 116, 118, 120, 121, 122, 126. The button 116 is a display element operated when performing image zoom (enlargement or reduction). The button 118 is a display element that is operated when the pen tool is used. When the button 118 is turned on, the shape of the cursor is changed to a pen shape, and a film pixel can be added using the cursor shape. It is also possible to change the pen size. The button 120 is a display element that is operated when using an eraser. When the button 120 is turned on, the shape of the cursor is changed to an eraser shape, and the film pixel can be deleted using the cursor shape. It is also possible to change the size of the eraser.

  The button 121 is a display element that is operated when painting. If any area is designated after the button 121 is turned on, the area is filled. It is also possible to select an arbitrary function from a plurality of functions prepared for painting (or labeling) by operating the button 121. By operating the object number (label) button 122, the color palette 124 is displayed. For example, a color selected from the color palette is given to the painted area. Thereby, the area is colored by the selected color. Each color is associated with an object number. If the same color, that is, the same object number is given to a plurality of regions across the layers, a three-dimensional lumen region in a specific cell is defined by these regions.

  The button 126 is a button for black and white reversal. When it is operated, a portion displayed in black in the displayed image is displayed in white, and conversely, a portion displayed in white is displayed in black.

  In addition to the above, it is desirable to provide a button for generating and displaying a three-dimensional image based on a plurality of temporary film images after labeling. The contents of the work window 100 shown in FIG. 5 are merely examples, and it is desirable that the contents be appropriately determined so that the workability is improved in the user work.

  FIG. 6 shows a temporary mask image 180 before correction and a temporary mask image 182 after correction. The temporary film image 180 before correction includes film break portions 184 and 188. In the corrected temporary membrane image 182, as shown by reference numerals 186 and 190, the film interruption portion is repaired. In addition, the temporary membrane image 180 before correction includes non-membrane portions 192 and 196. In the corrected temporary membrane image 182, as indicated by reference numerals 194 and 198, those non-membrane portions have disappeared. According to the corrected mask image 182, it is possible to accurately perform labeling, and consequently, it is possible to improve the quality of the three-dimensional labeling data.

  FIG. 7 illustrates a three-dimensional splicing process included in the labeling process. In FIG. 7, a plurality of temporary membrane images D1 to D4 arranged in the Z direction are shown. A temporary membrane image D1 corresponding to the depth Zi is a work target image. This is a reference image in the processing described below.

  In the first example of the three-dimensional joining process, a representative point is specified for the region R1 included in the mask image D1 that is the reference image. For example, a center point, a center of gravity point, etc. are specified as representative points. Next, a perpendicular C passing through the representative point is defined. From the reference image, for example, N masked membrane images are referred to on the deep side, and each region through which the perpendicular C passes is specified in those images. And the same label is provided with respect to area | region R1, R2, R3, R4, ... through which the perpendicular C passes. The above processing may be applied to N temporary film images on the shallow side from the reference image. The result of automatic labeling is usually visually confirmed by the user.

  In the second example of the three-dimensional splicing process, the region R1 (outer edge) included in the mask image D1 that is the reference image is projected onto the mask image D2, and the projection region R1a is defined. A region R2 that most overlaps the projection region R1a is specified on the mask image D2. Subsequently, the region R2 is projected onto the mask image D3, and the projection region R2a is defined. A region R3 that most overlaps the projection region R2a is specified on the mask image D3. Similarly, a projection region R3a is defined on the mask image D4, and the region R4 is specified based on the projection region R3a. The same label is assigned to the region R1 and all of the regions R2, R3, and R4 identified from the region R1.

  In the above processing, the projection source area is sequentially updated when the layer is changed, but it may be fixed. For example, the region R1 may be projected onto the mask images D2, D3, D4. In the above processing, the connection destination is searched from the reference image to one side in the Z direction, but the connection destination may be searched from the reference image to both sides in the Z direction. The search range may be selected by the user. In any case, the result of automatic labeling is usually visually confirmed by the user. In that case, the work window shown in FIG. 5 is used.

  The three-dimensional connection process described above is an example, and a three-dimensional connection process other than the above may be employed. In specifying the connection relationship between regions, one or a plurality of feature amounts for each region can be used. As the feature amount of the region, an area, a form, a perimeter, a barycentric point, a luminance histogram, a texture, or the like may be used. Alternatively, a superposition area, a distance between representative points, or the like may be used as a feature amount between regions.

  FIG. 8 schematically shows exaggerated changes in the observation depth range and changes in the SEM image that occur with changes in the applied voltage. Reference numerals 230, 232, and 234 indicate three different observation conditions. The observation condition 230 is a condition for setting the acceleration voltage to a low voltage. The observation condition 232 is a condition in which the acceleration voltage is a medium voltage. The observation condition 234 is a condition for increasing the acceleration voltage. The upper part (A) of FIG. 8 shows the relationship between the cross section of the sample piece 236 and the electron beams 244, 246, 248. Each cross section includes a film portion 238 extending in parallel with the Z direction and a film portion 240 extending in a direction oblique to the Z direction. A SEM image is shown in the lower part (B) of FIG.

  In the illustrated example, when the observation condition 230 is selected, the object to be observed by the electron beam 244 is exclusively the surface layer of the sample section 236. Therefore, on the SEM image, images 238a and 240a representing the surface layers of the two film portions 238 and 240 appear. When the observation condition 232 is selected, the object to be observed by the electron beam 246 extends to the intermediate layer of the sample slice 236. Therefore, on the SEM image, an image (intermediate integrated image) 238b representing the intermediate layer of the film portion 238 appears, and a deep image 240c appears in the vicinity of the image 240b for the film portion 240. When the observation condition 234 is selected, the object to be observed by the electron beam 248 extends to the entire sample section 236 in the illustrated example. Therefore, on the SEM image, an image (entire integrated image) 238c representing the entire film portion 238 appears, and a wide depth image 240e appears in the vicinity of the image 240d for the film portion 240.

  In order to avoid or reduce damage to the sample, it is required to observe the sample at a low acceleration voltage.On the other hand, when the membrane is estimated by the membrane estimator, the depth image of the cell membrane is also correctly estimated as a part of the cell membrane. Is required. It is necessary to firmly instruct the membrane estimator that the deep image of the cell membrane is a part of the cell membrane. That is, in the learning stage of the film estimator, it is necessary to give the film estimator an image including many deep images. From such a viewpoint, in the embodiment, in the learning process of the film estimator, a plurality of high voltage images are provided to the film estimator together with the plurality of low voltage images. Their ratio may be 1: 1, but the ratio can vary depending on the situation.

  The operation of the second image processing unit will be described using a specific example with reference to FIGS. 9 and 10. In the second image processing unit, first, as shown in FIG. 9, contrast enhancement 252 as moya enhancement is applied to the second original image (high voltage image) 250. Subsequently, segmentation 256 is performed on the image 254 after contrast enhancement. As shown in FIG. 10, many small regions are set in the image 258 after segmentation. On the image 258, a membrane label is given by the user to a plurality of small regions corresponding to the cell membrane. A membrane image 262 is created by painting a plurality of small regions to which a membrane label is attached. The film image 262 may be corrected after painting. The image may be corrected in an earlier stage. The film image 262 created in this way is used as the second correct image. In addition, in the film | membrane image 262 shown by FIG. 10, a white part is a film | membrane part.

  The effects of the embodiment will be described with reference to FIGS. 11 and 12. As already explained, in the embodiment, both the low voltage image and the high voltage image are given to the film estimator in the learning stage of the film estimator. On the other hand, the following comparative example gives only a low voltage image to the film estimator in the learning stage of the film estimator.

  An input image 270 is shown on the left side of FIG. 11, and a correct image 272 corresponding to the input image 270 is shown on the right side of FIG. In the correct image 272, the black part is the film part.

  On the left side of FIG. 12, a film likelihood map 274 according to a comparative example is shown. A membrane likelihood map 276 according to the embodiment is shown on the right side of FIG. In these film likelihood maps 274 and 276, the magnitude of the film likelihood is expressed in black. When the two film likelihood maps 274 and 276 are compared, the film portions A1, B1, C1, and D1 that are thinly expressed on the film likelihood map 274 are the film portions A2 that are expressed deeply on the film likelihood map 276. It has changed to B2, C2, and D2. Here, the film portions A1 and B1 (and the film portions A2 and B2) are considered to be portions including a deep image. As described above, by causing the membrane estimator to learn using a high-voltage image, it is possible to improve the estimation accuracy of cell membranes having various orientations.

  The land error for the comparative example was 0.012, while the land error for the embodiment was 0.009. The land error is defined by 1- (Rand Index), where Rand Index is a publicly known index indicating the degree of coincidence or similarity between the evaluation target image and the correct image.

  FIG. 13 shows a second configuration example of the main body. In addition, the same code | symbol is attached | subjected to the structure same as the structure shown in FIG. 2, and the description is abbreviate | omitted.

  The main body 280 includes a first image processing unit 200 and a second image processing unit 281. The second image processing unit 281 includes a machine learning type film estimator 282, a binarizer 284, and a correction unit 286. The film estimator 282 is composed of CNN in the illustrated example. The film estimator 282 applies a film estimation process to the second original image (high voltage image) 36H as the input image 208, and outputs a film likelihood map. The film estimator 282 is a learned estimator that has undergone learning using a high-voltage image. Similarly to the binarizer 50, the binarizer 284 generates a temporary membrane image from the membrane likelihood map. The correction unit 286 has the same function as that of the correction unit 52, accepts user correction on the mask image, and generates the second correct image 224 thereby. The second configuration example uses a membrane estimator 282 that is different from the membrane estimator 42 in generating the second correct image.

  In the above embodiment, two observation conditions are used, but three or more observation conditions may be used. Although CNN is used as the film estimator, other machine learning type film estimators may be used.

DESCRIPTION OF SYMBOLS 10 Biological tissue image processing system, 16 Scanning electron microscope (SEM), 18 Biological tissue image processing apparatus, 42 Membrane estimator, 44 Processing tool unit (work support part), 50 Binarizer, 52 Correction part, 54 Labeling Processing unit, 56 Volume data processing unit, 200 First image processing unit, 202 Second image processing unit.

Claims (10)

An electron microscope that generates an image by observing a sample to be analyzed that is a biological tissue;
A machine learning type estimator for applying a process of estimating an element of interest included in the analysis target sample to the image;
Including
The estimator is a learned estimator that has undergone learning using a learning original image set,
The learning original image set includes a first original image generated by observing a learning target sample that is a biological tissue with an electron microscope under a first observation condition, and an electron microscope under a second observation condition. A second original image generated by observing the learning target sample, and
The first observation condition is a condition set for an electron microscope when observing the analysis target sample, and the second observation condition can be observed over a depth range different from the first observation condition. Condition,
A biological tissue image processing system characterized by that. The biological tissue image processing system according to claim 1,
The second observation condition is a condition that enables observation over a depth range larger than the first observation condition.
A biological tissue image processing system characterized by that. The biological tissue image processing system according to claim 2,
The first observation condition is a condition for setting the first acceleration voltage as the acceleration voltage of the electron microscope,
The second observation condition is a condition for setting a second acceleration voltage higher than the first acceleration voltage as an acceleration voltage of an electron microscope.
A biological tissue image processing system characterized by that. The biological tissue image processing system according to claim 3,
A correct image creation unit that generates a second correct image based on the second original image;
A biological tissue image processing system characterized by that. The biological tissue image processing system according to claim 4, wherein
The correct image creation unit includes an enhancement unit that enhances a deep image of the element of interest in the second original image.
A biological tissue image processing system characterized by that. The biological tissue image processing system according to claim 5, wherein
The element of interest is a cell membrane,
The deep image of the element of interest is a deep image of the cell membrane,
A biological tissue image processing system characterized by that. The apparatus of claim 6.
The correct image creation unit
A segmentation unit that performs segmentation to divide an image that has undergone processing by the enhancement unit into a plurality of small regions;
In the image after the segmentation, an annotation unit that generates a second correct image by labeling a plurality of small regions corresponding to the cell membrane;
A biological tissue image processing system comprising: The biological tissue image processing system according to claim 4, wherein
The machine learning type estimator is a first estimator;
The correct image creation unit includes a machine learning type second estimator,
A biological tissue image processing system characterized by that. Biological tissue image processing including a machine learning type estimator that applies processing for estimating an element of interest included in the analysis target sample to an image generated by observing the analysis target sample as a biological tissue with an electron microscope A machine learning method for causing the estimator to learn in a system,
For the estimator, a first original image for learning generated by observing a learning target sample as a biological tissue with an electron microscope under a first acceleration voltage, and a first learning image corresponding thereto. Providing a correct image;
A second original image for learning generated by observing the learning object sample with an electron microscope under a second acceleration voltage higher than the first acceleration voltage with respect to the estimator, and corresponding thereto Providing a second correct image for learning;
A machine learning method comprising: In a biological tissue image processing system including a machine learning type estimator that applies a process of estimating an element of interest included in an analysis target sample to an image generated by observing the analysis target sample as a biological tissue with an electron microscope , A program for executing a machine learning method for causing the estimator to learn,
A first original image for learning generated by observing a learning target sample that is a biological tissue with an electron microscope under a first acceleration voltage, and a first learning image corresponding to the estimator. The ability to give correct images,
A second original image for learning generated by observing the learning target sample with a microscope under a second acceleration voltage higher than the first acceleration voltage with respect to the estimator, and learning corresponding thereto A function for providing a second correct image for
The program characterized by including.